PyAirtable File Service

Enterprise-grade file upload, storage, and processing microservice built with Go.

Overview

The PyAirtable File Service is a high-performance, scalable microservice designed for handling file operations in multi-tenant environments. It provides secure file upload, storage, processing, and retrieval capabilities with support for multiple storage backends and advanced file processing features.

Key Features

File Management

• Multi-format Support: Images, documents, videos, and any file type
• Multiple Storage Backends: Local filesystem, Amazon S3, Google Cloud Storage
• File Deduplication: Hash-based duplicate detection to save storage
• Secure Upload: Presigned URLs for direct client uploads
• Chunked Upload: Support for large file uploads with resumable sessions
• File Validation: MIME type checking, size limits, and virus scanning

Processing & Optimization

• Image Processing: Resize, crop, format conversion, thumbnail generation
• Document Processing: PDF preview generation, metadata extraction
• Compression: Automatic file optimization and compression
• Background Jobs: Asynchronous processing with job queues
• Metadata Extraction: Automatic EXIF, document, and media metadata extraction

Security & Access Control

• Tenant Isolation: Multi-tenant architecture with data segregation
• Access Control: Fine-grained permissions and public/private files
• Secure URLs: Time-limited, signed URLs for file access
• Virus Scanning: Integrated malware detection
• Audit Logging: Comprehensive access and operation logging

Performance & Scalability

• Streaming: Memory-efficient streaming for large files
• CDN Ready: Optimized for CDN integration
• Caching: Redis-based caching for frequently accessed data
• Quota Management: Per-tenant storage and file count limits
• Rate Limiting: Configurable rate limits and throttling

Quick Start

Prerequisites

• Go 1.21+
• Docker & Docker Compose
• PostgreSQL 15+
• Redis 7+

Local Development

1. Clone and setup:

   git clone <repository-url>
   cd fileservice
   cp .env.example .env

2. Start dependencies:

   make dev

3. Run the service:

   make run

The service will be available at http://localhost:8080

Using Docker

# Build and run with Docker Compose
make docker-compose-up

# Or build and run individually
make docker-build
make docker-run

API Endpoints

File Operations

Method	Endpoint	Description	Auth Required
POST	/files/upload	Upload file (multipart)	Yes
POST	/files/upload-url	Generate presigned upload URL	Yes
GET	/files/:id	Get file metadata	Yes
GET	/files/:id/download	Download file or get download URL	Yes
DELETE	/files/:id	Delete file	Yes
POST	/files/:id/process	Process file (resize, convert)	Yes
GET	/files	List files (tenant-scoped)	Yes
GET	/files/:id/thumbnail	Get thumbnail	Yes

Chunked Upload

Method	Endpoint	Description	Auth Required
POST	/files/upload/init	Initiate chunked upload	Yes
GET	/files/upload/:sessionId/status	Get upload status	Yes
POST	/files/upload/:sessionId/complete	Complete chunked upload	Yes
DELETE	/files/upload/:sessionId	Abort chunked upload	Yes
POST	/files/upload/chunk/:sessionId	Upload chunk	Yes

Processing & Jobs

Method	Endpoint	Description	Auth Required
GET	/files/:id/jobs	Get processing jobs for file	Yes
GET	/files/stats/tenant/:tenantId	Get file statistics	Yes

Quota Management

Method	Endpoint	Description	Auth Required
GET	/quota/:tenantId	Get tenant quota	Yes
PUT	/quota/:tenantId	Set tenant quota	Yes
GET	/quota/exceeded	Get quota exceeded tenants	Yes

System

Method	Endpoint	Description	Auth Required
GET	/health	Health check	No
GET	/metrics	Prometheus metrics	No

Authentication

Protected endpoints require a valid JWT token in the Authorization header:

curl -H "Authorization: Bearer <jwt-token>" \
     http://localhost:8080/api/v1/protected

Configuration

Configuration is managed through environment variables:

Variable	Description	Default
PORT	Server port	8080
DB_HOST	Database host	localhost
DB_PORT	Database port	5432
DB_NAME	Database name	file_service_db
REDIS_HOST	Redis host	localhost
REDIS_PORT	Redis port	6379
JWT_SECRET	JWT signing secret	your-secret-key
LOG_LEVEL	Logging level	info

See .env.example for a complete list.

Development

Available Commands

make help          # Show all available commands
make deps          # Download dependencies
make build         # Build the application
make run           # Run the application
make test          # Run tests
make test-coverage # Run tests with coverage
make bench         # Run benchmarks
make lint          # Run linter
make format        # Format code
make clean         # Clean build artifacts

Project Structure

fileservice/
├── cmd/file-service/           # Application entrypoint
├── internal/                      # Private application code
│   ├── config/                   # Configuration management
│   ├── handlers/                 # HTTP handlers
│   ├── services/                 # Business logic
│   ├── repositories/             # Data access layer
│   ├── middleware/               # Custom middleware
│   └── models/                   # Data models
├── pkg/                          # Public packages
│   ├── database/                 # Database utilities
│   ├── redis/                    # Redis utilities
│   ├── logger/                   # Logging utilities
│   └── metrics/                  # Metrics utilities
├── test/                         # Test files
├── deployments/                  # Deployment configs
│   ├── docker/                   # Docker configs
│   └── k8s/                      # Kubernetes manifests
└── .github/workflows/            # CI/CD workflows

Testing

# Run all tests
make test

# Run with coverage
make test-coverage

# Run only unit tests
go test ./test/unit/...

# Run only integration tests
go test ./test/integration/...

# Run benchmarks
make bench

Adding New Features

1. Add models in internal/models/
2. Create repositories in internal/repositories/
3. Implement services in internal/services/
4. Add handlers in internal/handlers/
5. Register routes in cmd/file-service/main.go
6. Add tests in test/unit/ and test/integration/

Deployment

Docker

# Build image
make docker-build

# Run container
make docker-run

Kubernetes

# Deploy to Kubernetes
kubectl apply -k deployments/k8s/

# Check deployment status
kubectl get pods -n fileservice

Environment-specific Deployments

The service includes Kustomize configurations for different environments:

• Development: Basic setup with minimal resources
• Staging: Production-like setup with monitoring
• Production: Full setup with HPA, ingress, and security policies

Monitoring

Health Checks

• Liveness: GET /health
• Readiness: GET /health (customize as needed)

Metrics

Prometheus metrics are available at /metrics:

• HTTP request metrics (count, duration)
• Database connection metrics
• Redis connection metrics
• Custom business metrics

Logging

Structured JSON logging with configurable levels:

{
  "time": "2024-01-01T12:00:00Z",
  "level": "INFO",
  "msg": "Server started",
  "port": "8080"
}

Security

• JWT Authentication: Secure token-based auth
• CORS: Configurable cross-origin policies
• Rate Limiting: Built-in request rate limiting
• Security Headers: Helmet middleware for security headers
• Input Validation: Request validation and sanitization

Contributing

1. Fork the repository
2. Create a feature branch: git checkout -b feature/amazing-feature
3. Commit changes: git commit -m 'Add amazing feature'
4. Push to branch: git push origin feature/amazing-feature
5. Open a Pull Request

Code Standards

• Follow Go best practices and conventions
• Write tests for new features
• Update documentation as needed
• Run linter and formatter before submitting

License

This project is licensed under the MIT License - see the LICENSE file for details.

Support

For support and questions:

• Author: PyAirtable Team
• Issues: GitHub Issues
• Documentation: Wiki

---

Generated with Go Microservice Template