Storage Service - Cacciaguida

The Storage Service in the Dante GPU Platform is designed to provide an abstraction layer for object storage. This service allows other components, like the Provider Daemon or API Gateway, to upload, download, and manage files such as user-uploaded datasets, AI model files, job input/output artifacts, and intermediate results without needing to know the specifics of the underlying storage technology.

1. Responsibilities

The primary responsibilities of the Storage Service include:

  • File Uploads: Providing an API endpoint for uploading files to a designated storage backend.

  • File Downloads: Offering an API endpoint for downloading previously stored files.

  • File Deletion: Allowing deletion of files from the storage backend.

  • File Listing: Providing functionality to list objects within a bucket, possibly filtered by a prefix.

  • Object Metadata: Retrieving metadata for a specific object.

  • Presigned URLs: Generating time-limited presigned URLs for direct client-side uploads (PUT) or downloads (GET) to/from the storage backend, reducing the load on the Storage Service itself.

  • Bucket Management: Ensuring specified buckets exist in the storage backend and creating them if necessary.

  • Service Discovery: Registering itself with Consul for discovery by other services.

2. Tech Stack

The Storage Service is built using the following technologies:

  • Programming Language: Go (version 1.22+)

  • Object Storage Backend: MinIO (S3-compatible)

  • HTTP Routing: Chi router

  • Logging: Zap for structured logging

  • Service Discovery: HashiCorp Consul (via Consul API Client)

  • Configuration: YAML files

3. Core Components and Logic

3.1. Configuration (configs/config.yaml)

The service's behavior is controlled by configs/config.yaml. Key parameters include:

  • instance_id: Unique identifier for the service instance, auto-generated if empty.

  • log_level: Logging verbosity (e.g., "info", "debug").

  • request_timeout: Default timeout for HTTP server requests.

  • Server Configuration (server):

    • host: Host to listen on (e.g., "0.0.0.0").

    • port: Port to listen on (e.g., 8082).

  • Consul Configuration (consul):

    • enabled: Whether Consul registration is active.

    • address: Consul agent address.

    • registration: Details for service registration in Consul (service name, ID prefix, tags, health check path, intervals).

  • MinIO Configuration (minio):

    • endpoint: MinIO server endpoint (e.g., "localhost:9000").

    • accessKeyID, secretAccessKey: Credentials for MinIO.

    • useSSL: Whether to use SSL for MinIO connection.

    • region: MinIO region (optional).

    • defaultBucket: Default bucket to use if not specified in requests.

    • autoCreateDefaultBucket: Whether to automatically create the default bucket on startup if it doesn't exist.

If the configuration file is not found, a default one is created.

3.2. Storage Backend (internal/storage/)

  • ObjectStorage Interface: Defines a standard set of operations for object storage, allowing different backends (like MinIO, S3) to be used interchangeably. Methods include Upload, Download, Delete, ListObjects, GetObjectInfo, EnsureBucket, and GetPresignedURL.

  • MinioClient: Implements the ObjectStorage interface using the MinIO Go SDK.

    • Initialization (NewMinioClient): Connects to the MinIO server using credentials from the config and performs a connection test.

    • Bucket Handling (EnsureBucket): Creates a bucket if it doesn't exist.

    • Uses the defaultBucket from config if a bucket name is not provided in an operation.

  • ObjectInfo Struct: Contains metadata about a stored object like key, size, last modified date, content type, and ETag.

3.3. API Handlers (internal/api/handlers.go)

  • StorageHandler: Handles all HTTP requests related to storage operations. It uses an ObjectStorage client.

  • Routes:

    • Bucket creation: POST /buckets/{bucketName}

    • Object operations:

      • GET /objects/{bucketName}/*: Download object.

      • PUT /objects/{bucketName}/*: Upload object.

      • DELETE /objects/{bucketName}/*: Delete object.

      • HEAD /objects/{bucketName}/*: Get object metadata.

      • GET /objects/{bucketName}/list: List objects.

    • Presigned URL generation: POST /presigned-url/{bucketName}/*

    • Convenience routes for default bucket are also provided (e.g., GET /objects/*).

  • Upload Handling:

    • Limits upload size (e.g., to 5GB).

    • Determines Content-Type from request header or by guessing from file extension; defaults to application/octet-stream.

  • Download Handling: Sets Content-Type, Content-Length, ETag, and Last-Modified headers in the response.

  • Presigned URL Generation: Accepts method ("GET" or "PUT") and expiry duration in the request body. Defaults to 15 minutes expiry.

3.4. Main Application (cmd/main.go)

  • Initialization: Loads config, sets up logger, initializes the MinIO client, and ensures the default bucket exists if configured.

  • HTTP Server: Sets up Chi router with middleware for request ID, real IP, structured logging, panic recovery, and request timeout.

  • Health Check: Exposes a /health endpoint (path configurable) that returns a JSON status.

  • Consul Registration: If enabled in config, registers the service with Consul, including its health check.

  • Graceful Shutdown: Handles SIGINT and SIGTERM signals to deregister from Consul (if applicable) and shut down the HTTP server gracefully.

4. API Endpoints

The service exposes RESTful API endpoints. Object keys in paths can include slashes.

  • POST /buckets/{bucketName}: Ensures a bucket exists. If it doesn't, it's created.

    • Response: 201 Created with { "message": "Bucket ensured successfully", "bucket": "bucketName" }

  • PUT /objects/{bucketName}/{objectKey...}: Uploads a file to the specified bucket and object key.

    • Request Body: Raw file data.

    • Headers: Content-Type (optional, will be guessed), Content-Length (optional).

    • Response: 201 Created with ObjectInfo JSON.

  • GET /objects/{bucketName}/{objectKey...}: Downloads a file.

    • Response: Raw file data with appropriate content headers.

  • DELETE /objects/{bucketName}/{objectKey...}: Deletes a file.

    • Response: 204 No Content.

  • HEAD /objects/{bucketName}/{objectKey...}: Retrieves metadata of a file.

    • Response: Empty body with content headers (Content-Type, Content-Length, ETag, Last-Modified).

  • GET /objects/{bucketName}/list: Lists objects in a bucket.

    • Query Parameters: prefix (string, optional), recursive (bool, optional, default: true).

    • Response: 200 OK with an array of ObjectInfo JSON.

  • POST /presigned-url/{bucketName}/{objectKey...}: Generates a presigned URL.

    • Request Body (JSON): {"method": "GET|PUT", "expiry": "15m"} (expiry is optional, defaults to 15 minutes).

    • Response: 200 OK with {"url": "presigned_url_string", "method": "GET|PUT", "key": "objectKey", "bucket": "bucketName"}.

  • Convenience Endpoints for Default Bucket:

    • PUT /objects/{objectKey...}

    • GET /objects/{objectKey...}

    • DELETE /objects/{objectKey...}

    • HEAD /objects/{objectKey...}

    • GET /objects/list

    • POST /presigned-url/{objectKey...}

  • GET /health (or configured path): Health check endpoint.

    • Response: 200 OK with {"status": "UP"}.

5. Workflow

  1. Startup:

    • Service loads configuration from configs/config.yaml.

    • Initializes the Zap logger.

    • Connects to the MinIO server using the provided endpoint and credentials.

    • If autoCreateDefaultBucket is true and defaultBucket is set, it ensures the default bucket exists.

    • If Consul is enabled, registers itself with the Consul agent, providing a health check endpoint.

    • The HTTP server starts, listening for API requests on the configured port.

  2. File Upload (PUT /objects/{bucket}/{key} ):

    • The uploadObjectHandler receives the request.

    • It extracts bucketName and objectKey from the URL.

    • The request body is read as the file stream. Upload size is limited.

    • Content-Type is determined.

    • The storageClient.Upload method is called, which in turn uses minioClient.PutObject.

    • The service responds with 201 Created and the ObjectInfo.

  3. File Download (GET /objects/{bucket}/{key} ):

    • The downloadObjectHandler receives the request.

    • storageClient.Download is called, which uses minioClient.GetObject and obj.Stat.

    • Relevant headers are set, and the file stream is copied to the HTTP response writer.

  4. Presigned URL Generation (POST /presigned-url/{bucket}/{key} ):

    • The generatePresignedURLHandler receives the request.

    • It expects a JSON body specifying the HTTP method (GET or PUT) and an optional expiry duration.

    • storageClient.GetPresignedURL is called, which uses minioClient.PresignedGetObject or minioClient.PresignedPutObject.

    • The generated URL is returned in the JSON response.

  5. Shutdown:

    • Upon receiving SIGINT or SIGTERM, the service deregisters from Consul (if enabled).

    • The HTTP server is shut down gracefully.

6. Future Considerations

  • Authentication/Authorization: Currently, endpoints are open. Integration with an auth system (like tokens validated by the API Gateway, or internal ACLs) will be necessary.

  • Support for Other Backends: Implement the ObjectStorage interface for other cloud storage services like AWS S3 or Google Cloud Storage.

  • Data Versioning: Add support for object versioning in the storage backend.

  • Lifecycle Management: Implement policies for managing the lifecycle of stored objects (e.g., moving to colder storage, deletion after a certain period).

  • Advanced Health Checks: The health check could be enhanced to verify connectivity and functionality of the underlying storage backend (e.g., write/read a small test object to MinIO).

  • Streaming Uploads/Downloads: Ensure efficient handling of large file streams. The current MinIO client usage seems to support this.

  • Multipart Uploads: For very large files, explicitly supporting multipart uploads through the API could be beneficial, though the MinIO SDK might handle this transparently for PutObject with unknown size.

  • Error Handling: More specific error types and codes could be returned to clients (e.g., ErrObjectNotFound mapped to 404 Not Found).

PreviousProvider Daemon - LeahNextBilling Payment Service - Geryon

Last updated