S3 and RGW Account Management API Documentation

This document provides an overview of the RESTful APIs for managing S3 buckets, files, and RADOS Gateway (RGW) accounts. The APIs are built using Express.js, with middleware for authentication, request validation, and error handling, and integrate with MongoDB, PostgreSQL, Redis, RabbitMQ, and AWS S3.

Table of Contents

• Prerequisites
• API Endpoints
  • S3 Bucket and File Operations
  • RGW Account Operations
  • Health Check
• Authentication
• Request Validation
• Error Handling
• Setup Instructions
• Dependencies

Prerequisites

• Node.js: Version 14.x or higher
• MongoDB: For storing metadata
• PostgreSQL: For queue configuration
• Redis: For caching
• RabbitMQ: For task queuing
• AWS S3 or Compatible Service: For object storage
• Environment Variables: Configure .env with JWT_SECRET, PORT, HOST, database credentials, and S3 configurations

API Endpoints

S3 Bucket and File Operations

All endpoints require authentication via a Bearer token in the Authorization header. The base path is /api/s3.

1. List Buckets

   • Endpoint: GET /api/s3/list-buckets
   • Description: Retrieves a list of S3 buckets for the authenticated user.
   • Request Body: None
   • Response:

     {
       "buckets": [
         {
           "Name": "string",
           "CreationDate": "string",
           "LastModified": "string",
           "IsEmpty": boolean,
           "Error": "string" (optional)
         }
       ],
       "message": "Buckets listed with details"
     }

   • Status Codes:
     • 200: Success
     • 400: Missing userId or organizationId
     • 500: Server error or failed to create S3 client

2. Create Bucket

   • Endpoint: POST /api/s3/create-bucket
   • Description: Queues a task to create a new S3 bucket.
   • Request Body:

     {
       "bucketName": "string"
     }

   • Validation:
     • bucketName: Required, 3-63 characters, lowercase, numbers, dots, or hyphens; must not already exist
   • Response:

     {
       "taskId": "string",
       "message": "Create bucket task queued for {bucketName}"
     }

   • Status Codes:
     • 200: Task queued successfully
     • 400: Invalid bucket name or bucket already exists
     • 500: Server error

3. Upload File

   • Endpoint: POST /api/s3/upload-file
   • Description: Queues a task to upload one or more files to an S3 bucket.
   • Request Body: Form-data with bucketName and files (up to 10 files, max 5GB each)

     bucketName: string
     files: file(s)
     

   • Validation:
     • bucketName: Required, must exist
     • At least one file required
   • Response:

     {
       "taskId": "string",
       "message": "File upload task queued for {n} files"
     }

   • Status Codes:
     • 200: Task queued successfully
     • 400: Invalid bucket name, bucket does not exist, or no files uploaded
     • 500: Server error

4. Download File

   • Endpoint: POST /api/s3/download-file
   • Description: Generates a pre-signed URL for downloading a file from an S3 bucket.
   • Request Body:

     {
       "bucketName": "string",
       "fileName": "string"
     }

   • Validation:
     • bucketName: Required, must exist
     • fileName: Required, must be a string
   • Response:

     {
       "fileUrl": "string",
       "message": "Presigned URL generated for {fileName}"
     }

   • Status Codes:
     • 200: Success
     • 400: Missing or invalid bucketName/fileName
     • 500: Server error or failed to create S3 client

5. Delete File

   • Endpoint: POST /api/s3/delete-file
   • Description: Queues a task to delete a file from an S3 bucket.
   • Request Body:

     {
       "bucketName": "string",
       "fileName": "string"
     }

   • Validation:
     • bucketName: Required, must exist
     • fileName: Required, must be a string
   • Response:

     {
       "taskId": "string",
       "message": "Delete file task queued for {fileName}"
     }

   • Status Codes:
     • 200: Task queued successfully
     • 400: Invalid bucket name or file name
     • 500: Server error

6. List Files in Bucket

   • Endpoint: POST /api/s3/list-files
   • Description: Lists all files in a specified S3 bucket.
   • Request Body:

     {
       "bucketName": "string"
     }

   • Validation:
     • bucketName: Required, must exist
   • Response:

     {
       "files": ["string"],
       "message": "Files listed successfully for bucket {bucketName}"
     }

   • Status Codes:
     • 200: Success
     • 400: Missing or invalid bucket name
     • 500: Server error

7. Get File Metadata

   • Endpoint: POST /api/s3/file-metadata
   • Description: Retrieves metadata for a specific file in an S3 bucket.
   • Request Body:

     {
       "bucketName": "string",
       "fileName": "string"
     }

   • Validation:
     • bucketName: Required, must exist
     • fileName: Required, must be a string
   • Response:

     {
       "status": "success",
       "data": { "metadata": {} },
       "message": "string"
     }

   • Status Codes:
     • 200: Success
     • 400: Missing or invalid bucket name/file name
     • 500: Server error

8. Delete Bucket

   • Endpoint: POST /api/s3/delete-bucket
   • Description: Queues a task to delete an empty S3 bucket.
   • Request Body:

     {
       "bucketName": "string"
     }

   • Validation:
     • bucketName: Required, must exist and be empty
   • Response:

     {
       "taskId": "string",
       "message": "Delete bucket task queued for {bucketName}"
     }

   • Status Codes:
     • 200: Task queued successfully
     • 400: Invalid bucket name or bucket not empty
     • 500: Server error

9. Refresh Download URL

   • Endpoint: POST /api/s3/refresh-download-url
   • Description: Refreshes a pre-signed URL for downloading a file (valid for 15 minutes).
   • Query Parameters:
     • bucketName: Required, must exist
     • fileName: Required, must be a string
   • Response:

     {
       "fileUrl": "string",
       "message": "Pre-signed URL refreshed for {fileName}"
     }

   • Status Codes:
     • 200: Success
     • 400: Missing or invalid bucket name/file name
     • 500: Server error

✅ RGW Account Operations

Base path: /api/rgwacc

1. Create RADOS Account

   • Endpoint: POST /api/rgwacc/create-account

   • Description: Creates a new RADOS Gateway account for the given user and organization.

   • Request Body:

     {
       "display_name": "string",
       "userId": "string",
       "organizationId": "string"
     }

   • Validation: All fields are required.

   • Response:

     {
       "message": "RADOS account created (mock)",
       "userId": "string",
       "subuserid": "string",
       "accessKey": "string",
       "secretKey": "string",
       "s3ClientEndpoint": "string",
       "s3ClientRegion": "string",
       "organizationId": "string",
       "display_name": "string"
     }

   • Status Codes:

     • 201: Success
     • 400: Validation error
     • 500: Internal error

2. Delete Ceph Key

   • Endpoint: POST /api/rgwacc/delete-key

   • Description: Deletes a non-default Ceph key for the authenticated user.

   • Request Body:

     {
       "keyId": "string"
     }

   • Validation: keyId must be a valid non-default key.

   • Response:

     {
       "message": "Successfully deleted Ceph key {keyId} for user {userId} in organization {organizationId}"
     }

   • Status Codes:

     • 200: Success
     • 400: Invalid input or non-default key missing
     • 404: Key not found
     • 500: Server error

3. Get Ceph Credentials

   • Endpoint: POST /api/rgwacc/get-credentials

   • Description: Retrieves non-default Ceph credentials for the user.

   • Request Body: None

   • Response:

     {
       "success": true,
       "data": [ { "access_key": "...", "secret_key": "...", ... } ],
       "message": "Successfully retrieved and synced non-default Ceph keys"
     }

   • Status Codes:

     • 200: Success
     • 500: Retrieval error

4. Generate Ceph Key

   • Endpoint: POST /api/rgwacc/generate-key

   • Description: Generates a new Ceph access key for the user (max 4).

   • Request Body: None

   • Response:

     {
       "success": true,
       "message": "Successfully generated new Ceph key",
       "data": {
         "accessKey": "string",
         "secretKey": "string",
         "uid": "string"
       }
     }

   • Status Codes:

     • 201: Created
     • 403: Max key threshold reached
     • 500: Failure to generate or store key

5. Delete Ceph User

   • Endpoint: POST /api/rgwacc/delete-user

   • Description: Deletes a Ceph user and associated metadata.

   • Request Body: None

   • Response:

     {
       "message": "Successfully deleted Ceph user {uid} and all associated resources"
     }

   • Status Codes:

     • 200: Deleted
     • 404: User not found
     • 500: Server error

---

Authentication

• All endpoints (except /health) require a Bearer token in the Authorization header.
• The token is validated using JWT_SECRET and must contain userId and organizationId.
• Invalid or missing tokens result in a 401 Unauthorized response.

Request Validation

• Requests are validated using express-validator for required fields, formats, and existence checks (e.g., bucket existence).
• Validation errors return a 400 Bad Request response with details:

  {
    "errors": [{ "msg": "string" }]
  }

Error Handling

• Errors are handled by the errorHandler middleware, returning structured JSON responses:

  {
    "status": "error",
    "error": "string",
    "details": "string",
    "code": "string" (optional)
  }

• Common error types:
  • ValidationError: 400 (e.g., invalid input)
  • MongoError: 400 (e.g., duplicate key)
  • NoSuchBucket: 404
  • NoSuchKey: 404 (file not found)
  • Redis connection error: 503
  • Invalid AWS credentials: 401
  • Generic server errors: 500