TaskForge
TaskForge is a scalable background job processing system built with Node.js, Redis, BullMQ, and MongoDB. It demonstrates production-style asynchronous task execution using a queue-based architecture with worker concurrency control, retry strategies, and persistent job tracking.
Overview
TaskForge follows a producer–consumer architecture:
Client → Express API → MongoDB (job record) → BullMQ Queue → Redis → Worker → MongoDB (status update)
The API immediately responds after job creation.
Jobs are processed asynchronously by workers.
Job status is stored and updated in MongoDB.
Redis handles queue storage and job scheduling.
Features
Asynchronous job processing
Multiple job types (email, file upload, image processing, report generation)
Retry mechanism with exponential backoff
Job priority support
Worker concurrency control
Persistent job status tracking
Automatic cleanup of completed and failed jobs
Dockerized Redis and MongoDB setup
RESTful API endpoints
Job statistics aggregation
Tech Stack
Backend
Node.js
Express.js
MongoDB
Mongoose
Queue System
BullMQ
Redis (ioredis client)
Infrastructure
Docker
Docker Compose
Utilities
dotenv
express-rate-limit
uuid
Installation
-
Clone the repository git clone https://github.com/your-username/taskforge.git cd taskforge
-
Install dependencies npm install
-
Configure environment variables
Create a .env file:
NODE_ENV=development PORT=3000
MONGODB_URI=mongodb://localhost:27017/taskforge
REDIS_HOST=localhost REDIS_PORT=6379
MAX_CONCURRENT_JOBS=5 JOB_TIMEOUT=60000
RATE_LIMIT_WINDOW=15 RATE_LIMIT_MAX_REQUESTS=100
Running the Project Start MongoDB and Redis (Docker) docker-compose up -d
Verify containers:
docker ps
Start the API server npm run dev
Start the worker (in a separate terminal) node src/workers/worker.js
The worker must run independently from the API server.
API Endpoints Health Check
GET
/health
Create Job
POST
/api/jobs
Example request body:
{ "type": "email", "data": { "to": "user@example.com", "subject": "Welcome", "body": "Hello from TaskForge" }, "priority": 1 }
Get Job by ID
GET
/api/jobs/:jobId
Get All Jobs
GET
/api/jobs
Query parameters:
status
type
limit
skip
Get Job Statistics
GET
/api/jobs/stats
Example response:
{ "success": true, "data": { "pending": 0, "processing": 0, "completed": 1, "failed": 0 } }
Reliability Mechanisms
3 retry attempts per job
Exponential backoff (2s → 4s → 8s)
Automatic cleanup of old jobs
Persistent job status updates
Configurable worker concurrency
Future Improvements
Authentication and user-based job isolation
Queue dashboard (Bull Board)
Delayed and scheduled jobs
Dead-letter queue
Cloud deployment
Metrics and monitoring integration