Phase 3 of 7

Background Workers and Job Queue

Add asynchronous processing so expensive tasks do not block user-facing flows.

Overview

Stand up a worker service backed by Redis and database state. APIs should enqueue jobs, track status, and expose outcome visibility.

What to build

Deliverables

Advance only when these outputs exist in your code or compose definitions.

  1. Create a worker service with job processing logic
  2. Integrate Redis-backed queueing between API and worker
  3. Add worker scaling and compose service definition
  4. Add job submission endpoints for report generation
  5. Add worker status and retry/dead-letter handling

Done when

Success criteria

These are acceptance indicators, not a checklist to start from.

  • Worker starts and connects to Redis and database
  • Jobs can be queued and their status can be queried
  • Worker failures are observable and recoverable
  • Worker scale-up with `--scale worker=3` works
  • Jobs are not processed more than once under normal failure recovery

Verification

Testing and validation

Run these in order. Confirm each result before moving to the next step.

  1. docker compose -f docker-compose.dev.yml up -d --scale worker=2

    `docker compose -f docker-compose.dev.yml up -d --scale worker=2`

  2. curl -X POST http://localhost:8000/api/reports/generate -H "Content-Type: application/json" -d '{"report_type":"daily_summary"}'

    `curl -X POST http://localhost:8000/api/reports/generate -H "Content-Type: application/json" -d '{"report_type":"daily_summary"}'`

  3. curl http://localhost:8000/api/jobs/<job_id>

    `curl http://localhost:8000/api/jobs/<job_id>`

  4. docker compose logs -f worker

    `docker compose logs -f worker`

  5. docker compose kill worker

    and confirm replacement worker processes the queue