Why slow work needs its own pattern
The Managing Long-Running Tasks pattern splits API requests into two phases: immediate acknowledgment and background processing. When users submit heavy tasks like video encoding, the web server validates the request, pushes a job to a queue, and returns a job ID — all within milliseconds. Separate worker processes poll the queue, execute the time-consuming work, and update job status in a database.
This is the deep dive behind Common patterns. Pair with Handling large blobs (upload then transcode), Real-time updates (progress %), Message queues (delivery semantics), and Multi-step processes (job dependencies).
Return job ID fast.
Durable buffer.
Pull and process.
Poll or push status.
The problem
Imagine a simple profile page: a quick database query, format, respond — under 100 milliseconds. Life is good.
Now generate a PDF report of annual activity: query multiple tables, aggregate millions of rows, render charts, produce a document — at least 45 seconds.
With synchronous processing, the browser sits waiting. Most load balancers timeout around 30–60 seconds. Even if it completes, UX is poor — no progress, no feedback. Users assume failure and click again, creating duplicate work.
- Video uploads — transcoding takes minutes
- Profile photos — resize, crop, multiple thumbnails
- Bulk ops — newsletters, CSV imports
- Each far exceeds what users will reasonably wait for
Async worker pool architecture
Decouple request acceptance from processing. Web servers become lightweight routers — validate, enqueue, return job ID. A separate worker pool handles heavy lifting at its own pace.
The queue is a durable buffer. Popular choices: Redis with Bull/BullMQ, AWS SQS, RabbitMQ, or Kafka at higher scale. It gives visibility into pending jobs, processing time, and failures.
UX improves dramatically: immediate confirmation, optional queue position, notification via email, push, or WebSocket when complete. Users navigate away and come back when work is done.
Trade-offs
Managing long-running tasks isn't magic. You gain fast responses and independent scaling; you pay in complexity.
What you gain
- Fast response — API returns in ms, not 30s timeouts
- Independent scaling — web and workers scale separately
- Fault isolation — one worker crash doesn't take down the API
- Better resource use — GPU workers on GPU boxes; web on cheap instances
What you lose
- Complexity — queues, workers, status tracking
- Eventual consistency — work isn't done when API returns
- Job status infra — DB load, status endpoints, retries
- Monitoring — queue depth, worker health, failure rates
How to implement
Two core technologies: a message queue and a worker pool.
Message queue
Must be durable, handle concurrent workers without duplicating work. Interview defaults:
- Redis + Bull/BullMQ — startup default; simple retries and priority; memory-first durability
- AWS SQS — managed; pass job IDs (1MB limit); pay per message
- RabbitMQ — complex routing; self-hosted ops
- Kafka — replay, fan-out, ordering; when already in your stack
Workers
- Normal servers — 10–20 processes in a loop; full control; SSH debugging; idle capacity cost
- Serverless (Lambda) — auto-scale; 15–60 min limits; cold starts; great for spiky workloads
- Containers (K8s/ECS) — middle ground; long jobs + orchestrator scaling
# Simplified worker loop
while True:
job = queue.pop() # blocks until available
if job:
process_job(job)
mark_complete(job.id)Putting it together
- Web server validates and creates job record (
pending) - Push message with job ID only — not full payload
- Return job ID to client immediately
- Worker pulls message, fetches job details from DB
- Update status to
processing - Perform work; store results (S3, DB)
- Update status to
completedorfailed
When to use in interviews
Don't wait for the interviewer to ask. Recognize signals and suggest async proactively.
- Slow operations named — "video transcoding", "PDF generation", "bulk emails" → return job ID immediately
- Math doesn't work — "1M images/day × 10s each = 120s processing per second of wall clock" → dedicated workers
- Mixed hardware — GPU video + login API → separate tiers
- Scale/failure questions — "worker crashes mid-job?" → another worker picks up from queue
Interview scenarios
- YouTube — transcode 1080p/720p/480p, thumbnails, captions, moderation — hours of worker time per video
- Instagram — multiple sizes, filters, metadata, content policy; fan-out to follower feeds
- Uber — ride matching async; "Finding drivers…" while workers evaluate routes and pricing
- Stripe — fraud detection, 3DS, webhooks async; bank transfers return pending
- Dropbox — virus scan, search indexing, previews after upload returns
When NOT to use
Login, payment authorization the user is staring at, anything under ~1–2 seconds where poll UX is worse than waiting. Sync is simpler and often correct.
Common deep dives
Handling failures
Worker crashes mid-job? Another worker retries. Use heartbeat/visibility timeout so the queue knows the worker died. SQS visibility timeout, RabbitMQ heartbeat, Kafka session timeout. Start with 10–30 seconds — long enough to avoid false positives from GC pauses.
Repeated failures — DLQ
Poison messages crash workers forever without a Dead Letter Queue. After 3–5 failures, move to DLQ for human investigation. Monitor DLQ growth — it usually signals a bug.
Preventing duplicate work — idempotency
User clicks "Generate Report" three times? Use idempotency keys. Check if job exists before creating. Make the work itself idempotent — safe to retry halfway through.
def submit_job(user_id, job_type, job_data, idempotency_key):
existing = db.get_job_by_key(idempotency_key)
if existing:
return existing.id
job_id = create_job(user_id, job_type, job_data)
db.store_idempotency_key(idempotency_key, job_id)
queue.push(job_id)
return job_id
Queue backpressure
Black Friday 10× jobs? Queue grows to millions. Set depth limits — return "system busy" instead of accepting work you can't handle. Autoscale workers on queue depth, not CPU. By the time CPU is high, you're already backed up.
Mixed workloads
5-second PDFs and 5-hour year-end exports in one queue? Long jobs block short ones. Separate fast and slow queues with different worker counts and instance types.
queues:
fast:
max_duration: 60s
worker_count: 50
instance_type: t3.medium
slow:
max_duration: 6h
worker_count: 10
instance_type: c5.xlargeJob dependencies
Fetch data → generate PDF → email? Simple chains: each worker queues the next step with full context. Complex branching: use Step Functions, Temporal, or Airflow — see Multi-step processes.
{
"workflow_id": "report_123",
"step": "generate_pdf",
"previous_steps": ["fetch_data"],
"context": {
"user_id": 456,
"data_s3_url": "s3://bucket/data.json"
}
}Cost and performance levers
Interview Q&A by level
Practice saying these out loud for long-running tasks. Interviewers grade clarity and judgment more than buzzwords.
Match depth to the bar: define → trade off → operate. Don't dump principal answers in an entry-level screen.
Conclusion
Managing long-running tasks shows up constantly in interviews. When you hear video uploads, PDF generation, or bulk processing — think queue, job ID, async workers.
Be proactive: say "that's too slow for a synchronous request" before they ask. Pick a queue you're comfortable with (Kafka is a safe default), know the tradeoffs, and handle DLQ, idempotency, and backpressure deep dives.
Ack fast with a job ID; process in workers — but only queue when the work is truly too slow for the HTTP request.