Background Jobs

1. What It Is

A background job is a task the server runs outside of the HTTP request/response cycle. Instead of doing slow or unreliable work synchronously (making the client wait), you enqueue the work, return immediately to the client, and process it asynchronously.

Common examples: sending emails, generating PDFs, resizing images, syncing data to a third-party system, sending push notifications, running batch reports, or charging credit cards after a trial expires.

Why it matters: background jobs are a core reliability and performance tool. Any production system that does work beyond basic CRUD will have them.

2. How It Works in Practice

The Basic Pattern

Client ──POST /orders──▶ API Server
                              │
                              │  1. Save order to DB
                              │  2. Enqueue "send-confirmation-email" job
                              │  3. Return 201 immediately
                              │
                         Job Queue (Redis/SQS/RabbitMQ)
                              │
                         Worker Process
                              │  4. Dequeue job
                              │  5. Send email via SMTP
                              │  6. Acknowledge job (remove from queue)

The API server and the worker are decoupled. The worker can be a separate process or service. If the worker crashes, the job stays in the queue and gets retried.

Components

• Queue: holds pending jobs (Redis, AWS SQS, RabbitMQ, PostgreSQL with a jobs table).
• Producer: the application code that enqueues jobs.
• Worker: a separate process that dequeues and executes jobs.
• Job: a unit of work, usually serialized as JSON: { type: "send-email", payload: { to: "...", templateId: "order-confirm" } }.

Celery (Python) Example

Copy
# tasks.py
from celery import Celery

app = Celery('myapp', broker='redis://localhost:6379/0')

@app.task(bind=True, max_retries=3, default_retry_delay=60)
def send_confirmation_email(self, order_id: int):
    try:
        order = Order.objects.get(id=order_id)
        email_service.send(order.user.email, 'order-confirm', {'order': order})
    except Exception as exc:
        raise self.retry(exc=exc)

# In your API view
send_confirmation_email.delay(order.id)   # enqueue asynchronously

BullMQ (Node.js / TypeScript) Example

Copy
import { Queue, Worker } from 'bullmq';

const emailQueue = new Queue('emails', { connection: { host: 'localhost', port: 6379 } });

// Producer (in API handler)
await emailQueue.add('send-confirmation', { orderId: order.id, userId: user.id });

// Worker (separate process)
const worker = new Worker('emails', async (job) => {
  if (job.name === 'send-confirmation') {
    await emailService.sendConfirmation(job.data.orderId);
  }
}, { connection: { host: 'localhost', port: 6379 } });

3. Common Real-World Usage

Email and notifications: the most common use case. Never send email synchronously in a request handler — SMTP is slow and failure would fail the user's request.

Webhooks and third-party calls: enqueue webhook deliveries and retry on failure rather than dropping them.

Batch processing: scheduled jobs (cron-style) that run reports, charge subscriptions, or clean up stale records.

Media processing: upload an image → return 200 → background job resizes/transcodes and stores the result.

Data sync: replicate data to analytics databases, search indexes (Elasticsearch), or third-party CRMs.

Queuing libraries by ecosystem:

• Python: Celery (Redis/RabbitMQ broker) or Redis Queue (RQ)
• Node.js: BullMQ (Redis-backed)
• Ruby: Sidekiq (Redis-backed)
• Java: Spring Batch, Quartz
• Go: Asynq (Redis-backed), Temporal
• Managed queues: AWS SQS, GCP Cloud Tasks, Azure Service Bus

4. Trade-offs and Common Mistakes

Idempotency

The most important operational property: jobs must be safe to run more than once. Networks fail, workers crash, acknowledgements get lost. The queue will redeliver. If your job runs twice, it must not double-charge a card, double-send an email, or double-insert a row.

Copy
# Idempotent: check before acting
def charge_subscription(user_id):
    if Payment.objects.filter(user_id=user_id, period=current_period).exists():
        return  # already charged, skip
    charge_stripe(user_id)
    Payment.objects.create(user_id=user_id, period=current_period)

Retries and Dead-Letter Queues

Failed jobs should retry with exponential backoff. After N retries, move to a dead-letter queue (DLQ) — a holding area for jobs that consistently fail. Monitor the DLQ and alert on it; don't silently discard failures.

Visibility and Observability

You can't console.log your way to understanding a background job. You need:

• Job status tracking: pending, running, succeeded, failed.
• Logging with context: log the job type, job ID, and relevant entity IDs in every log line.
• Metrics: job throughput, queue depth, processing latency, error rate.
• Alerting: alert on DLQ growth or jobs stuck in running state.

Common Mistakes

Storing too much in the job payload: store only the ID, not the full object. The worker fetches fresh data at execution time. Large payloads bloat the queue and can cause outdated data to be processed.

No acknowledgement-on-completion: if a job is acknowledged before it finishes and the worker crashes mid-execution, the work is lost. Acknowledge after completion (or use at-least-once delivery with idempotent handlers).

Long-running jobs without heartbeats: if a job takes 10 minutes but the queue timeout is 5 minutes, the queue redelivers it while the first run is still in progress. Configure visibility timeout and heartbeat mechanisms correctly.

Mixing real-time and batch in the same queue: a batch job that processes 10,000 records can starve real-time jobs (e.g., email sends). Use separate queues with different priorities.

5. Interview Angle

How to explain it

"Background jobs let you defer slow, unreliable, or non-critical work out of the request cycle. The pattern is: the API enqueues a job and returns immediately; a separate worker process dequeues and executes it. The key operational concerns are idempotency (safe to run twice), retries with backoff, a dead-letter queue for persistent failures, and observability — you need to know what's in the queue and what's failing."

Common interview questions

Q: Why would you use a background job instead of doing work in the request handler? A: Three reasons: performance (the client doesn't wait for slow work), reliability (if the downstream service is down, you can retry without failing the user's request), and decoupling (the API server and the processing worker scale independently).

Q: What is idempotency and why is it critical for background jobs? A: Idempotency means running the same job twice has the same effect as running it once. Critical because queues use at-least-once delivery — a job may be delivered multiple times due to retries or worker crashes. A non-idempotent job (e.g., charge a card) that runs twice causes real-world harm.

Q: What happens when a background job fails? A: It should be retried with exponential backoff. After a configurable number of retries, the job moves to a dead-letter queue (DLQ) so you can inspect it. The failure should be logged and alerted on — not silently dropped.

Q: How would you implement a scheduled/cron job? A: Options: (1) use a cron expression in the job scheduler (Celery Beat, BullMQ's repeat option); (2) use the system cron to trigger a script; (3) use a managed service like AWS EventBridge + Lambda. The job logic itself still runs as a worker task.

Q: How do you ensure a job doesn't run multiple times concurrently? A: Most queues provide exclusive locking at dequeue time (the job is invisible to other workers while one worker holds it). For extra safety: use a distributed lock (Redis SETNX) keyed on the job's entity ID, or design the job to be idempotent so concurrent runs are harmless.

6. Practical Examples

Celery with exponential retry

Copy
@app.task(
    bind=True,
    max_retries=5,
    autoretry_for=(Exception,),
    retry_backoff=True,          # 1s, 2s, 4s, 8s, 16s
    retry_jitter=True,           # adds randomness to avoid thundering herd
)
def process_webhook(self, webhook_id: int):
    webhook = Webhook.objects.get(id=webhook_id)
    response = requests.post(webhook.url, json=webhook.payload, timeout=10)
    response.raise_for_status()
    webhook.mark_delivered()

BullMQ with retry and DLQ (Node.js)

Copy
const queue = new Queue('webhooks', {
  connection: redis,
  defaultJobOptions: {
    attempts: 5,
    backoff: { type: 'exponential', delay: 1000 },
    removeOnComplete: 100,    // keep last 100 completed jobs
    removeOnFail: false,      // keep failed jobs for inspection
  },
});

// Add a job
await queue.add('deliver', { webhookId: 42 });

// Worker
const worker = new Worker('webhooks', async (job) => {
  await deliverWebhook(job.data.webhookId);
}, { connection: redis });

worker.on('failed', (job, err) => {
  logger.error({ jobId: job?.id, err }, 'webhook delivery failed');
});

Idempotency key pattern

Copy
import hashlib

def send_welcome_email(user_id: int):
    idempotency_key = f"welcome-email-{user_id}"
    if redis.set(idempotency_key, 1, ex=86400, nx=True):
        # nx=True: only set if not exists — first runner wins
        email_service.send_welcome(user_id)
    # else: already sent, skip silently