Message Queues & Async Writes (Kafka, SQS, RabbitMQ)

Tier 1 — Building Block

1. What Is It?

A message queue is a durable, ordered buffer that decouples the producer of work (the service that generates a message) from the consumer (the service that processes it). Producers write messages to the queue without waiting for processing to complete; consumers read and process messages independently at their own pace.

Without message queues, every service call is synchronous and tightly coupled: if Service B is slow, Service A blocks and its latency increases. If Service B crashes, Service A's request fails. If there's a burst of 100K requests in one second but Service B can only process 10K/sec, the excess is dropped. Message queues solve all three problems: they absorb bursts (buffering), decouple failure domains (if consumer is down, messages accumulate in the queue until it recovers), and enable async processing (producer gets an immediate acknowledgment; processing happens later).

2. How It Works

Core Model

A topic (or queue) is a named channel that messages are published to. Producers send messages to a topic; consumers read from it. Topics decouple producers from consumers — they don't need to know about each other.

-+ResetFullscreen

Message Lifecycle

-+ResetFullscreen

Kafka Architecture (Log-Based)

Kafka is fundamentally different from traditional queues: it is an immutable, append-only distributed log. Messages are never deleted on consumption — they are retained for a configurable period (e.g., 7 days). Multiple consumer groups can read the same topic independently, each maintaining their own offset.

-+ResetFullscreen

Key Kafka concepts:

• Partition: The unit of parallelism. A topic's data is split across partitions. One consumer per partition per consumer group (parallelism = partition count).
• Offset: A message's sequential position within a partition. Consumers commit their offset after processing. On restart, consumption resumes from the committed offset.
• Consumer Group: A logical set of consumers that collectively consume a topic. Each partition is assigned to one consumer within the group. Multiple consumer groups consume the same topic independently.
• Replication: Each partition has 1 leader + N-1 followers. Producers write to the leader; followers replicate. If the leader dies, a follower is elected leader.

3. Variants & Comparisons

Delivery Semantics

In practice: at-least-once + idempotent consumer is the most common production pattern. Exactly-once is expensive to implement correctly and rarely necessary if consumers are idempotent.

4. When to Use It (and When NOT To)

Use Message Queues When:

• Write throughput spikes: The queue absorbs burst traffic; consumers process at their own pace. Order placement, payment initiation, file processing — any operation where the producer is faster than the consumer.
• Decoupling services: Service A should not care if Service B is temporarily down. A queue buffers the messages until B recovers.
• Async processing: Send email, resize image, generate report — work that doesn't need to complete synchronously within the user-facing request.
• Fan-out: One event needs to trigger multiple consumers (e.g., "order placed" → payment service + inventory service + email service). Kafka topics enable one event to feed multiple consumer groups.
• Audit log / event sourcing: Kafka's immutable log is a perfect audit trail; replay events to rebuild state.

Decision triggers:

• "If a downstream service is slower than upstream and can't be sped up → add a queue to absorb the mismatch"
• "If a crash of Service B would cause Service A's requests to fail → add a queue"
• "If one event needs to trigger multiple independent services → use Kafka topic with multiple consumer groups"

Do NOT Use Message Queues When:

• Low latency is required: A queue adds ~10ms+ latency. For P99 < 5ms requirements (financial trading), direct RPC is better.
• Request-response pattern: If the caller needs the result synchronously, a queue is the wrong abstraction. Use gRPC or REST.
• Simple in-process communication: Don't over-engineer. A function call is faster and simpler than a queue for intra-service communication.
• You need strong consistency within a transaction: "Publish message AND update database atomically" is hard — requires transactional outbox pattern. If you can't tolerate the complexity, keep operations synchronous.

Anti-patterns:

• Ignoring backpressure: If consumers can't keep up with producers, the queue grows without bound. Monitor queue depth and alert when it exceeds a threshold. If queue depth grows persistently, add consumer instances or shard the queue.
• Non-idempotent consumers with at-least-once delivery: Messages WILL be delivered more than once (network retries, consumer crashes after processing but before ACK). If your consumer doubles charges, sends duplicate emails, etc. — you have a bug. Always design consumers to be idempotent.
• No dead letter queue (DLQ): If a consumer crashes processing a specific message N times, without a DLQ the message loops forever (poison pill), blocking the rest of the queue. Always configure a DLQ.
• Using Kafka as a pure database: Kafka is a log, not a queryable database. "Find all orders for user X" is a scatter-gather across all partitions — use a real database for those queries.

5. Real-World Usage

LinkedIn (Kafka origin): Kafka was created at LinkedIn in 2010 to handle the activity feed pipeline — every page view, click, and profile update was an event. LinkedIn needed multiple downstream systems (search index, recommendations, ads targeting, analytics) to consume the same event stream independently. Traditional queues deleted messages on consume; Kafka's log-based model let each system replay from any offset. LinkedIn now processes over 7 trillion messages per day across their Kafka clusters.

Uber (Kafka + Flink for real-time pricing): Uber uses Kafka as the backbone for real-time dynamic pricing. Every GPS ping from every driver and rider (~1M messages/sec peak) flows through Kafka topics. A Flink stream processing layer consumes these topics to compute surge pricing, estimated arrival times, and supply/demand ratios in real time. The immutability of the Kafka log means Uber can replay events to debug pricing anomalies or test new pricing models against historical data.

Amazon (SQS for decoupling): Amazon's "Building Microservices" architecture uses SQS extensively to decouple services. The canonical example: when a customer places an order, the order service publishes an OrderPlaced event to an SQS queue. The inventory service, fulfillment service, and email service each have separate queues triggered by the event (via SNS fan-out to multiple SQS queues). If the email service is down during a deploy, orders continue processing and email notifications catch up when the service restarts.

6. Interview Cheat Sheet

Key sentences to demonstrate depth:

1. "Message queues decouple producers from consumers in three dimensions: time (process asynchronously), rate (absorb burst traffic), and availability (consumer downtime doesn't fail the producer)."
2. "Kafka's key insight: messages are never deleted on consume — it's a durable log. Multiple consumer groups independently consume the same topic, each replaying from their own offset. This is why Kafka is used for fan-out to multiple services."
3. "At-least-once delivery is the practical default — messages may be redelivered on failure. Design all consumers to be idempotent: processing the same message twice produces the same result as processing it once."
4. "Kafka throughput scales with partition count: consumers = partition count per consumer group. If you need N× throughput, add N partitions. But you can't reduce partitions after the fact — plan partition count with growth in mind."
5. "The transactional outbox pattern solves the 'publish and write DB atomically' problem: write the event to an outbox table in the same transaction as the DB write, then a separate process tails the outbox table and publishes to Kafka. This prevents the 'write succeeded but publish failed' split-brain."

Common follow-up questions:

Connections to other building blocks:

• WAL & LSM Trees: Kafka's log storage is LSM-like: sequential append-only writes with periodic compaction (log compaction for event sourcing use cases). Kafka's write durability is a WAL.
• Sharding: Kafka partitions are sharding of the log stream. The partition key determines which shard (partition) a message goes to, providing ordering within a shard.
• Rate Limiter: Message queues naturally implement rate limiting by controlling consumer processing rate. The queue absorbs traffic spikes; consumers process at a controlled pace.
• Unique ID Generator: Messages in Kafka are identified by (topic, partition, offset). For idempotency, producers often include a message ID (Snowflake or UUID) in the payload so consumers can deduplicate.
• CDN / Cache: A message queue is conceptually the write-path complement to a cache (which handles the read path). Writes go to queue (async processing); reads go to cache.