Distributed Message Queue — System Design Interview

Phase 1: Clarify, Scope & Constraints

---

Interviewer: "Let's design a distributed message queue — think Apache Kafka. How do you approach this?"

Candidate: "Message queues are load-bearing infrastructure — the semantics matter deeply. Let me clarify the core user journeys:

1. Produce journey — A producer application sends messages to a named topic; the system durably stores them
2. Consume journey — One or more consumer applications read messages from a topic at their own pace, with offset tracking
3. Fan-out journey — Multiple independent consumer groups each receive their own complete copy of all messages
4. Replay journey — A consumer reprocesses historical messages by seeking to an earlier offset (after a bug fix, for example)

The dominant complexity is durability vs. throughput. A message queue must survive broker failures without losing messages, but must also support very high write throughput. LinkedIn's Kafka benchmark shows 2 million writes/sec on 3 commodity machines. These pull in opposite directions: durability requires writes to multiple replicas before acknowledging; throughput requires fast acknowledgment.

WHY: The secondary complexity is delivery semantics. At-least-once, at-most-once, and exactly-once have profoundly different implementation complexity. Exactly-once requires distributed transactions; at-least-once is achievable with simple acknowledgment schemes. Most systems build at-least-once and push idempotency to consumers.

Non-goals for V1:

• Multi-datacenter geo-replication
• Schema registry and schema evolution
• Stream processing (joins, aggregations) — that is a compute layer on top of the queue
• Dead-letter queue handling

Interviewer: "How does Kafka differ from SQS or RabbitMQ architecturally?"

Candidate: "Three fundamental differences:

1. Log vs. Queue: Kafka is a persistent, append-only…