Kafka Anatomy

1. What Is It?

Kafka is a distributed, replicated, append-only commit log packaged as a message broker. Producers append events to topics; topics are split into partitions that live on brokers (the physical Kafka servers); consumers read at their own pace from offsets within partitions. Metadata coordination is handled by a quorum-based controller (KRaft in modern Kafka; ZooKeeper in older deployments).

The problem it solves: most messaging systems delete messages once consumed, so you can't add a new consumer later, debug history, or replay after a bug. Kafka treats the log itself as the source of truth, retains messages for a configured window, and lets many independent consumer groups read at independent positions. That single design choice — log as primitive — is why Kafka became the spine of event-driven backends.

2. How It Works

The key components:

• Broker — a Kafka server process. A cluster is N brokers; production clusters are typically 3 to dozens.
• Topic — a named stream of messages, conceptually a category like orders or clicks.
• Partition — a single append-only log file (actually segment files on disk). A topic is split into P partitions; each partition lives on one broker (the leader) with replicas on other brokers.
• Offset — a monotonically increasing 64-bit integer assigned per-partition. Consumer position = (topic, partition, offset).
• Producer — client library that appends to a partition (chosen by key hash or round-robin).
• Consumer — client library that reads from one or more partitions in order.
• Consumer group — a set of consumers cooperating to share the partitions of subscribed topics.
• Controller — one broker elected to manage cluster metadata (partition leaders, topic creation). In KRaft mode this is a Raft quorum of controller brokers; in ZooKeeper mode it's external.

-+ResetFullscreen

Concrete example. You create topic payments with 12 partitions and replication factor 3 on a 5-broker cluster.

• Kafka assigns 12 partition leaders spread across the 5 brokers (so each broker holds ~2–3 leaders).
• Each partition has 3 replicas (1 leader + 2 followers) on 3 different brokers.
• A producer sending payment_id=ABC hashes that key, gets partition 7, and writes to whatever broker currently leads partition 7.
• A consumer group fraud-scoring with 6 instances ends up with each instance owning 2 partitions; another group analytics with 1 instance owns all 12 — independently.

3. What Mid-Senior SWEs Actually Need to Know

• Partition is the unit of parallelism, ordering, and assignment. You don't get more consumer concurrency than partitions. You don't get ordering across partitions. Choose partition count carefully — easy to add, painful to reduce.
• Replication factor (RF) and min.insync.replicas (min ISR) together set durability. Common safe combo: RF=3, min.insync.replicas=2, acks=all. This survives one broker loss without data loss or unavailability for writes.
• Brokers are stateful and disk-bound. Each broker holds the partitions assigned to it on local disk. Replacing a broker means streaming all its replica data from peers — costly above ~TB. Operationally, treat brokers more like databases than stateless services.
• Topic configs override broker defaults. Retention (retention.ms, retention.bytes), compaction (cleanup.policy), and replication can all be per-topic. Defaults bite you — set them per topic.
• Controller is mission-critical metadata. In KRaft mode, the controller quorum is its own broker set (or co-located). Don't size it too small; outages here mean leader-election and topic-create operations stall.
• Common useful CLI surface:
  • kafka-topics.sh --describe --topic foo — shows partition leaders, replicas, ISR.
  • kafka-consumer-groups.sh --describe --group bar — current offset, end offset, lag. This is your primary operational signal.
  • kafka-configs.sh --entity-type topics --entity-name foo --describe — per-topic overrides.
• Common misunderstanding: "Add more consumers to go faster." Only works up to the partition count. Beyond that, extra consumers sit idle.

4. Tradeoffs & Decisions

Key tradeoff: partition count. Too few → bottlenecked throughput and consumer parallelism. Too many → metadata load on controller, slow rebalances, slow topic operations. Plan for 2–4× current peak throughput, not 100×.

Replication tradeoff: acks=all, min.insync.replicas=2, RF=3 is the standard safe setting. Lowering min.insync.replicas to 1 increases availability during failover but exposes you to data loss; raising it to 3 makes any single broker outage block writes.

5. Interview & System Design Cheat Sheet

• Kafka is a distributed, replicated, append-only log — that primitive (log, not queue) is what unlocks replay, multi-consumer fan-out, and durable history.
• The partition is the unit of ordering, replication, and consumer assignment — almost every Kafka design question reduces to "how do you partition this?"
• Durability is set jointly by RF, min.insync.replicas, and producer acks — never quote one without the others.
• A consumer group can have at most partitions active consumers; you scale consumer parallelism by partitioning more, not by adding consumers.
• Brokers are stateful, disk-heavy services — operationally closer to a database than to a stateless microservice.

Common follow-ups:

• "What's the role of ZooKeeper / KRaft?" — Cluster metadata: which brokers are alive, who leads each partition, topic configs. KRaft replaces ZooKeeper with an internal Raft quorum for the same job.
• "Why is replication factor usually 3?" — Survives one planned outage + one unplanned outage simultaneously while still maintaining a quorum of replicas. RF=2 + min.insync.replicas=2 blocks writes whenever any broker is down.
• "How does Kafka know which broker leads a partition?" — The controller assigns leaders; clients fetch metadata from any broker (it returns the leader map) and then talk directly to the leader broker for that partition.

If asked to design X, anchor on this: Sketch the topic(s), the partition count, the partition key, the replication factor, and the consumer-group layout. That five-element sketch is a Kafka design.