Brokers & Replication

1. What Is It?

A broker is one Kafka server process, holding partitions on local disk. Replication is Kafka's mechanism for keeping multiple copies of each partition across different brokers so that the loss of one broker (disk, host, AZ) doesn't lose data or stop the cluster. Each partition has one leader broker (handles reads and writes) and N-1 follower brokers that keep in-sync replicas.

The problem this solves: any single disk or machine eventually fails. Without replication, a broker outage means losing every partition assigned to it — both for reads and writes — and likely losing data. Replication turns single-machine durability into cluster-level durability; the cost is more disks, more network, and a more careful write path.

2. How It Works

Roles

• Leader of a partition: handles all client produce and fetch requests for that partition. Each partition has exactly one leader at a time.
• Followers: continuously fetch from the leader and write the data to their own logs.
• In-Sync Replicas (ISR): the set of replicas that are "caught up" — currently within replica.lag.time.max.ms of the leader. The leader is always in the ISR.

Write path with acks=all

1. Producer sends the record to the partition leader.
2. Leader appends to its local log.
3. All followers in the ISR fetch the record and append to their logs.
4. Once every ISR follower has written the record, the leader advances the high watermark and acknowledges the producer.
5. Consumers can only read up to the high watermark — uncommitted records are invisible.

Leader failure

1. If the current leader fails (no longer in ISR or fails health check), the controller promotes one of the remaining in-sync replicas to leader.
2. Producers and consumers fetch fresh metadata, retry against the new leader.
3. Failover typically takes seconds (network heartbeat + controller decision + metadata propagation). Brief produce/fetch errors are normal.

min.insync.replicas

• Topic-level setting: minimum ISR size required for an acks=all produce to succeed.
• If too few replicas are in-sync, the leader returns NotEnoughReplicasException and the producer cannot write (preserving durability over availability).

-+ResetFullscreen

Concrete example. Topic orders with replication factor 3, min.insync.replicas=2, acks=all, on a 3-broker cluster spanning 3 AZs:

• Steady state: Every partition has 3 replicas (1 leader + 2 followers), all in-sync.
• One broker dies (e.g., AZ outage): ISR drops to 2. Writes still succeed because min.insync.replicas=2 is met. Affected partitions whose leader was on the dead broker fail over to one of the remaining replicas. Brief produce/fetch errors during failover.
• Two brokers die: ISR drops to 1. Writes fail (NotEnoughReplicasException) because min.insync.replicas=2 is not met. The cluster preserves durability — you'd rather block writes than lose data.
• Dead broker comes back: Follower catches up by re-fetching from the leader; once within replica.lag.time.max.ms, it rejoins the ISR.

3. What Mid-Senior SWEs Actually Need to Know

• The standard safe combo is RF=3, min.insync.replicas=2, acks=all. This tolerates one broker failure without data loss or write outage. Anything weaker risks losing acknowledged messages during failover.
• acks=all does not mean "wait for all replicas" — it means "wait for all in-sync replicas." That's why min.insync.replicas exists: it prevents acks=all from silently downgrading to acks=1 when the ISR has shrunk.
• Unclean leader election is a footgun. With unclean.leader.election.enable=true, an out-of-sync replica can become leader after a worst-case failure — preserving availability at the cost of losing acknowledged messages. Default is false; keep it false unless you understand the tradeoff (rarely worth it).
• Rack awareness must be configured. Set broker.rack on each broker (e.g., to AZ). Otherwise Kafka may place all 3 replicas of a partition in the same AZ, and a single AZ outage takes the partition fully offline.
• Replication is network and disk heavy. Each produce writes 1 leader + (RF-1) followers — total cluster write amplification is RF. Plan inter-broker network bandwidth and disk IOPS accordingly.
• A shrinking ISR is a leading indicator of trouble. Use kafka-topics --describe and the JMX metric UnderReplicatedPartitions — if it's nonzero, something is wrong (slow disk, network saturation, GC pause, broker overloaded).
• Broker storage is local-disk by design. Adding/removing a broker streams partition data across the network, which can take hours for large clusters. Replacements are not zero-cost; plan capacity for replication catch-up.
• Common misunderstanding: "If RF=3 I can lose 2 brokers and keep writing." Only if you set min.insync.replicas=1 — which means you might lose acknowledged data if the lone surviving broker dies before catching up. Most teams want min.insync.replicas=2 and accept that 2 simultaneous broker losses block writes.

4. Tradeoffs & Decisions

Key tradeoff: durability vs availability vs cost. More replicas survive more failures but cost more disk and network. Higher min.insync.replicas protects data but blocks writes earlier when replicas are unhealthy. There is no single right answer — match the topic's business value.

Secondary tradeoff: fast failover vs stable leadership. Aggressive failure detection (low replica.lag.time.max.ms) catches dead followers fast but flaps ISR membership under load spikes. The defaults are reasonable; tune only with evidence.

5. Interview & System Design Cheat Sheet

• A broker is one Kafka server; a cluster is N brokers. Each partition has one leader (serves reads/writes) and N-1 followers (replicate).
• In-Sync Replicas (ISR) = the set of replicas currently caught up; min.insync.replicas defines how many must be in the ISR for acks=all produces to succeed.
• Standard safe setting: RF=3, min.insync.replicas=2, acks=all, unclean.leader.election.enable=false, with rack awareness spanning AZs.
• Leader failure triggers controller-driven failover in seconds; brief produce/fetch errors are expected and clients retry transparently.
• UnderReplicatedPartitions is the canary metric — nonzero means a follower is behind, which is the first symptom of disk saturation, network issues, or broker overload.

Common follow-ups:

• "What's the difference between acks=all and min.insync.replicas?" — acks=all is the producer asking the leader to wait for all ISR followers. min.insync.replicas is the leader's check that the ISR is large enough to honor that contract; without it, acks=all silently weakens when followers fall out of sync.
• "What does unclean leader election cost?" — Availability win after worst-case failure (no in-sync replicas left) at the cost of losing acknowledged messages. Disable unless you have explicitly chosen availability over correctness.
• "How do you tune for inter-AZ deployments?" — broker.rack for replica placement; tune replica.fetch.max.bytes and socket.receive.buffer.bytes for higher RTTs; expect ~ms-level cross-AZ replication latency.

If asked to design X, anchor on this: State the replication factor, the min.insync.replicas, the acks setting, whether you're rack-aware, and your DR posture (single-region with multi-AZ replication, or multi-region with MM2 / Cluster Linking). Those five answers are the durability story.