Opening scenario
A producer team complains their writes are intermittently slow — p99 latency spikes from 5 ms to 800 ms several times a day. They're using acks=all. The Kafka cluster has replication factor 3 and is "healthy" by Datadog dashboards: no broker down, no under-replicated partitions reported. You SSH into a broker and tail the controller log. You see periodic lines like Shrinking ISR for partition orders-7 from [1,2,3] to [1,2] followed seconds later by Expanding ISR for partition orders-7 to [1,2,3]. Producer latency spikes line up exactly with these events.
To explain what's happening, you need a precise model of how Kafka maintains replicas, what "in-sync" means, and how acks=all interacts with the replica set's shape.
What does "the leader replicates writes to followers" actually mean operationally?
You can answer the shape: each partition has one leader broker and N-1 follower brokers. Producers write to the leader. But what do followers do — does…
In the full tutorial
- What does the ISR represent, and what causes a follower to be kicked out?
- How does acks=all plus min.insync.replicas enforce a durability contract?
- What does it mean for a partition to be "under-replicated" vs "offline"?
- What does the leader epoch buy us, and why was it added?
- Closing: When to reach for this