Opening scenario
It's 2 AM. A broker (broker-7) in your 9-broker Kafka cluster crashes after a kernel panic. Within seconds, alerts fire: "Leader changed for 240 partitions." Producers see brief NOT_LEADER_OR_FOLLOWER errors (named NotLeaderForPartition before Kafka 2.6), then recover. Consumer lag spikes on dozens of consumer groups, then drains. Five minutes later, broker-7 reboots, joins the cluster, but you notice it has zero leadership — every partition that was led by broker-7 is now led by another broker, and there's no automatic rebalancing back. By morning, two brokers in the cluster lead 50% of partitions and broker-7 leads 0%.
You need to explain three things to the team: how the new leaders got chosen so fast, why broker-7 doesn't automatically reclaim leadership, and how to think about the resulting hot-spot.
Who decides who the leader is, and how does the decision propagate?
You can guess the shape: some coordinator picks. But what coordinator, what's its input, how does the cluster agree on…
In the full tutorial
- Why is being in the ISR a hard requirement for being elected leader?
- Who elects the controller, and how does that change between ZooKeeper and KRaft?
- What does a producer or consumer actually experience during a leader change?
- How do you cause a graceful leader change, and why would you?
- Closing: When to reach for this