Opening Scenario
You operate a Kafka consumer group of 24 instances reading a 96-partition topic for a payment-event pipeline. Every time a deploy rolls — even a one-instance restart — the entire group goes silent for 30–45 seconds. Lag spikes from near-zero to 200K, downstream services see a gap in events, and the payments team has filed a ticket asking why deploys keep "breaking" the pipeline. You check logs: no errors, just "Successfully joined group" messages a half-minute apart from each consumer. Nothing's broken, exactly — but the rebalance behavior is unacceptable for the SLA. You need to understand what's happening, why it's so disruptive, and what knobs make it less so.
Question Progression
1. What actually happens during a rebalance?
Question. Twenty-four consumers and 96 partitions, and one of them restarts. What's the chain of events inside the group?
Reasoning. A consumer group has a group coordinator (a specific broker chosen…
In the full tutorial
- When to Reach for This