Consumer Groups

1. What Is It?

A consumer group is a set of consumers that cooperate to read from the same topic(s), where each partition is read by exactly one consumer in the group at a time. Different consumer groups read the same topic independently — each maintains its own position. The consumer group is how Kafka delivers both queue semantics (within a group) and pub/sub semantics (across groups) over the same log primitive.

The problem it solves: a single consumer process cannot keep up with a high-throughput topic. You need to fan out the work across multiple processes while preserving per-partition ordering and exactly-one-owner semantics. And independently, you want multiple unrelated systems (analytics, billing, fraud) to each read every event without coordinating. Consumer groups handle both with one mechanism.

2. How It Works

1. Each consumer specifies group.id (e.g., fraud-scoring). All consumers sharing that ID form one group.
2. One broker is selected as the group coordinator for the group, responsible for membership and offset commits.
3. When members join or leave, the coordinator triggers a rebalance — the partition assignment is recomputed and distributed to members. Different assignors (range, round-robin, cooperative sticky — modern default) choose how.
4. Each member then independently reads its assigned partitions, processes records, and commits offsets to a special internal topic __consumer_offsets (keyed by group.id + topic + partition).
5. If a member fails its heartbeat (session.timeout.ms) or doesn't poll within max.poll.interval.ms, the coordinator kicks it out and rebalances.

-+ResetFullscreen

Concrete example. Topic orders with 6 partitions, two consumer groups:

• fraud-scoring with 3 instances → each gets 2 partitions. If a 4th instance joins, assignment becomes 6/4 ≈ 1–2 per consumer. If a 7th joins, one sits idle.
• analytics with 1 instance → reads all 6 partitions independently. Its offset progress has nothing to do with fraud-scoring's offset progress.

The two groups don't know about each other. Adding a new group email-receipts later doesn't affect the existing two; it just starts reading from auto.offset.reset (earliest or latest, depending on config).

3. What Mid-Senior SWEs Actually Need to Know

• Partition count is the ceiling on group concurrency. A 6-partition topic supports at most 6 active consumers per group. Adding a 7th gives you idle capacity, not more throughput. Plan partition count for peak consumer parallelism.
• Group ID identifies progress. Two consumers with the same group.id share work and share offsets. Two with different group.ids each read everything. Never accidentally reuse a group.id between unrelated apps — they'll fight over partitions.
• Rebalances pause everyone in the group. During a rebalance, all members stop processing while the coordinator reassigns. With the older eager assignment, every member gives up all partitions, then re-acquires. With cooperative sticky (modern default), only moved partitions are revoked — much smoother.
• Rebalances are triggered by: members joining or leaving, partition count changing, topic subscriptions changing, member failing heartbeat or max.poll.interval.ms.
• Slow processing → rebalance storm. The most common production pathology: one consumer takes too long in processBatch(), misses max.poll.interval.ms, gets kicked, group rebalances, lag spikes, work isn't drained, the next consumer also misses the deadline. Fix: speed up processing, lower max.poll.records, or move slow work off the poll thread.
• Offset commits are per-group, per-partition. They live in __consumer_offsets. Deleting a group ID (or changing it) loses progress — the new group starts from auto.offset.reset.
• auto.offset.reset only applies when there's no committed offset. Once you've committed even one offset, future restarts resume from there, ignoring auto.offset.reset.
• Static membership (group.instance.id) avoids rebalances on rolling restart. A consumer that disappears briefly (e.g., during a deploy) keeps its assignment until session.timeout.ms * a long timeout, so a fast restart resumes the same partitions. Worth using for stateful consumers.
• Common misunderstanding: "I'll just add more consumers to scale." Works until you hit the partition count. Real scaling = more partitions and more consumers, and partition count is hard to change later.

Useful CLI:

• kafka-consumer-groups.sh --describe --group fraud-scoring — shows current offset, log end, lag, and which consumer owns which partition.
• kafka-consumer-groups.sh --reset-offsets --group X --to-earliest --topic Y --execute — operational tool for replays (use carefully — only when group has no active members).

4. Tradeoffs & Decisions

Key tradeoff: rebalance disruption vs failure detection speed. Aggressive timeouts catch dead consumers fast but rebalance frequently on transient slowdowns. Cooperative sticky + static membership + reasonable timeouts (5–10s heartbeat, several minutes max.poll.interval.ms) is the sane default.

Secondary tradeoff: explicit commit vs auto-commit. Auto-commit is simpler but breaks at-least-once on a crash mid-batch. Explicit commit after processing is the correct default for production; auto-commit is fine for "metrics consumer where I don't care about a few lost records."

5. Interview & System Design Cheat Sheet

• A consumer group is the unit of cooperative consumption: one partition → one consumer in the group, but unlimited groups can read the same topic independently.
• Partition count is the ceiling on group concurrency — choose it for peak consumer parallelism, not current.
• Rebalances pause everyone — minimize them via cooperative sticky assignor, static membership, and keeping processBatch under max.poll.interval.ms.
• Offset commits are per-group, per-partition, stored in __consumer_offsets. Group ID changes = lost progress.
• The same Kafka topic delivers queue semantics within a group and pub/sub semantics across groups — that's the entire model.

Common follow-ups:

• "What's the difference between an eager and cooperative rebalance?" — Eager: all members release all partitions, then everyone re-acquires (long stop-the-world). Cooperative (incremental sticky): only moving partitions are revoked, others keep processing. Modern default is cooperative.
• "How would you replay data for a new feature?" — Either spin up a new consumer group with auto.offset.reset=earliest, or use kafka-consumer-groups to reset offsets on the existing group (after stopping it).
• "My consumer lag is spiking — what's going on?" — Check (a) processing throughput per partition, (b) rebalance rate, (c) whether one specific partition is hot, (d) whether max.poll.interval.ms is being exceeded. Lag spikes from rebalances look very different from lag spikes from slow processing — diagnose by metric.

If asked to design X, anchor on this: Identify which systems are independent (separate group.ids) and within each, how many consumer instances you need vs how many partitions the topic has. The answer to "how do we scale this consumer" is almost always "more partitions and more consumers — but first verify processing isn't the bottleneck."