Monitoring Consumer Lag

Opening Scenario

It's 2 AM. PagerDuty fires: "checkout-fraud consumer lag > 100K." You SSH in, run kafka-consumer-groups.sh --describe, and confirm — lag is 180K and growing by ~500 records/sec. The consumer process is alive, CPU is at 35%, no exceptions in logs, no recent deploys. Two hours ago lag was zero. Your job is to figure out (a) whether this is an outage or a brief bump, (b) what's actually slow, and (c) whether to wake the on-call engineer for the upstream service. Every minute of debugging is a minute of growing risk that retention deletes unprocessed messages.

---

Question Progression

1. What is consumer lag, exactly?

Question. The dashboard says "lag = 180K." What does that number actually measure?

Reasoning. Consumer lag is log-end-offset - committed-offset, per partition. The log end offset is the offset the broker will assign to the next record in that partition; the committed offset is the offset…