Backpressure

Opening Scenario

You run a Flink job that reads from Kafka, enriches each event with a Redis lookup, and writes to a downstream Kafka topic. Throughput targets are 30K events/sec. In production, the consumer lag on the source topic grows by 100K records per minute despite the job appearing "healthy" — no exceptions, no restarts, CPU only at 40%. The Flink UI shows several operators flagged as "backpressured." Your manager wants to know whether to scale up, scale out, or fix something — and what's actually being backpressured by what.

---

Question Progression

1. What does it mean for an operator to be "backpressured"?

Question. The Flink UI says an operator is backpressured. What is physically happening inside the pipeline at that moment?

Reasoning. A streaming pipeline is a chain of operators connected by bounded buffers. The source reads from Kafka and hands records to the next operator via a buffer. That…