Opening scenario
Your team's Flink job tracks per-user running totals: every transaction event updates a MapState<UserId, Double> keyed by user. The job runs fine for two weeks, then memory pressure climbs steadily, eventually OOMing. You add more task managers; the OOM recurs a week later, just at a higher data volume. You enable RocksDB state backend instead of heap; latency goes up but the job stops crashing. Now you notice checkpoint duration has climbed from 200 ms to 45 seconds, and a savepoint operation last week took 12 minutes. You're not sure if the latency is from RocksDB itself, from checkpoint snapshotting, or from state size — and you don't have a model for why moving state to disk changed things so dramatically.
Stream processors are fundamentally stateful: every aggregation, join, windowed computation, deduplication, and pattern detection holds state per key. How you manage that state determines latency, throughput, fault tolerance, and what queries you can express. Understanding the model is what separates running streaming jobs from operating them.
In the full tutorial
- Why does stream processing need first-class state, and why can't I just use a database?
- What are the types of state you'll actually use, and how do they differ operationally?
- What does the state backend actually decide?
- How do checkpoints make state durable, and what determines checkpoint duration?
- What does state expiration buy us, and what does it not?
- How does savepoint differ from checkpoint, and when do you use each?
- Sample stateful operator (Flink DataStream API)
- Closing: When to reach for this