Opening Scenario
You operate two Flink jobs side by side. Job A processes Kafka topics with 5 GB of keyed state and runs at 60K events/sec, all comfortably in JVM heap. Job B does similar work but with 800 GB of state across a long-window sessionization pipeline. A new requirement lands: Job B needs to grow to 1.5 TB of state and survive frequent rescaling. The team's first instinct is "throw more heap at it." You know that won't work — heap-based state hits GC walls long before 1.5 TB. You need to understand what a Flink state backend actually is, why RocksDB is the canonical choice for large state, and where it falls down.
Question Progression
1. What is a state backend and why does Flink need to abstract over it?
Question. Operators in Flink read and write keyed state via ValueState, MapState, ListState. What's actually backing those calls, and why is the implementation pluggable?
In the full tutorial
- When to Reach for This