Opening Scenario
You run a payments service that emits two Kafka topics: payment_initiated (when a checkout starts) and payment_authorized (when the gateway approves). Product asks for a real-time dashboard showing time-to-authorize per payment — the duration between the two events. You wire up a Flink job that reads both topics keyed by payment_id and joins them. In a load test it produces correct numbers. The next day in staging, half the dashboard rows are missing, and the job's RocksDB state size is climbing past 200 GB. You're not sure if you have a correctness bug, a state bug, or a watermark bug — and you're starting to suspect "join two streams" is hiding a lot.
Question Progression
1. Why is joining two streams fundamentally different from joining two tables?
Question. In SQL you write SELECT * FROM initiated i JOIN authorized a ON i.id = a.id. Both sides are finite, indexed, sitting still. A streaming engine reading those same two topics can't run that query directly.…
In the full tutorial
- When to Reach for This