Opening Scenario
You run a Flink job that counts ad impressions per advertiser in 1-minute tumbling windows, with a 30-second watermark allowance. Reports look correct. Then mobile traffic spikes one Sunday: a flight of users on a flaky cellular network start uploading buffered impression events from 6, 12, 30 minutes ago. The window for 10:00–10:01 closed at 10:01:30 and emitted a count of 14,200 to the reporting database. At 10:08, a buffered batch arrives carrying 1,800 more impressions for that same window. The job neither errors nor stops; the records are silently dropped, and the dashboard is permanently wrong. Product comes asking why mobile counts are 12% lower than the ad network's API confirms. You realize you have a "late data" problem you never explicitly designed for.
Question Progression
1. What does "late" even mean in a streaming pipeline?
Question. Every system has clocks. In what sense is an event late, and late relative…
In the full tutorial
- When to Reach for This