Streaming Systems

DataStream API

9 min read

Opening Scenario

You inherit a Flink job written entirely in Flink SQL. It works for the original use case — windowed aggregations over a Kafka topic — but a new requirement lands: per-user fraud-risk state that needs custom decay logic, side outputs for flagged transactions, and a backfill mode that reads from S3 in chunks. The SQL planner can't express any of this cleanly. The team lead says "rewrite the new path in DataStream." You've used SQL but never written a DataStream job. You need to understand what DataStream actually is, when SQL is enough, and what the DataStream code structure looks like for a non-trivial keyed-state job.

Question Progression

Question. Flink offers SQL, Table API, DataStream API, and ProcessFunction. They sound like a hierarchy. What's the actual relationship?

Reasoning. Think of it as layers of abstraction,…

In the full tutorial

  • When to Reach for This

Glossary history

Click dotted jargon to save explanations here.