Opening scenario
You're building a stream processing job that reads from an orders topic, enriches each order with inventory state, and writes to two output topics: order-enriched (for analytics) and inventory-updates (for the warehouse system). The job runs in Flink with a Kafka sink. Today the team treats the pipeline as at-least-once: on failure, the job restarts from the last checkpoint and reprocesses, which means the downstream warehouse system occasionally sees duplicate inventory deductions and the analytics team sees duplicate rows.
The CFO has asked: can we make this exactly-once, end-to-end? You know idempotent producers exist but they don't cross partition boundaries or survive process restarts. You know consumers commit offsets. The question is whether all those pieces can be tied together into "every input is processed once and produces exactly one output per sink, even across crashes."
The answer is Kafka transactions, but understanding when they apply and what they don't cover is the difference between a working system and a misleading SLA.
In the full tutorial
- What does "exactly-once" actually mean in a streaming system?
- How do Kafka transactions atomically span multiple partitions and topics?
- How does the read-process-write loop become exactly-once?
- What does Flink's "exactly-once" sink actually do under the hood?
- What happens when a transactional producer crashes mid-transaction?
- Closing: When to reach for this