Delivery Semantics

1. What Is It?

Delivery semantics describe how many times a message produced into a streaming system can be observed by a consumer: at-most-once, at-least-once, or effectively-once (often called exactly-once). The choice is forced by the combination of producer acknowledgments, broker durability, consumer offset/commit strategy, and any external side effects the consumer performs.

The problem they solve: distributed systems can fail at any step (producer crash before broker ack, broker crash before replication, consumer crash before commit). Each guarantee defines a different way of trading off duplicates, drops, and complexity. Without an explicit choice, you default to whatever the broker happens to give you and you'll be surprised — usually by duplicates in production after the first failover.

2. How It Works

The three guarantees correspond to where the system handles failure on the producer-to-broker and broker-to-consumer hops:

Kafka producer mechanics:

• acks=0 → at-most-once on the produce hop (no broker ack).
• acks=1 → durability only to the leader; data lost if leader dies before replication.
• acks=all + enable.idempotence=true → no duplicates from retries, no loss on single broker failure. This is the floor for at-least-once on the producer side.
• acks=all + idempotence + transactions (transactional.id) + Kafka-to-Kafka consumer using read_committed → the building blocks for end-to-end exactly-once within Kafka.

Consumer mechanics:

• Auto-commit before processing → at-most-once.
• Commit after processing → at-least-once (duplicates possible if crash between processing and commit).
• Transactional commit alongside the output write → effectively-once.

-+ResetFullscreen

Concrete example. Consumer reads payments, calls Stripe to refund, then commits offset:

• Crash after refund, before commit. On restart, the message is redelivered; Stripe is called again — duplicate refund. To avoid this, either (a) use an idempotency key Stripe deduplicates by, or (b) move the commit into the same transaction as the write — which only works for Kafka-to-Kafka sinks.

3. What Mid-Senior SWEs Actually Need to Know

• Default Kafka behavior is at-least-once. If you do nothing special, you will see duplicates on consumer restarts and rebalances. Design every consumer with that assumption.
• "Exactly-once" is end-to-end and contextual. Kafka EOS (exactly-once semantics) only covers Kafka-to-Kafka (or Kafka → state → Kafka). The moment your consumer calls an external system (HTTP, DB, third-party API), you are responsible for idempotency on that side.
• Idempotency in the sink is usually cheaper than transactions. A unique key (event ID, deterministic primary key, upsert) on the sink turns at-least-once into effectively-once at the system boundary.
• Producer idempotence (enable.idempotence=true) is essentially free. It eliminates duplicates from retries within the same producer session. Default true in modern Kafka. Always on.
• Transactions cost throughput. Transactional producers add ~10–30% overhead and a few ms of latency from the commit protocol. Use them where exactly-once matters; skip them otherwise.
• The order of commit and side-effect determines the semantic. This is the one rule to internalize: commit-before-process = at-most-once; process-before-commit = at-least-once; commit-with-process = effectively-once (and requires a transactional or idempotent sink).
• Common misunderstanding: Turning on Kafka EOS does not make your consumer → REST API call idempotent. EOS is a Kafka-side guarantee; the REST endpoint still sees duplicates unless it dedupes.

4. Tradeoffs & Decisions

Key tradeoff: complexity / throughput vs duplicate-tolerance. At-least-once is operationally simple and fast; you push duplicate handling into the sink, where you usually have natural keys. Transactional exactly-once is heavier and only worth it inside Kafka-native pipelines (e.g. Kafka Streams, Flink → Kafka) where no external system needs deduping.

Secondary tradeoff: scope creep. People say "exactly-once" when they mean "no duplicates in the user-visible output." Almost always, the right design is at-least-once with idempotent writes, not Kafka EOS everywhere.

5. Interview & System Design Cheat Sheet

• The three modes — at-most-once, at-least-once, effectively-once — are determined by the order of side effect and offset commit, not by a single broker setting.
• Kafka's default is at-least-once; producer idempotence + acks=all should always be on; that's the floor.
• "Exactly-once" in Kafka means Kafka EOS — only across Kafka-to-Kafka boundaries. Anything else requires idempotency at the external boundary.
• The dominant production pattern is: at-least-once delivery + idempotent sink writes. It's simpler, faster, and crosses external boundaries cleanly.
• Late and out-of-order events are a separate problem from delivery semantics — don't conflate "may arrive twice" with "may arrive after the window closed."

Common follow-ups:

• "How does Kafka EOS work, at a high level?" — Transactional producer atomically writes data + offsets to Kafka in one transaction; consumers in read_committed mode see only committed messages. Coordinator manages two-phase commit.
• "How would you design idempotent processing for a payment refund?" — Pass an idempotency key (e.g. event ID) to the payment provider; provider deduplicates on its side. Locally, use UPSERT on a refunds(idempotency_key UNIQUE) table.
• "What goes wrong if I set acks=0 to speed up the producer?" — On any broker failure or network glitch you'll silently lose messages. Acceptable for fire-and-forget telemetry, never for business events.

If asked to design X, anchor on this: Default to at-least-once + idempotent sink. Reach for Kafka EOS only when the pipeline is Kafka-native end-to-end and the sink can't be made idempotent another way. Be explicit about which boundary your guarantee applies to — that's the senior-vs-junior distinction.