Streaming Systems

Compaction & Retention

9 min read

Opening scenario

Your team operates two Kafka topics that look superficially similar:

  • user-clicks (90 days of click events for analytics, RF=3, ~2 TB on disk, growing)
  • user-profile-state (current profile per user, used by a notification service that bootstraps state from this topic on startup, RF=3, ~30 GB on disk, stable size despite billions of writes lifetime)

Both are partitioned 64-way. Both have RF=3. Yet they behave totally differently:

  • user-clicks shrinks when retention deletes old data, and the consumer that lags by a day fails with "offset out of range."
  • user-profile-state never grows past 30 GB despite producing billions of records, but a consumer reading from offset 0 sees only the current value per user — old values are gone.

The topics differ in one config: cleanup.policy. user-clicks uses delete (the default), user-profile-state uses compact. Understanding the difference between these two policies — and when to combine them — is foundational to designing topics correctly.

In the full tutorial

  • When should a topic delete old data vs keep the latest per key forever?
  • How does delete-policy retention decide what to drop, and why does my disk sometimes not shrink?
  • What does the log cleaner actually do during compaction?
  • What does combining compact,delete actually do?
  • What's the consumer-side impact of compaction that surprises people?
  • Closing: When to reach for this

Glossary history

Click dotted jargon to save explanations here.