Opening scenario
You join a team running a Kafka cluster that's been in production for two years. The on-call rotation tells you: "Don't ever delete files in /var/kafka-logs manually, even if disk is full — call us first." You poke around and find a topic directory with hundreds of files: 00000000000000000000.log, 00000000000000000000.index, 00000000000000000000.timeindex, then more pairs at offsets 12000000, 24000000, and so on. There's also a leader-epoch-checkpoint file. The newest .log file is being written to right now; the older ones haven't been touched in days but disk usage keeps climbing.
You need to understand what's on disk before you can reason about retention, replication, compaction, or recovery. Everything else in Kafka is built on top of this.
Why is the log split into many files instead of one giant file per partition?
You can partially answer this from systems intuition: a single growing file means you can never delete old data without an expensive truncate, and random reads inside a multi-terabyte…
In the full tutorial
- How does a consumer asking for "offset 47,000,123" find the right byte in the file?
- Why are old segments still on disk after the retention period expired?
- What does compaction actually do to these files, and how is it different from deletion?
- When the broker restarts, how does it know the log isn't corrupted?
- Closing: When to reach for this