Opening Scenario
You're designing a new Kafka topic for order events at a marketplace company. Traffic projections say 50K events/sec peak. Downstream consumers include: a fraud-detection service that needs per-buyer ordering, a warehouse-routing service that needs per-seller ordering, an analytics pipeline that doesn't care about ordering at all, and an audit log that must preserve total ordering across all events. Your tech lead asks two questions before approving the topic: how many partitions, and what's the partition key. You realize these two answers will lock in the topic's scalability ceiling, its hot-spot behavior under traffic, and the correctness of every consumer for years — and that some of the requirements above directly contradict each other.
Question Progression
1. What does the partition count actually decide?
Question. Why does anyone even ask "how many partitions?" — couldn't Kafka just figure it out?
Reasoning. A partition is a unit of three things at…
In the full tutorial
- When to Reach for This