Topics & Partitions

1. What Is It?

A topic is a named, durable stream of messages — e.g., orders, clicks, user-events. A partition is one append-only log within a topic; a topic is split into 1..N partitions. Every message lives in exactly one partition. Partitions are the unit of parallelism, ordering, and replication in Kafka.

The problem this solves: a single log can only be written and read serially. To horizontally scale a topic — both producer throughput and consumer throughput — Kafka splits it into independent partitions that can live on different brokers and be consumed by different workers in parallel. Without partitions, a topic is a single-machine bottleneck.

2. How It Works

1. When you create a topic, you choose the partition count (and replication factor). E.g., kafka-topics --create --topic orders --partitions 12 --replication-factor 3.
2. Kafka spreads partition leaders across brokers (broker 1 leads P0, P3, P6, P9; broker 2 leads P1, ...).
3. Producers decide which partition each record goes to:
   • If the record has a key → partition = hash(key) % numPartitions.
   • If no key → round-robin or sticky (batched) assignment.
   • Or explicit partitioner code.
4. Within a partition, messages are appended in order and assigned monotonically increasing offsets.
5. Consumers within a group are assigned disjoint subsets of partitions; each consumer reads its assigned partitions sequentially.

-+ResetFullscreen

Concrete example. Topic payments with 12 partitions, partitioning by customer_id:

• All events for customer_id=42 always land in the same partition (hash(42) % 12). They are processed in order by whichever consumer is assigned that partition.
• Events for different customers spread across all 12 partitions.
• A consumer group with 4 instances gets 3 partitions each. With 12 instances, each gets exactly 1. With 13 instances, one sits idle.

3. What Mid-Senior SWEs Actually Need to Know

• Partition count caps consumer parallelism. 12 partitions = at most 12 active consumers in a group. Choose partitions for peak future throughput, not current.
• Partition count is sticky. You can increase partitions on a topic but changing the count breaks key-based ordering: a key that used to hash to partition 3 may now hash to partition 9, so its old and new messages live in different partitions and arrive out of order to consumers. Treat partition count as an architectural decision.
• Partitioning key choice is critical.
  • Pick a key with high cardinality to spread load evenly. Partitioning by country_code with 5 unique values → uneven distribution and hot partitions.
  • Pick a key that matches your required ordering unit. If you need per-order ordering, key by order_id; if per-user, key by user_id.
  • No key (or null key) → no ordering guarantees, even within a single producer call.
• Hot partitions are a top operational pain. One key with 1000× the traffic of any other ⇒ one broker overloaded, one consumer lagging. Monitor per-partition byte rate.
• Partition placement and rack awareness. Configure broker.rack so replicas of one partition span racks/AZs. Otherwise an AZ outage can take an entire partition (all replicas) offline.
• Sizing heuristics (orders of magnitude, not laws):
  • Aim for ~tens of MB/sec per partition as a comfortable upper bound; many production topics live well below this.
  • Total partitions per broker in the low thousands is typical; beyond that, controller metadata pressure starts to bite.
  • Each partition has a fixed RAM/file-descriptor cost; don't go to 10,000 partitions on a single topic without reason.
• Common misunderstanding: "I'll just set 1000 partitions and never have to think about it." Cost: longer rebalance times, larger metadata, slower failover, and once your consumer group is bigger than the partitions, you cannot scale further without changing partition count.

4. Tradeoffs & Decisions

Key tradeoff: partition count = parallelism vs metadata cost vs commit weight. Senior engineers pick a count that comfortably handles ~2–4× current peak throughput, with the assumption it won't change again for the life of the topic.

Secondary tradeoff: ordering vs evenness. Sharper ordering guarantees mean choosing a narrower key, which risks skew. Broader keys spread load but weaken ordering. Match the key to the actual ordering unit you need — usually finer-grained than instinct suggests.

5. Interview & System Design Cheat Sheet

• A topic is a logical stream; a partition is a single append-only log within it. Parallelism, ordering, and replication all happen at the partition level.
• Choose the partitioning key from the smallest unit you need ordered — usually the entity ID (user_id, order_id, device_id).
• The partition count is effectively immutable once consumers depend on key-based ordering. Pick for headroom: 2–4× expected peak.
• A consumer group can have at most one consumer per partition — partition count is the ceiling on consumer parallelism.
• Hot partitions are the dominant failure mode in real systems; monitor per-partition throughput, not just topic throughput.

Common follow-ups:

• "What happens if I add partitions to a live topic?" — Existing data stays where it is; new messages with a given key may land in a different partition than before. Per-key ordering across the change boundary is broken.
• "How do you avoid hot partitions for power users?" — Either accept the skew and partition more aggressively, use a composite key (user_id + bucket), or move that tenant to a separate topic.
• "How do you size partitions?" — Estimate peak throughput per topic, divide by a conservative per-partition rate (e.g. tens of MB/sec), then add headroom for growth and parallelism.

If asked to design X, anchor on this: Name the topic, the partition count, the partitioning key (and why — what ordering does it enforce?), and the replication factor. Those four numbers define a topic.