Pub/Sub vs Queue

1. What Is It?

A queue is a delivery channel where each message is consumed by exactly one worker — multiple workers compete for messages and share the load. A pub/sub (publish/subscribe) channel is one where each message is delivered to every subscriber — multiple consumers each get their own copy. Kafka's "consumer groups" express both patterns within the same system: workers in the same group share the load (queue behavior), workers in different groups each see every message (pub/sub behavior).

The problem they solve: how to move work or facts between processes asynchronously. Without queues, you can't horizontally scale a worker pool against a backlog. Without pub/sub, you can't fan out the same event to many independent consumers without the producer knowing each one.

2. How It Works

Queue (work distribution):

1. Producer writes a message to the queue.
2. The broker delivers the message to one of N competing workers.
3. The worker acks; the broker deletes (or hides) the message.
4. If the worker fails before acking, the message becomes visible again and another worker picks it up.

Pub/sub (broadcast):

1. Producer publishes a message to a topic.
2. The broker delivers a copy to every subscriber.
3. Each subscriber tracks its own progress independently.
4. A new subscriber may (or may not) see historical messages, depending on the broker.

Kafka unifies both via consumer groups. Partitions in a topic are distributed across the consumers of a single group (queue semantics within the group). Multiple groups each independently consume every partition (pub/sub across groups).

Concrete example. An image-upload event needs two things to happen: generate a thumbnail (one worker is enough), and update analytics (separate system).

• Queue: A thumbnail-jobs SQS queue with 20 worker processes. Each upload produces one job; one worker handles it.
• Pub/sub: An uploads SNS topic. The thumbnail service subscribes; the analytics service subscribes. Each gets every event.
• Kafka unified: Topic uploads with consumer group thumbnailer (20 instances sharing partitions) and consumer group analytics (separately reading every event).

3. What Mid-Senior SWEs Actually Need to Know

• Queue semantics need ack/nack and visibility timeouts. SQS, RabbitMQ: a message is "in flight" while a worker holds it; if not acked within the timeout, it reappears. Setting the timeout too low causes duplicates; too high causes stuck messages on crashed workers.
• Pub/sub without persistence loses messages. Redis pub/sub delivers only to subscribers connected at publish time. SNS drops a message once its delivery retries run out (unless a dead-letter queue is attached). A consumer that was down too long misses events forever. Modern pub/sub (Kafka, Pulsar, Kinesis) persists for a retention window precisely to fix this.
• "Fan-out from a queue" is a common antipattern. Don't have one queue with N workers each filtering "is this for me?" — use pub/sub (or a topic per consumer) instead.
• Kafka consumer-group counts matter. Workers within a group are limited by partition count: 12 partitions = at most 12 active workers in a group; adding a 13th means one sits idle.
• Ordering rules differ.
  • Queues like SQS standard: no ordering. FIFO SQS: per MessageGroupId.
  • Kafka: per-partition ordering. Partition by your ordering key.
  • Pub/sub topics: typically no global ordering across subscribers.
• Common misunderstanding: SNS-style pub/sub is sometimes thought of as "real-time" or "low latency" — it is, but it has no replay. If you need both fan-out and replayability (analytics, debugging, new consumer onboarding), you need a log-based pub/sub like Kafka.

4. Tradeoffs & Decisions

Key tradeoff: delete-on-consume vs append-only log. Classic queues delete messages once consumed — cheap, simple, but no replay. Log-based brokers retain messages for a window — heavier storage, but they let you onboard a new consumer that re-reads from the beginning, debug by inspecting history, and reprocess after a bug.

5. Interview & System Design Cheat Sheet

• A queue distributes one message to one worker — used for work. A topic delivers one message to every subscriber — used for facts.
• Kafka collapses both into one mental model: partitions + consumer groups. Within a group = queue. Across groups = pub/sub.
• Always ask: "Will another team need this data later?" If yes, use a log-based pub/sub now — retrofitting fan-out onto a queue is painful.
• Ordering requires partition-key control (Kafka) or MessageGroupId (SQS FIFO). Anything else is best-effort.
• The replay capability of log-based systems is the single biggest reason Kafka displaced traditional message queues in event-driven backends.

Common follow-ups:

• "Why not just use SNS → SQS fan-out?" — Works well for low-throughput notification fan-out without replay needs. Fails when you want a new analytics consumer to backfill the last week of events.
• "How does Kafka act as a queue?" — One consumer group with N consumers shares the topic's partitions. That's a queue. The "queue" persists, and unlike SQS, workers are capped at the partition count and progress is a committed offset per partition, not a per-message ack.
• "What guarantees does pub/sub give about ordering?" — Generally none across subscribers. Within a single partition / message-group, ordering is preserved per consumer.

If asked to design X, anchor on this: Decide for each interaction whether the consumer side is "work to distribute" (queue) or "facts to broadcast" (pub/sub). Then ask whether replay matters. Those two answers pick the broker.