Design a Distributed Message Queue

Prompt

"Design a durable publish-subscribe message queue that internal teams can use: producers publish to topics, consumers subscribe, messages are not lost. Assume it must handle the traffic of a large company."

Clarifying questions

• Delivery guarantee? At-least-once by default. Exactly-once is a follow-up. At-most-once is never acceptable.
• Ordering? Per key ordering within a topic is required (all messages with the same key are delivered in publish order). Global ordering across a topic is not.
• Retention? Messages are retained for a configurable period, default 7 days, regardless of consumption. Consumers can replay from any retained position.
• Message size and rate? Up to 1 MB, typically 1-10 KB. Aggregate in the millions of messages per second.
• Consumer model: push or pull? Pull, with long polling. Consumers commit their position explicitly.
• How many subscribers per topic? Usually a few, occasionally hundreds. Each subscriber group reads the whole topic independently.
• Durability expectation? A published message that has been acknowledged survives the loss of any single machine and any single zone.

Requirements

Functional

• publish(topic, key, payload) returns only after the message is durable.
• subscribe(topic, group) and pull(group, partition) deliver messages in per-key order.
• commit(group, partition, offset) records consumer progress; replay from an older offset is supported.
• Topics can be created with N partitions and a retention period.

Non-functional

• Publish latency p99 under 20 ms within a region.
• No acknowledged message lost under single-zone failure.
• Throughput scales linearly with partitions and brokers.
• Consumer lag is observable per partition.

Back-of-envelope

• Assume 2 million messages/s aggregate at 2 KB average: 4 GB/s ingest, 32 Gbit/s.
• With replication factor 3, internal write bandwidth is 12 GB/s. Consumers: assume 3 subscriber groups per topic on average, so 12 GB/s read from brokers. Total broker network: ~24 GB/s in and out, ~200 Gbit/s across the fleet.
• Storage: 4 GB/s x 86,400 s = 346 TB/day; x 3 replicas = ~1 PB/day; 7-day retention holds ~7.3 PB. Per year of writes it is 126 PB but only 7.3 PB is live at any time. Disk, not memory.
• Broker sizing: a broker with 24 disks can sustain about 2 GB/s sequential writes and hold ~200 TB. 7.3 PB / 200 TB = ~37 brokers by capacity; 12 GB/s / 2 GB/s = 6 by write throughput. Capacity dominates: plan 50-60 brokers with headroom.
• Partitions: for 2 M msg/s and a comfortable 10 K msg/s per partition, ~200 partitions carry the aggregate load; with many topics and headroom per topic, the fleet holds tens of thousands of partitions. Metadata for that many partitions is itself a thing to design.