Design a Metrics Monitoring and Alerting System

Prompt

"Design a system that collects metrics from every server and service in a large fleet, lets engineers graph them, and fires alerts when a rule is violated. It must keep working when everything else is broken."

Clarifying questions

• What is a metric? A named numeric time series with labels, for example http_requests{service=api, method=GET, status=500} sampled every 10 seconds. Counters, gauges and histograms.
• How many sources and series? Say 1 million hosts and containers, each exporting a few thousand series.
• Query patterns? Dashboards over the last hours, ad-hoc exploration over days, alerting rules evaluated every minute over the last few minutes.
• Retention? Raw resolution for 2 weeks, downsampled to 1-minute for 3 months, 1-hour for 2 years.
• Alert delivery? Page, email, ticket, via an existing notification service. Deduplication and grouping of alerts are in scope.
• Consistency of counts? Approximate is fine for graphs; alerting must not miss a sustained violation, but may be delayed by a minute.
• Push or pull collection? Your choice; explain the trade-off.

Requirements

Functional

• Collect labelled time series from every source at 10-second resolution.
• Query: select series by label matchers, aggregate across labels, over a time range, with functions like rate and percentile.
• Alerting rules expressed as queries with a threshold and a duration ("for 5 minutes").
• Notifications with grouping, silencing and deduplication.

Non-functional

• Ingest at fleet scale with no single collector in the critical path.
• Dashboard queries under 1 s for hour-scale ranges; alert evaluation completes every minute even under load.
• Independence: the monitoring system must degrade gracefully when the fleet it monitors is failing, and must not depend on services it monitors.
• Data loss tolerated in seconds, not minutes.

Back-of-envelope

• Sources: 1 million x 2,000 series = 2 billion active series. Sample every 10 s: 200 million samples/s. This is the number everything else follows from.
• Sample size: a timestamp and a float64 is 16 bytes raw; with delta-of-delta timestamp and XOR float compression, ~1.5-2 bytes per sample in practice. Raw storage per day: 200 M/s x 86,400 x 2 B = ~35 TB/day compressed; 2 weeks of raw is ~500 TB.
• Downsampled 1-minute for 3 months: 6x fewer samples, 2 B x 200 M/6 x 86,400 x 90 = ~520 TB. Hourly for 2 years: 360x fewer samples than raw, 2 B x 200 M/360 x 86,400 x 730 = ~70 TB (more like 250 TB if each bucket keeps four aggregates: min, max, sum, count). Total live footprint on the order of 1 PB.
• Series index: each series' label set is ~200 bytes; 2 B x 200 B = 400 GB of label metadata. Fits across a sharded index, not on one machine.
• Ingest bandwidth: 200 M samples/s x ~50 bytes on the wire (before compression, including labels amortised by batching) = 10 GB/s. Batch and compress at the source; with label dictionaries and batching, ~1-2 GB/s into the ingest tier.
• Queries: assume 10,000 dashboard panels refreshing every 30 s and 50,000 alert rules per minute: ~330 dashboard queries/s plus ~830 rule evaluations/s. Each touches tens to thousands of series; the read fan-out is the challenge, not the query count.