Metrics Monitoring and Alerting System

Phase 1 — Clarify & Scope

Interviewer: Design a metrics monitoring and alerting system — think Datadog, Prometheus, or a large-scale internal observability platform. What do you want to clarify first?

Candidate: A few questions to nail the scope:

1. What are we ingesting — infrastructure metrics (CPU, memory), application metrics (request latency, error rate), or both?
2. What's the write volume — how many metrics/second?
3. Query patterns: real-time dashboards, alerting, or long-term trend analysis?
4. Retention requirements — 15 days, 1 year, multiple years with downsampling?
5. Alerting: threshold-based, anomaly detection, SLA breach?

Interviewer: Infrastructure + application metrics. 10,000 hosts each emitting 100 metrics every 15 seconds. Real-time dashboards and alerting — latency under 5 seconds. Retain raw data for 15 days, downsampled for 1 year.

Candidate: Let me size this:

$\text{Ingest rate} = \frac{10{,}000 \times 100}{15} \approx 66{,}667 \text{ data points/sec}$

$\text{Raw storage (15 days)} = 66{,}667 \times 86{,}400 \times 15 \approx 86.4 \text{ billion samples}$

At Gorilla compression (~1.37 bytes/sample average), that's:

$86.4 \times 10^9 \times 1.37 \approx 118 \text{ GB compressed}$

Long-term with 5-minute downsampling (from 15-second raw):

$\text{Reduction} = \frac{20 \text{ (fewer points)}}{5 \text{ (aggregates per point)}} = 4\times \Rightarrow \frac{118\text{ GB}}{15\text{ days}} \times \frac{365}{4} \approx 718 \text{ GB/year for 1yr window}$

Interviewer: Good math. What are the core functional requirements?

Candidate:

• Collect time-series metrics from agents on every host
• Store time-series data with configurable retention
• Query metrics…