Time-Series & Metrics Storage

Metrics and IoT data are write-heavy, time-ordered, and rarely updated — a different shape than relational rows. We use time-series databases or columnar stores tuned for append and range queries.

We think in three dimensions when sizing a metrics pipeline:

Time-series data has write-heavy append patterns and time-range queries — specialized stores beat general SQL. Three lenses:

Walk ingestion and query as interview steps. Step 1 — Ingest: agents push metrics with timestamp + tags. Step 2 — Buffer: batch writes to LSM or columnar chunks. Step 3 — Index tags: inverted index on labels for filter. Step 4 — Query: time-range scan + aggregation (sum, rate, histogram). Step 5 — Retention: downsample old data; state roll-up policy.

Tag cardinality, compression, and downsampling — characteristics we compare across TSDBs:

• Append-only writes — samples are inserted, rarely updated in place.
• Compression — delta encoding and Gorilla-style algorithms exploit sequential timestamps.
• Retention policies — raw 7–15 days, aggregated months/years at coarser resolution.
• Pull vs push — Prometheus pulls from /metrics endpoints; InfluxDB and M3 use push ingest (Telegraf, remote write). M3 (Uber) shards time series across nodes with embedded etcd — mention when interviewer asks about Prometheus at multi-million-series scale.

Optimized time-range queries trade general-purpose SQL flexibility — we state both:

Cardinality explosion, hot shards on single metric, and query timeout — we name mitigations:

Monitoring dashboards, IoT telemetry, and financial ticks use time-series stores:

Reach for TSDB when writes are append-only time-stamped events and queries are range aggregates:

Time-series connects to stream processing, observability metrics, and analytics write scaling:

• #20 Stream Processing Basics — real-time aggregation before samples land in TSDB.
• #26 Probabilistic Data Structures — HyperLogLog for approximate unique counts.
• #29 Observability & Distributed Tracing — metrics complement traces and logs.
• #86 Time-Series Database (problem) — full TSDB HLD with retention and query paths.
• #39 Distributed Metrics Aggregation (problem) — multi-tenant metrics pipeline.
• #40 Real-Time Dashboard (problem) — Grafana-style visualization layer.

Compression: delta and Gorilla encoding

Adjacent samples in a time series are often similar. Delta encoding stores the difference from the previous value instead of the full float. Timestamps with fixed scrape intervals compress further with delta-of-delta (if every point is +15s apart, you store one constant). Facebook's Gorilla paper applies XOR compression to floats — similar values share leading zero bits — achieving roughly 1–2 bytes per sample in practice.

Rollup Tier Design

Production systems rarely keep raw 1-second resolution forever. A typical tier ladder:

Raw (1s resolution)     → retain 7 days   → SSD hot tier
Rollup 1 (1min avg/max) → retain 90 days  → SSD warm tier
Rollup 2 (1h avg/max)   → retain 2 years  → object storage / columnar
Rollup 3 (1d avg/max)   → retain 5+ years → cold archive (Parquet)

Background compaction jobs merge raw blocks into coarser buckets. Queries auto-select the finest tier that satisfies the requested time range — dashboards for "last 6 months" hit hourly rollups, not raw samples.

Prometheus Pull Model vs Push Gateway

Prometheus scrapes HTTP /metrics endpoints on a fixed interval (typically 15s). This gives uniform sampling and service discovery integration. Short-lived batch jobs cannot be scraped — they push to a Pushgateway, which Prometheus then scrapes. Interview pitfall: pushing every sample from every pod creates a single point of failure; pull is preferred for long-running services.

Histograms and Percentile Estimation

Storing every request latency as a raw sample is expensive at billions of QPS. Histograms bucket values (le=0.1, le=0.5, le=1.0 seconds) and increment counters per bucket. Percentiles are estimated from cumulative bucket counts — not exact, but stable under load. Choose bucket boundaries to match SLO thresholds (e.g., 100ms, 300ms, 1s for API latency).

TSDB vs Columnar Warehouse vs Search Index

• TSDB (Prometheus, InfluxDB, M3): low-latency range queries on metrics; weak at ad-hoc SQL joins.
• ClickHouse / BigQuery: excellent for analytics over event tables with time partition; higher query latency, cheaper at petabyte scale.
• Elasticsearch: full-text log search with time filters; not a replacement for numeric metric rollups at Prometheus scale.

In interviews, metrics dashboards → TSDB; business analytics over click events → Kafka + columnar warehouse; grep-able logs → search index. Many production stacks use all three with clear boundaries.

Capacity Sketch

10k pods x 500 metrics x 1 sample/15s ≈ 333k samples/sec. At ~2 bytes compressed per sample (optimistic), that is ~600 MB/min raw — retention and cardinality multiply this quickly. Always estimate active series count (unique metric+label combinations), not just sample rate.