Design a Real-Time Dashboard and Metrics System

Dashboard Stampede: Incident Causes Mass Concurrent Query Load

When a critical outage occurs, hundreds of engineers open identical dashboards within seconds. Without coordinated defenses, this sudden wave of concurrent panel executions can overwhelm upstream metric databases.

Incident Stampede Scenario:
  During a major operational outage, 500 engineers open the identical production dashboard simultaneously, triggering up to 5,000 panel queries at the exact same moment.

Defensive Architecture:
  1. Query Result Caching: The first request triggers an upstream database query, while subsequent requests within the effective cache TTL resolve directly from Redis.
  2. Singleflight Request Coalescing: Concurrent cache misses for the same query key share a single in-flight backend execution, preventing duplicate queries.
  3. Auto-Refresh Jitter: The client adds a plus-or-minus 10% randomized timing offset to auto-refresh timers, smoothing request spikes across time.
  4. Pre-Computed Materialized Views: Critical P0 dashboards are precalculated by background workers every 30 seconds so queries read static tables instead of scanning raw time series.

Data Source Timeout: Independent Asynchronous Panel Loading

A single slow telemetry backend must never prevent responsive panels from rendering. Querying data sources independently in parallel guarantees localized failure isolation.

Independent Panel Loading Architecture:
  Flawed Synchronous Approach:
    The user interface waits for the slowest panel to finish querying before rendering the dashboard, delaying page visibility by up to 30 seconds.

  Resilient Asynchronous Pattern:
    1. Independent Panel Queries: Every panel initiates an asynchronous query request in parallel without blocking sibling widgets.
    2. Progressive Rendering: Panels querying fast caches or local metric stores render immediately in under 100 milliseconds.
    3. Isolated Timeout Handling: Slow or unresponsive panels display localized loading indicators and fail gracefully after a 30-second timeout while the remainder of the dashboard functions normally. The proxy cancels the upstream query when the backend supports cancellation so timed-out work does not continue consuming resources.

Alert Evaluation: State Transitions and Flapping Prevention

Alert rules evaluate time series conditions across sliding evaluation windows using the same validated query execution path as dashboards. Persisted rule definitions provide the trigger threshold, recovery threshold, evaluation interval, notification targets, and sustained duration, while the evaluator maintains the OK, PENDING, FIRING, and RESOLVED states independently from dashboard rendering. Notification events use stable deduplication identities so evaluator retries remain idempotent. Incorporating pending states and hysteresis thresholds prevents rapid alert flapping caused by transient metric spikes.

Alert Evaluation State Machine:
  Evaluation Rule: Trigger an alert if service error rate exceeds 5% continuously for 5 minutes.
  The evaluator reads the persisted alert rule definition, including evaluation interval and notification targets, executes its normalized read only query through the shared query path, and maintains evaluation state independently from dashboard rendering. Alert evaluation can use cached results only when they satisfy the rule's freshness requirement, while the sustained PENDING interval is tracked durably enough to survive evaluator restarts without losing the rule's evaluation window. Notification delivery uses a stable alert event identity so evaluator retries do not create duplicate notifications.

  State Transition Lifecycle:
    1. OK: Metric remains below the configured threshold.
    2. PENDING: Metric breaches the 5% threshold and a timer initializes for the 5-minute sustained duration window.
    3. FIRING: Metric has remained continuously breached for 5 minutes, prompting webhook notifications to PagerDuty or Slack.
    4. RESOLVED: Metric recovers below the recovery threshold, returning state to OK.

  Hysteresis Configuration:
    Trigger threshold: Fire when error rate exceeds 5.0%.
    Recovery threshold: Resolve only when error rate drops below 3.0%.
    A 2.0% hysteresis margin eliminates alert flapping on noisy boundary metrics.

Multi-Region Federated Queries

Production dashboards often aggregate metrics across regional time series database clusters. The Query Proxy fans out sub-queries in parallel to each regional cluster, aligns and merges returned time series along unified timestamp boundaries, and caches the merged result in Redis. Operators accept 30 to 60 seconds of cache staleness on global rollups during incidents to protect cross-region network links. Enforcing strict panel level deadlines ensures that an outage in one region never blanks the entire operational dashboard.

Related Problems and Core Concepts

Explore complementary architectural designs spanning telemetry ingestion, time series storage, stream processing, and on call alerting workflows:

• Distributed Metrics Aggregation System: Scalable telemetry ingestion pipeline that collects raw samples, computes rolling window aggregations, and compresses long term time series.
• Distributed Tracing System: Trace collection and span graph indexing engine enabling drill-downs from anomalous dashboard latency spikes directly into distributed request paths.
• On-Call Escalation System: Incident paging and escalation platform that routes alert firing events evaluated by dashboard rules to responsible engineering rotations.
• Distributed Stream Processing: Stateful stream computing architecture handling high throughput tumbling and sliding window rollups for under one second telemetry dashboards.
• Distributed Cache: Memory caching architecture providing singleflight coalescing and low latency response delivery for high-concurrency query patterns.
• Time-Series and Metrics Storage: In-depth analysis of Gorilla delta-of-delta compression, inverted label index structures, and tiered storage lifecycles.
• Stream Processing Basics: Foundations of event-time processing, watermark semantics, and sliding window aggregations powering live dashboards.
• Network Protocols: HTTP, gRPC, WebSocket, DNS: Protocol trade-offs governing persistent WebSocket push channels vs stateless HTTP polling for real time visualization.
• Caching Patterns and Invalidation: Cache-aside, read-through, and singleflight coalescing strategies designed to prevent stampedes during incident traffic surges.

Interview Walkthrough

When presenting this real time dashboard design in a system design interview, maintain focus on query proxy caching, asynchronous panel isolation, and pre-aggregation mechanics:

• 25-minute interview pacing guide

  Balance dashboard visualization concerns against backend query performance and stampede mitigation.

  • Separate dashboard rendering from metric storage and clarify real time latency targets (5 min)
  • Design preaggregated rollups at ingest and define the versioned dashboard schema (6 min)
  • Architect the Query Proxy with Redis caching and singleflight request coalescing (5 min)
  • Implement independent panel queries with localized timeout and degradation boundaries (5 min)
  • Formulate the alert evaluation state machine with hysteresis to eliminate alert flapping (4 min)
• Separate dashboard rendering from underlying time series storage so the visualization platform and telemetry storage tiers scale independently.
• Pre-aggregate common queries at ingest into 1-minute and 5-minute rollups so dashboard widgets query precomputed buckets rather than scanning raw metric samples.
• Push live updates via WebSocket subscriptions keyed by dashboard identifier so connected clients receive delta refreshes instead of issuing full re-queries.
• Cache query results with time-to-live settings aligned with dashboard refresh intervals and apply singleflight coalescing so hundreds of engineers opening an incident dashboard share a single backend query.
• Load each panel independently with per panel timeouts so fast data sources render immediately while slow or degraded backends display localized loading states.
• Implement alert evaluation as a deterministic state machine moving from OK to PENDING to FIRING, incorporating hysteresis to prevent flapping on noisy telemetry thresholds.
• Add random jitter to auto-refresh intervals so thousands of active client browser tabs do not synchronize their requests on the same second.
• Highlight the common operational trap of permitting every browser refresh to trigger raw database queries, which causes catastrophic query stampedes during critical outages.