Design a User Analytics Pipeline (like Google Analytics)

Failure Matrix

High volume analytics pipelines must preserve event continuity across network drops, stream worker crashes, and skewed customer workloads. The failure matrix covers durability, stream recovery, client reliability, abuse control, and partition skew.

1. Real Time Session Stitching and Watermarks

Out-of-order events from mobile devices and spotty networks make it difficult to determine when a user session concludes. The pipeline coordinates Flink session windows with event time, watermarks, and an explicit allowed-lateness policy.

Session Definition: A sequence of user interactions within a site containing no inactivity gap greater than 30 minutes.

Flink Session Windowing:
1. Group incoming events by site_id and client_id using Flink session windows with a 30-minute inactivity gap.
2. Advance event time watermarks with a 5-minute bounded-out-of-orderness tolerance, then keep window state available for the separately configured allowed lateness.

Late Event Edge Case:
- Condition: Mobile events arriving after the watermark has advanced beyond the session window.
- Streaming impact: Events arriving within the configured 5-minute allowed-lateness period can update or merge an existing session. Events arriving after window state has been cleaned up may form a separate session or be routed to a late-event stream. In this scenario, late events beyond the allowed lateness are expected to cause approximately 1% real time session count inflation.
- Batch mitigation: Hourly Spark jobs re-stitch raw S3 events using true event timestamps to reconcile accurate historical session counts.

2. Browser Exit Loss and Beacon Delivery

When a user abruptly closes a browser tab, pending asynchronous HTTP requests are often cancelled. The client SDK deploys background delivery methods and persistent local buffers to reduce telemetry loss during page lifecycle transitions:

Problem: Client browsers often terminate or navigate away before asynchronous requests complete, causing lost exit events.

Delivery Mechanisms and Fallbacks:
1. Primary transport (navigator.sendBeacon): Queues an HTTP POST asynchronously without blocking page navigation. A successful return means the browser accepted the payload for transfer, not that the analytics server acknowledged it. Browser implementations also limit the queued payload size to roughly 64 KiB.
2. Page lifecycle guidance: Flush on visibilitychange when the document becomes hidden, with pagehide as a fallback, rather than relying on unload or beforeunload.
3. Fallback transport 1 (Pixel Tracking): Sends an HTTP GET request to /pixel.gif with encoded event parameters. It can work in some browser contexts but may be blocked by content blockers and privacy tooling.
4. Fallback transport 2 (Durable Client Buffer): The web SDK saves unconfirmed events in browser storage and the mobile SDK uses app-local durable storage. Buffered batches retry when the application becomes active and connectivity returns.
5. Tolerance threshold: Treat 2% to 5% event loss as an explicit product-level tolerance for this scenario rather than an industry-wide guarantee. Stronger requirements should use durable client queues, event IDs, and retry semantics.

3. High Scale Unique Counting with HyperLogLog

Maintaining exact distinct user sets in memory across tens of thousands of client websites creates severe memory bottlenecks. The system deploys Redis HyperLogLog for probabilistic cardinalities:

Problem: Counting distinct active users across 50,000 corporate websites in real time creates massive memory bottlenecks if tracking raw sets.

Flink + HyperLogLog (HLL) Solution:
1. Write: The Flink keyed window maintains an HLL-style sketch per site and one-minute bucket. Each client_id updates the sketch instead of materializing the full distinct-user set.
2. Read: The window aggregate emits an estimated distinct-user count to Redis as active_users:{site_id}, with a 120-second TTL so stale live metrics expire naturally.
3. Accuracy: HyperLogLog provides an approximate distinct count with a standard relative error of approximately 0.81% for the canonical configuration.
4. Memory efficiency: Each HyperLogLog sketch consumes at most about 12 KB regardless of cardinality.
5. Scale footprint: One active minute bucket for 50,000 customer websites multiplied by 12 KB yields roughly 600 MB before allocator and replication overhead. Retaining 60 minute buckets would require roughly 36 GB before overhead, so retention policy directly affects state sizing.

Alternative Redis-native implementation:
PFADD active_users:{site_id}:{minute} {client_id}
EXPIRE active_users:{site_id}:{minute} 120
PFCOUNT active_users:{site_id}:{minute}

4. Event Identity and Deduplication

The ingestion contract treats the tenant-scoped pair (site_id, event_id) as the stable idempotency key because the pipeline uses at least once delivery. Retries can therefore produce the same logical event more than once. Flink streaming aggregates the (site_id, event_id) pair within a bounded retention period before updating live counters, while batch jobs deduplicate immutable S3 events before producing historical rollups. The retention horizon must cover the expected retry and replay window. ClickHouse retains site_id and event_id in the fact schema so reconciliation jobs can identify duplicate records and audit lineage.

5. Late Arriving Event Windowing Accuracy

Balancing dashboard update immediacy against historical analytical precision requires clear delineation between processing time and event time:

Dashboard Query: SELECT COUNT(*) FROM events WHERE timestamp > NOW() - INTERVAL 1 HOUR

Case 1 (Processing Time): Count events at the moment they reach the ingestion collector.
- Advantage: Stable output because historical numbers never shift retroactively.
- Disadvantage: Inaccurate because delayed mobile events are attributed to the wrong time window.

Case 2 (Event Time): Count events using the client side recorded creation timestamp.
- Advantage: Provides more faithful historical attribution based on when the user action occurred, subject to timestamp quality, clock skew, duplicates, and the configured lateness policy.
- Disadvantage: Dynamic shifting occurs because metrics for the latest hour update as delayed events arrive.
- Operational convention: Display a 5-minute settling window warning in user interfaces indicating that current hour counts remain provisional until the configured 5-minute settling period has elapsed.