Design a User Analytics Pipeline (like Google Analytics)

1. Edge Collector & Enrichment Architecture

The client SDK batches events locally before forwarding them to reduce network overhead. The stateless collectors enrich the incoming payload:

POST /collect
{
  "site_id": "site-123",
  "client_id": "anon-uuid",      // browser fingerprint / cookie
  "session_id": "sess-uuid",
  "events": [
    {"type": "pageview", "url": "/pricing", "ts": 1710320000, "referrer": "google.com"},
    {"type": "click", "element": "#signup-btn", "ts": 1710320005},
    {"type": "custom", "name": "video_play", "props": {"video_id": "v123"}, "ts": 1710320010}
  ]
}

• GeoIP Lookup: Resolve client IP to country and city utilizing an in-memory MaxMind database.
• User-Agent Resolution: Parse client browser, OS version, and device type.
• Bot Filtering: Cross-reference signatures against known bot lists to filter noise.

2. Real-Time Processing (Apache Flink)

Flink consumes the enriched event stream from Kafka, executing windowed aggregations:

1. Active Users (1-minute tumbling window):
   Count distinct client_ids per site_id per minute
   Write to Redis: SETEX active_users:{site_id} 120 {count}

2. Events per second (10-second sliding window):
   Count events per site_id → write to Redis time-series database.

3. Top Pages (1-minute window):
   Count pageviews per URL per site_id → update Redis sorted set
   ZADD top_pages:{site_id} {count} {url}

3. Batch Processing & Session Stitching (Apache Spark)

Spark pulls raw records from S3 partitions, running intensive hourly processing jobs:

1. Read raw events from Kafka → S3 (stored as Parquet, partitioned by date + site_id)
2. Session reconstruction: group events by session_id, calculate total session duration and page depth.
3. Funnel analysis aggregation: compute retention steps across funnel definition matrices.
4. Materialized pre-aggregations: generate daily pageview rollups grouped by country × device type.
5. Archival compaction: compact small files into large Parquet blocks in cold S3 storage.

4. ClickHouse OLAP Columnar Store

Row-oriented relational databases (like PostgreSQL) fail when executing analytical aggregates across billions of rows. We leverage ClickHouse's columnar MergeTree engine:

CREATE TABLE events (
    event_date    Date,
    site_id       String,
    session_id    String,
    client_id     String,
    event_type    LowCardinality(String),
    url           String,
    referrer      String,
    country       LowCardinality(String),
    device_type   LowCardinality(String),
    browser       LowCardinality(String),
    custom_props  Map(String, String),
    timestamp     DateTime64(3)
) ENGINE = MergeTree()
PARTITION BY (event_date, site_id)
ORDER BY (site_id, event_date, session_id, timestamp);

Event Bus Design (Kafka)

Topic: user_analytics_pipeline-events
  Partitions: 64 (scale consumers horizontally)
  Partition key: entity_id (user_id / order_id — preserves per-entity ordering)
  Retention: 7 days (compliance) or 24h (high-volume telemetry)
  Replication factor: 3, min.insync.replicas: 2

Producer: idempotent producer enabled (enable.idempotence=true)
Consumer: consumer group "user_analytics_pipeline-processors"
  - At-least-once delivery + idempotent handlers (dedup by event_id)
  - DLQ topic: user_analytics_pipeline-events-dlq (poison messages after 3 retries)
  - Lag alert: consumer lag > 60s → scale workers

Design a User Analytics Pipeline (like Google Analytics): async side effects MUST NOT block the synchronous API response.
  Sync path: validate → persist source of truth → publish event → return 201
  Async path: consumers update caches, indexes, notifications, aggregates