Design an A/B Testing and Experimentation Platform

Deterministic User Assignment

Sticky assignment is the foundation of statistical validity: if the same user sees different variants on refresh, conversion lift becomes meaningless. MurmurHash3 on experiment_id:user_id gives uniform, reproducible bucketing in under 100 ns without per-request randomness or assignment database lookups.

User must ALWAYS see the same variant. No randomness per request.

Algorithm: hash-based assignment

  def get_variant(user_id, experiment_id, variants, traffic_percent):
      hash_input = f"{experiment_id}:{user_id}"
      # Gate: is this user in the experiment at all?
      if murmurhash3(hash_input) % 10000 >= traffic_percent * 100:
          return None
      
      # Independent hash for the variant split so it's uniform
      # across the in-experiment population (not just the gated range).
      variant_hash = murmurhash3(hash_input + ":variant") % 10000
      cumulative = 0
      for variant_name, weight in variants:
          cumulative += weight * 100
          if variant_hash < cumulative:
              return variant_name
      
      return variants[-1][0]

Properties:
  - Deterministic: same user + experiment -> same variant
  - Uniform: murmurhash3 is uniformly distributed
  - Independent: adding/removing experiments doesn't change other assignments
  - Fast: murmurhash3 is < 100 ns

Mutual Exclusion (Experiment Layers)

With 1,000 concurrent experiments, overlapping tests on the same surface corrupt attribution, making it impossible to determine whether the green button or the new layout drove the conversion lift. Experiment layers partition the hash space so each user enters at most one experiment per layer, while independent layers (UI vs backend) can run in parallel without interaction.

Problem: Experiment A tests checkout button color. Experiment B tests checkout page layout.
If same user is in both: which caused the conversion improvement?

Solution: Experiment Layers (Google's Overlapping Experiment Infrastructure)

  Layer 1 (UI experiments):
    Experiment A: button color (control: blue, treatment: green)
    Experiment B: header layout (control: v1, treatment: v2)
    Within a layer: user is in AT MOST one experiment
    
  Layer 2 (Backend experiments):
    Experiment C: recommendation algorithm (control: v1, treatment: v2)
    Independent of Layer 1

Implementation:
  Each experiment belongs to a layer.
  Assignment: hash(user_id + layer_id) % total_traffic
  Each experiment "owns" a non-overlapping range of hash values.

Statistical Analysis Engine

Raw event counts are not experiment results; product teams need lift, confidence intervals, and guardrail checks before shipping. An hourly batch job on ClickHouse aggregates runs two-proportion z-tests, applies multiple-comparison correction, and auto-pauses variants that degrade crash rate or latency beyond 2σ.

1. Conversion rate per variant
2. Relative lift = (treatment_rate - control_rate) / control_rate
3. Statistical significance (two-proportion z-test):
   p_pooled = total_conversions / total_impressions
   se = sqrt(p_pooled * (1-p_pooled) * (1/n_control + 1/n_treatment))
   z = (treatment_rate - control_rate) / se
   p_value = 2 * (1 - norm_cdf(abs(z)))
4. Confidence interval (95%): CI = diff ± 1.96 * se
5. Multiple comparison correction: Bonferroni or Benjamini-Hochberg
6. Guardrail metrics: auto-pause experiment if degraded > 2 std dev

Event Bus Design (Kafka)

The event bus decouples fire-and-forget SDK telemetry from analytics compute, leveraging the stream processing patterns detailed in Stream Processing Basics. Partitioning by user_idco-locates all events for one user so Flink can deduplicate exposures and conversions within tumbling windows before inserting pre-aggregated rows into ClickHouse, ensuring that raw 50B events/day never hit the OLAP store directly.

Topic: experiment-events
  Partitions: 640 (50B events/day to scale Flink horizontally)
  Partition key: user_id (co-locate all events for one user)
  Retention: 7 days (replay for ClickHouse backfill)
  Replication factor: 3, min.insync.replicas: 2

Producer: Client SDK (idempotent producer, fire-and-forget)
  Event: { event_id, user_id, experiment_id, variant, event_type: "exposure"|"conversion", value, timestamp }

Consumer groups:
  1. metric-aggregator: Flink dedup by (user_id, event_type, experiment_id, timestamp_bucket)
     -> tumbling 1-min windows -> ClickHouse materialized views
  2. guardrail-monitor: crash_rate, p99_latency, error_rate per variant -> auto-pause if > 2σ
  3. srm-detector: chi-squared test on assignment counts (expected 50/50 vs actual)

Sync path: variant assignment is deterministic hash; no Kafka on read path (< 5ms)
Async path: 50B events/day (~500K/sec avg) -> Flink pre-aggregate before ClickHouse insert
DLQ: experiment-events-dlq; alert when consumer lag > 60s