Design a Fraud Detection System

Interview Walkthrough

• 25-minute cut

  Skip arch50/arch75 depth unless staff.

  • Latency budget: scoring must complete in <50 ms (5 min)
  • Tiered pipeline: rules first, ML second (6 min)
  • Flink-updated velocity counters in Redis feature store (5 min)
  • Hybrid fail-open/fail-close by transaction amount (5 min)
  • Offline GNN batch pipeline for fraud rings (4 min)
• Frame as a latency-budget problem: scoring must complete in <50 ms, prioritizing rules first (deterministic, explainable) and ML second (novel patterns).
• Walk through the tiered pipeline: rule engine blocks known patterns instantly → feature store lookup → ML model score → decision (allow/review/decline).
• Explain feature freshness: Flink-updated velocity counters in Redis with staleness detection that biases toward REVIEW when features lag.
• Cover hybrid fail-open/fail-close: auto-allow low-amount trusted users, auto-decline high-amount when scorer is down, queue medium for async review.
• Mention offline GNN batch pipeline for fraud rings, where nightly community detection writes cluster risk scores back to Redis for O(1) lookup at scoring time.
• Discuss champion/challenger model deployment with shadow scoring logged to ClickHouse before promoting a new model version.
• Common pitfall: pure fail-close when the ML service hiccups, because blocking every transaction costs more revenue than the fraud it prevents.

Why Both Rules AND ML

Rules: deterministic, explainable, instant deployment for known patterns.
  "Block all transactions from sanctioned countries" -> rule, not ML.

ML: catches novel patterns, handles complex feature interactions.
  "User's spending pattern changed subtly over 3 weeks" -> ML, not rules.

Together: Rules catch known fraud immediately. ML catches new fraud patterns.
  Defense in depth: if ML misses it, rules may catch it (and vice versa).

Feature Freshness & Staleness Handling

Detection: Every feature hash includes _updated_at timestamp
  At scoring time: feature_age = now() - features._updated_at
  
Strategy 1: Conservative bias
  If velocity features are stale → add 0.15 to fraud score
  Rationale: "I can't see recent activity → assume higher risk"
  
Strategy 2: Default substitution
  Replace stale feature with population median
  
Strategy 3: Feature importance gating
  If top-5 most important features are stale → route to REVIEW (skip ML)

Weekly vs Daily Model Retraining

Hybrid approach (recommended):
  Base model: retrained weekly (stable foundation)
  Rule engine: updated within HOURS for known new patterns
  → Analyst spots new pattern → creates rule → deployed in 30 min
  → Rule catches new fraud immediately while model takes a week to learn