Design a Video Recommendation Engine

Two-Stage Retrieval and Ranking Funnel

The synchronous request path operates as a two-stage funnel: Stage 1 retrieves approximately 500 candidates from a FAISS ANN index in under 10ms, while Stage 2 runs a cross-network ranking model on that shortlist in under 40ms. All other processes, including feature updates, model training, and A/B experiments, remain off the hot path via Kafka.

Event Bus Design (Kafka)

Watch behavior is the lifeblood of personalization. The player SDK emits idempotent watch events, and Flink consumers update the Redis feature store in near real time while a separate training pipeline batches historical data for nightly model retrains.

Topic: watch-events
  Partitions: 256
  Partition key: user_id (preserves per-user watch sequence for feature updates)
  Retention: 7 days (training data replay)
  Replication factor: 3, min.insync.replicas: 2

Producer: video player SDK (idempotent producer)
  Event: { user_id, video_id, watch_pct, duration_sec, timestamp, device }

Consumer groups:
  1. realtime-features: Flink updates user_features:{user_id} and user_watch_history List (last 50)
  2. training-pipeline: batch to S3, followed by Spark feature engineering and weekly model retrain
  3. ctr-computer: join recommendation-served and recommendation-clicked for A/B metrics

Topic: video-published: triggers embedding generation and ANN index update
Topic: model-events: A/B framework automated rollback upon metric regression

Sync path: GET /recommendations reads Redis feature store and Faiss ANN in under 50ms
Async path: watch events never block playback, and near-real-time features update in under 5 minutes
DLQ: watch-events-dlq, with alerts when Flink consumer lag exceeds 300s

Embedding-Based Retrieval and ANN Search

User and video embeddings reside in a shared 256-dimensional vector space. The two-tower model trains offline so that the dot product of user and video embeddings correlates directly with watch probability. This mathematical alignment enables ANN search at 500M-video scale. Without separate user and item towers, pre-computing embeddings and searching in sub-10ms would be impossible.

User and video embeddings in 256-dim vector space.

Two-Tower Model training:
  User tower: user_id embedding + avg watched video embeddings + demographics
  Video tower: video_id embedding + title embedding + category + channel
  Objective: Maximize dot(user_emb, pos_video_emb) - dot(user_emb, neg_video_emb)

ANN Index (FAISS IVF-PQ):
  500M videos x 256 x 4 bytes = 512 GB raw → compressed to 32 GB with IVF-PQ
  IVF: cluster into 4096 groups → search nearest 50 groups
  PQ: compress each vector to 64 bytes (16x compression)
  Query time: < 5 ms per user embedding → serves 200K QPS across 10 replicas

Near-Real-Time Feature Updates

Recommendations feel fresh when user features update within minutes of a watch event. The streaming path updates lightweight features in Redis, while the heavy two-tower retraining job runs nightly and rebuilds the FAISS index.

User watches a video at 3:00 PM. By 3:05 PM, recommendations reflect this.

Pipeline:
  1. User finishes video → watch event to Kafka
  2. Flink streaming job:
     a. Update user's recent watch history (last 50)
     b. Update user interest vector (exponential moving average)
     c. Lightweight user embedding update (avg of last 50 watched)
     d. Update video stats (views, avg_watch_pct)
  3. Next recommendation request (< 5 min later) uses updated features

Full model retraining (daily):
  Spark job: extract all watch events from last 30 days
  Train two-tower model on GPU cluster (4-8 hours)
  Generate new embeddings for ALL users and videos
  Build new FAISS index → blue-green deployment swap

Post-Ranking Filters: Diversity and Quality

The ranking model optimizes engagement, while post-ranking filters enforce essential product constraints such as diversity, freshness, quality floors, and business rules so the feed does not devolve into a monoculture from a single channel.

After ranking model outputs top 100 scored candidates:

1. Diversity filter: Max 3 videos per channel, 30% per category. MMR algorithm.
2. Freshness boost: < 24h old → 1.2x, < 7 days → 1.1x
3. Quality filter: Remove < 40% avg watch pct, high dislike ratio, flagged content
4. Explore vs Exploit: 90% exploitation + 10% exploration (ε-greedy)
5. Business rules: Inject ads, boost premium, suppress blocked creators
6. Dedup: Remove watched, too-similar (pHash), reuploads

Cold Start for New Users and New Videos

Cold start is where many recommendation system designs stumble in interviews. Address this with a deliberate fallback ladder (trending content, followed by content-based heuristics, and finally collaborative filtering) that gradually hands off to full personalization as watch history accumulates.

New User:
  - Onboarding: select interests → initialize interest vector
  - Fallback: popular by country, language, time of day, referral source
  - Progressive: 80% popular → 40% → 10% as watch history accumulates

New Video:
  - Content-based embedding from title/description/tags (BERT → 256-dim)
  - Channel prior: inherit channel's avg engagement metrics
  - Exploration allocation: 10% slots for < 24h old videos
  - Multi-armed bandit (Thompson Sampling): learn CTR after ~1000 impressions
  - Blend: blended_emb = α x content_emb + (1-α) x collab_emb (α decays from 1.0 to 0.2)