Design a Search Ranking System (Learning to Rank)

Overview

Staff level depth centers on position debiasing, multistage candidate distillation, feature consistency, and safe model deployment using shadow evaluation and interleaving.

Learning to Rank Approaches

Learning to rank approaches differ in their loss formulations. Pointwise methods treat relevance independently, pairwise methods optimize relative preferences, and listwise methods optimize a ranking objective over a result list. LambdaMART is best described as a pairwise lambda based approach that weights pair updates using changes in the target ranking metric such as NDCG.

Pointwise: Treat ranking as regression or classification
  Input: (query, document) -> Output: relevance score (0-4)
  Loss: Mean Squared Error (MSE) or cross entropy
  Problem: Does not optimize relative ranking order directly

Pairwise: Predict which of two documents is more relevant
  Input: (query, doc_A, doc_B) -> Output: P(A better than B)
  Better than pointwise: learns relative pairwise ordering
  Example: LambdaMART uses pairwise lambda gradients weighted by changes in a target ranking metric such as NDCG

Listwise: Optimize a ranking objective across a whole result list
  Input: (query, [doc_1, ..., doc_n]) -> Output: ranked list
  Loss: listwise objectives such as ListNet or ListMLE
  Advantage: models the full result list rather than isolated pairs

Relevance Labels and Training Data

High quality training combines human relevance judgments with continuous implicit feedback. Human labels provide the gold standard, while click data provides scale once it is debiased for position and selection effects. In practice, monitor propensity estimates for instability and clip extreme inverse propensity weights when necessary so a small number of rare observations does not dominate training.

Expert evaluators judge (query, document) pairs on a calibrated 0-4 scale, providing the highest quality supervised signal. Click logs add scale once they are debiased for position and selection effects. Each impression records the model version, rank position, experiment assignment, retrieval source, timestamp, and query context so feedback can be replayed and analyzed without ambiguity. Dwell time, scroll depth, and shares can reinforce relevance signals when interpreted with care.

Key Features for Search Ranking

Feature categories span static document qualities, dynamic query attributes, and dense semantic interaction embeddings that the ranker combines into unified scores.

Query Features:
  - Query token length, intent classification (navigational, informational, or transactional)
  - Named entity tags, historical query frequency, and spell corrected canonical tokens

Document Features:
  - PageRank, domain authority scores, and content freshness timestamps
  - Document body length, historical click through rate, and spam probability score

Query Document Interaction Features:
  - BM25 score (title, body, and anchor text) and TF-IDF cosine similarity
  - Optional cross encoder score for final candidates using query and document text
  - Dense embedding cosine similarity computed via bi encoder models

User Features (Personalization):
  - User search and click history, geolocation, language preference, and device type

Feature Store Architecture

The feature store powers both low latency online inference and reproducible offline model training. Point in time joins eliminate future feature leakage, while explicit feature schema versions prevent training and serving drift.

Feature Store Architecture:

- Feature Pipeline: Processes raw event logs via Apache Beam or Flink.
  Computes rolling aggregations defined in feature registry and writes to online and offline stores.

- Online Feature Store (Redis): Low latency storage for real time inference.
  Keyed by entity_id, returning dense feature vectors in under 5ms.

- Offline Feature Store (S3 / Parquet): Historical feature repository.
  Maintains point in time correct values joined with historical impression labels.

- Feature Versioning: Model contracts pin to explicit feature set versions,
  preserving legacy version schemas until all production rankers migrate.

Serving Optimization and Candidate Budgeting

At 100K QPS, the system cannot rely on raw per query model execution. Retrieval and ranking services batch work aggressively, reuse head query and feature results when safe, and apply early exits when the latency budget is threatened. Candidate budgets are explicit so every later stage receives a bounded workload.

• Head Query Caching: Cache highly repeated query results and stable query features with short TTLs while bypassing the cache for personalized or freshness sensitive searches.
• Micro batching: Group requests over very short windows to improve accelerator utilization without violating the 200ms p99 budget.
• Candidate Budgeting: Treat 10K retrieved candidates and 500 fine ranking candidates as capacity targets. Reduce the fine ranking set when GPU saturation or tail latency rises.
• Early Exit: Skip expensive cross encoder evaluation when an offline validated score margin indicates that the remaining candidates are unlikely to change the top results, or when the remaining latency budget is too small.

Retrieval Quality and Candidate Fusion

Hybrid retrieval should optimize recall before ranking optimizes precision. BM25 and ANN retrieval can run in parallel, then merge and deduplicate their candidates into the bounded 10K pool. Monitor recall at several cutoffs against a sampled exhaustive relevance set so ANN recall regressions are detected before they contaminate downstream ranking metrics.

• Lexical Recall: Preserve strong exact match behavior for names, rare terms, identifiers, and highly specific queries.
• Semantic Recall: Use ANN retrieval to recover paraphrases and conceptually related documents that lexical matching may miss.
• Candidate Fusion: Merge candidates from both sources, deduplicate by document ID, and retain retrieval provenance for downstream features and debugging.
• Recall Guardrail: Alert when Recall@K drops against a stable judged benchmark, even if downstream CTR has not moved yet. This catches retrieval regressions before the ranker can compensate.
• Retrieval Provenance: Record the retrieval source and index snapshot version with each impression so a ranking decision can be reproduced after index refreshes or retrieval model changes.

Model Deployment Pipeline

Safe deployment guardrails reduce the chance that a new ranker degrades user experience. Shadow traffic exposes feature and latency issues without changing responses, while interleaving provides a paired comparison of rankers before canary traffic is increased. Offline NDCG remains the primary judged quality measure, while live monitoring relies on click and satisfaction proxies until enough judged data is available.

Stage 1: OFFLINE EVALUATION
  Train challenger model, evaluate on holdout test set:
  Check if NDCG@10 improved >= 0.5% and latency p99 <= 150ms.

Stage 2: SHADOW MODE (1-3 days)
  Both models score every live query asynchronously.
  Only the champion's results are returned to users.
  Evaluate: NDCG on labeled shadow samples or judged replay, execution latency, and error distribution.

Stage 3: INTERLEAVING EXPERIMENT (3-7 days)
  Interleave ranked results from both models in the same SERP (Team Draft algorithm).
  Measure which model's candidate results capture higher genuine clicks.
  Treat 10x fewer queries as an illustrative sample efficiency target rather than a universal statistical guarantee. Actual sample efficiency depends on query volume, preference strength, and experiment design.

Stage 4: CANARY (1% traffic, 2-3 days)
  Serve challenger output to 1% of live search users.
  Monitor judged or shadow based NDCG, CTR@1, abandonment rate, and latency p99.
  Auto rollback triggers if the evaluated quality signal drops > 1%, p99 latency > 200ms, or error rate > 0.1%.

Stage 5: GRADUAL RAMP (1-2 weeks)
  1% -> 5% -> 25% -> 50% -> 100% with a 2-day bake period per tier.
  Rollback at any stage executes via dynamic configuration in under 60 seconds.