Design a News Feed System (Facebook / Instagram)

Post Ingestion Service and Transactional Outbox

The Post Service handles post creation, schema validation, media asset references, and privacy configurations before persisting records to the database.

• Atomic Ingestion: Persists the post record into MySQL sharded by user_id and simultaneously inserts an event record into a local post_outbox_events table within the same database transaction. This prevents the failure window where a database write succeeds but the service crashes before publishing to Kafka.
• Reliable Dispatch: An asynchronous Outbox Publisher or change data capture (CDC) process reads pending outbox events, dispatches them to Kafka with at-least-once delivery guarantees, and marks them as published upon broker acknowledgment.
• Media Processing: Validates image and video upload receipts stored in Amazon S3, attaching CDN delivery URLs to the post metadata.

Fan-Out Service and Worker Fleet

The Fan-Out Service consumes events from Kafka and coordinates the distribution of post pointers into follower timeline caches.

• Follower List Resolution: Queries the Social Graph Service to retrieve follower lists in paginated chunks, avoiding loading millions of graph edges into worker memory simultaneously.
• Adaptive Routing: For creators below the adaptive celebrity threshold (such as 10,000 followers), workers pipeline Redis ZADD commands to follower sorted sets. For high-follower creators, the write-time push step is bypassed.
• Idempotent Processing: Because Kafka can deliver events more than once, workers rely on the unique member property of Redis sorted sets (post_id) to ensure re-delivered events update scores rather than creating duplicates. Offsets are committed only after successful Redis pipeline execution.
• Inactive User Optimization: When fan-out queue lag rises, workers can defer or shed write-time materialization for accounts that have been inactive for over 30 days. When an inactive user returns, their timeline is dynamically backfilled at read time.

Social Graph Service

The Social Graph Service manages bidirectional follow relationships and provides low-latency adjacency list queries.

• In-Memory Caching: Redis sets store following:{user_id} and followers:{user_id} for microsecond lookups of active social graphs.
• Durable Source of Truth: Cassandra and relational tables persist follow relationships partitioned by user_id, guaranteeing durable relationship storage and supporting paginated retrieval for large follower graphs.

Feed Assembly and Candidate Generation Service

The Feed Service coordinates feed retrieval requests, combining precomputed timelines with dynamic celebrity candidate pools.

• Precomputed Timeline Fetch: Retrieves top candidate post IDs from the user's Redis sorted set.
• Bounded Celebrity Pull: Queries the Social Graph Service for followed high-follower accounts and retrieves a strictly bounded set of recent posts (for example, the top 10 recent posts per celebrity within a 24-hour window) from edge caches or author timelines.
• Merge and Deduplication: Merges precomputed and pulled post IDs, deduplicating any overlapping entries.
• Visibility and Validity Filtering: Enforces the core rule that timeline membership is not final authorization to view a post. It filters out deleted posts, moderated content, private posts no longer accessible, and posts from blocked or muted accounts before ranking.
• Hydration and Response: Passes filtered candidates to the Ranking Service, hydrates full post metadata from cache or database for the top selected items, and returns the response with a pagination cursor.

Machine Learning Ranking Service

The Ranking Service scores candidate posts to personalize feed ordering based on predicted engagement probability.

• Candidate Pool: Accepts a bounded candidate pool of approximately 500 post IDs from the Feed Service.
• Feature Extraction: Retrieves user affinity scores, post age decay, content modality preferences, and engagement velocity from an in-memory feature store.
• Inference Engine: Executes lightweight scoring models (such as logistic regression or gradient boosted decision trees) to rank candidates.
• Resilience Fallback: If the ranking model exceeds a 50 ms timeout or fails, the pipeline automatically falls back to reverse-chronological sorting.

Redis Feed Cache Architecture

Redis sorted sets maintain lightweight chronological timelines for active users.

• Data Structure: Stored under feed:{user_id} where score represents the post timestamp or Snowflake ID and member represents the post_id string.
• Capacity Cap: Capped at 500 entries per user using ZREMRANGEBYRANK to keep the memory footprint around 8 KB per user.
• Idempotency and Trimming: Writing an existing post ID updates its score rather than creating a duplicate member, ensuring safe worker retries.

Kafka Event Bus Configuration

The event streaming topology structures message distribution, partition keys, and dead letter handling:

Topic: new-posts
  Partitions: 128
  Partition key: author_user_id (preserves chronological order per author)
  Retention: 7 days
  Replication factor: 3, min.insync.replicas: 2

Producer: Transactional Outbox Relay / CDC Worker
  Guarantees: At-least-once delivery from MySQL post_outbox_events table
  Event Schema: { event_id, post_id, author_user_id, created_at, is_celebrity, follower_count, content_preview }

Consumer Groups:
  1. fan-out-workers: Writes post_id pointers into follower Redis sorted set timelines (bypassed for high-follower celebrity accounts)
  2. notification-workers: Emits push notifications to close friends and high-affinity contacts
  3. search-indexer: Streams post content into Elasticsearch for full-text discovery
  Dead Letter Queue: new-posts-dlq after 3 retries, with lag alerts configured for delays above 30 seconds

Synchronous Ingress Path:
  Validate payload -> Persist post record and outbox event in MySQL in a single atomic transaction -> Return 201 Created

Asynchronous Outbox and Fan-Out Path:
  Outbox Publisher polls or tails transaction log -> Publishes event to Kafka -> Fan-out workers query follower graph -> Pipeline idempotent ZADD operations into follower Redis timelines -> Commit Kafka consumer offsets