Design a Threaded Discussion Forum (Reddit)

Post Service

• Create post: Validate content (size limits, flair required for some subs), upload media to S3 if present, write to PostgreSQL, publish post-created event to Kafka
• Post types: Text, link, image, video, poll: each stored with type-specific metadata
• Flair: Per-subreddit flair system (tags like "Discussion", "Meme", "Question"): enforced by Post Service
• Soft delete: Deleted posts keep the entry (for comment tree integrity) but content is replaced with "[deleted]"

Subreddit Service

• Manages: Subreddit creation, membership (join/leave), rules, moderator assignments, banned users
• Membership: user_subreddits table (user_id, subreddit_id): used by Feed Service to build home feed
• Access control: Public, restricted (anyone can view, only approved can post), private (invite-only)
• Moderator actions: Remove posts/comments, ban users, pin posts, configure automod rules

Comment Tree Service: The Key Challenge

Reddit's nested comment tree is the most complex data model:

Approach 1: Adjacency List (simple but slow for deep nesting)

comments (comment_id, post_id, parent_comment_id, ...)
-- To get full tree: recursive query or multiple DB round trips

Approach 2: Materialized Path ? (recommended)

comments (comment_id, post_id, path, ...)
-- path = "001/003/007/015" (ancestors from root)
-- To get all descendants: WHERE path LIKE '001/003/%'
-- Sorting by path gives pre-order traversal (natural comment display order)

Approach 3: Nested Set Model (fast reads, slow writes)

comments (comment_id, post_id, lft, rgt, ...)
-- All descendants: WHERE lft > parent.lft AND rgt < parent.rgt
-- But inserting a comment requires updating lft/rgt of many rows

Approach 4: Closure Table (flexible but storage-heavy)

comment_tree (ancestor_id, descendant_id, depth)
-- One row per ancestor-descendant pair
-- Fast queries but O(depth) inserts

Recommendation: Materialized Path for Reddit-like system. Efficient reads, reasonable writes, and supports sorting.

Vote Service

• Challenge: 500M votes/day = high write throughput on potentially hot rows
• Architecture:
  1. Vote request → check Redis if user already voted on this post/comment
  2. If new vote or changed vote → publish to Kafka vote-events
  3. Kafka consumer updates PostgreSQL (source of truth) and Redis (cached counts)
  4. Redis stores: vote_count:{entity_type}:{entity_id} as atomic counter
  5. Redis stores: user_votes:{user_id}:{post_id} → direction (up/down/none)
• Why not direct DB write: Hot posts get thousands of concurrent votes → DB contention. Kafka buffers and Redis absorbs reads

Ranking Algorithms: Deep Dive

Hot Ranking (Reddit's actual algorithm)

import math
from datetime import datetime

def hot_score(ups, downs, created_at):
    score = ups - downs
    order = math.log10(max(abs(score), 1))
    sign = 1 if score > 0 else -1 if score < 0 else 0

    # Epoch: Dec 8, 2005 (Reddit's birthday)
    epoch = datetime(2005, 12, 8, 7, 46, 43)
    seconds = (created_at - epoch).total_seconds()

    return round(sign * order + seconds / 45000, 7)

• Key insight: Time dominates. A post from 12.5 hours ago needs 10× the votes to rank equally with a new post
• The log scale means going from 10 → 100 votes has same effect as 100 → 1000

Wilson Score (for "Best" comment ranking)

import math

def wilson_score(ups, downs, confidence=0.95):
    n = ups + downs
    if n == 0:
        return 0
    z = 1.96  # 95% confidence
    p = ups / n

    return (p + z*z/(2*n) - z * math.sqrt((p*(1-p) + z*z/(4*n)) / n)) / (1 + z*z/n)

• Handles the "1 upvote, 0 downvote" vs "100 upvote, 1 downvote" problem
• Lower bound of confidence interval → more data = more confident

Controversial

def controversial_score(ups, downs):
    if ups <= 0 or downs <= 0:
        return 0
    magnitude = ups + downs
    balance = min(ups, downs) / max(ups, downs)
    return magnitude * balance

Feed Service

• Home feed = aggregation from subscribed subreddits
• Pre-computation: Background job periodically computes top posts per subreddit → stores in Redis
• On request: Merge top posts from subscribed subreddits, interleave by score
• Popular/All: Global feed → single pre-computed ranked list in Redis

Search Service (Elasticsearch)

• Indexed: Post title, body text, subreddit name, author username
• Features: Full-text search with BM25 ranking, filter by subreddit/time range/post type, sort by relevance or recency
• Sync: Post creates/updates → Kafka → Elasticsearch consumer indexes in near real-time

Moderation Pipeline

• Consumes post and comment events from Kafka for automated moderation:
  • Spam ML model: Detect spam posts (link farms, repetitive content, new account patterns)
  • Toxicity classifier: Flag toxic/hateful content for review or auto-removal
  • AutoMod rules: Per-subreddit regex rules (e.g., "remove posts with banned keywords", "require minimum karma to post")
  • Report queue: User reports → prioritized queue for human moderators
  • Auto-removal: Content matching high-confidence spam/toxicity → removed immediately, mod notified

Event Bus Design (Kafka)

Topic: reddit_discussion_forum-events
  Partitions: 64 (scale consumers horizontally)
  Partition key: entity_id (user_id / order_id — preserves per-entity ordering)
  Retention: 7 days (compliance) or 24h (high-volume telemetry)
  Replication factor: 3, min.insync.replicas: 2

Producer: idempotent producer enabled (enable.idempotence=true)
Consumer: consumer group "reddit_discussion_forum-processors"
  - At-least-once delivery + idempotent handlers (dedup by event_id)
  - DLQ topic: reddit_discussion_forum-events-dlq (poison messages after 3 retries)
  - Lag alert: consumer lag > 60s → scale workers

Design a Threaded Discussion Forum (Reddit): async side effects MUST NOT block the synchronous API response.
  Sync path: validate → persist source of truth → publish event → return 201
  Async path: consumers update caches, indexes, notifications, aggregates