Design a Video Streaming Platform (YouTube / Netflix)

Upload Flow (Transactional Outbox Pattern)

Keep the upload data path completely decoupled from application API servers so video payloads never traverse API gateways. Direct-to-storage transfers prevent application servers from becoming high-bandwidth bottlenecks.

1. The client requests a pre-signed multipart upload URL from the Upload Service, specifying file size, MIME type, and expected chunk count.
2. The Upload Service validates creator permissions and generates short-lived pre-signed S3 upload URLs with scoped object keys.
3. The client uploads raw video chunks directly to S3 in 5 MB to 10 MB parts using S3 multipart upload, allowing retries of failed parts without restarting the upload.
4. Upon completing all chunk uploads, the client invokes the complete-upload endpoint, supplying part numbers and S3 entity tags (ETags).
5. The Upload Service verifies S3 chunk completeness and executes an atomic MySQL transaction: updating the video record status to processing and inserting an event into the outbox_events table.
6. Debezium Change Data Capture reads the MySQL binary log and streams the video-uploaded event to Kafka with at-least-once durability, completely eliminating dual-write failure windows between MySQL and Kafka.
7. The distributed workflow orchestrator consumes the event from Kafka to schedule the transcoding Directed Acyclic Graph (DAG).

Video Processing Pipeline: Codecs, GOP Chunking, and Packaging

Transcoding represents the most computationally intensive write-path stage, requiring keyframe-aligned segment parallelization and adaptive bitrate packaging.

GOP Alignment and Parallel Chunking: Workers probe the master container and split the video strictly along Group of Pictures (GOP) and Instantaneous Decoder Refresh (IDR) keyframe boundaries into 4-second chunks. Splitting at arbitrary byte offsets is avoided because it corrupts temporal compression references. Distributed GPU workers pull chunks from S3 and transcode them in parallel across the canonical 5 baseline video renditions: 240p (400 Kbps), 360p (800 Kbps), 480p (1.5 Mbps), 720p (3 Mbps), and 1080p (5 Mbps), while extracting and encoding an independent 128 Kbps AAC and Opus audio track. A conditional 4K rendition (15 to 20 Mbps) is generated only when the uploaded master file supplies native 4K resolution, strictly prohibiting artificial upscaling. Workloads produce dual codec ladders: H.264 for universal device playback, alongside HEVC or AV1 for modern hardware to achieve 30% to 50% bandwidth savings.

Adaptive Bitrate Packaging with CMAF: Common Media Application Format (CMAF) standardizes fragmented MP4 (fMP4) containers across streaming protocols. Packagers encapsulate chunked media so a single set of media segments is referenced simultaneously by HLS master playlists (master.m3u8) and DASH manifest descriptors (manifest.mpd), cutting origin storage requirements in half compared to storing separate MPEG-TS and fMP4 assets. Client players monitor buffer occupancy and network throughput: downgrading quality when the buffer falls below 5 seconds, and upgrading when buffer occupancy exceeds 15 seconds with verified bandwidth headroom. Switching occurs strictly at keyframe boundaries to guarantee artifact-free transitions.

Digital Rights Management (DRM) and CENC: Protected catalog content is secured using ISO Common Encryption (CENC AES-128). Video chunks are encrypted using CTR mode (cenc) for DASH, Google Widevine, and Microsoft PlayReady, or CBC mode (cbcs) for HLS and Apple FairPlay. Protection System Specific Header (PSSH) metadata is written directly into media container headers. Authenticated client players exchange session entitlement tokens with a dedicated DRM license server to acquire decryption keys before decoding frames.

CDN and Edge Delivery Architecture

Edge caching is the foundational layer that makes low-latency global video streaming economically viable by absorbing over 95% of traffic before it reaches origin storage.

• Tiered Caching: Popular videos in the top 20% are preferentially retained across edge PoPs based on access frequency, LRU eviction policies, and cache TTL headers, serving over 80% of streaming bytes directly from edge memory and NVMe drives.
• Warm Content Handling: Less frequently viewed content is fetched from origin upon first request and cached at edge PoPs with standard time-to-live intervals.
• Long-Tail Content: Rarely accessed archive assets stream from origin storage through pass-through streaming to prevent cache thrashing and preserve edge capacity for active titles.
• Global Edge Footprint: Over 200 edge locations (Points of Presence) terminate TLS connections close to viewers worldwide, minimizing round-trip latency.
• Origin Shielding and Request Coalescing: An intermediate centralized caching layer aggregates regional edge cache misses. Origin shield nodes coalesce concurrent requests for identical missing segments into one or a small number of upstream origin fetches, protecting origin storage from thundering herds.
• Cache Ingestion Modes: Highly anticipated releases are proactively pre-warmed to regional edge PoPs, while catalog titles use lazy pull-on-demand fetching.

Recommendation Service Integration

Recommendation integration supplies personalized candidate feeds and ingests watch telemetry asynchronously, while machine learning ranking model internals and offline training pipelines remain out of scope.

• Collaborative Filtering: Analyzes cross-user consumption patterns to identify videos frequently co-watched across similar user cohorts.
• Content-Based Filtering: Recommends media sharing similar metadata attributes including genre, creator, tags, language, and duration.
• Deep Learning Embeddings: Ingests audio and video frame embeddings to discover latent visual and acoustic similarities across catalog items.
• Low-Latency Serving: Pre-computes candidate recommendations using offline Spark pipelines, hydrates results into Redis clusters, and serves recommendations to clients in under 50 milliseconds.

View Count Aggregation Pipeline

View count aggregation buffers high-velocity telemetry writes through Kafka and Apache Flink so viral video playback surges never overwhelm database counters.

For this design, a qualified/countable view is defined as continuous playback exceeding 30 seconds or 25% of total video duration (whichever is shorter), filtering out accidental clicks and immediate bounces. The client video player transmits lightweight telemetry beacons every 10 seconds to a stateless ingestion API that forwards events directly to the view-events Kafka topic. The topic uses composite partition keys (video ID combined with a shard bucket index from 0 to 99) to prevent viral videos from overloading individual Kafka brokers. Downstream Apache Flink applications process partition streams across tumbling windows, deduplicating records by user ID and video ID across 24-hour sliding windows. Flink flushes 1-minute aggregated increments to Redis via atomic INCRBY commands against views:{video_id}, providing an approximate serving counter for frontend display. Authoritative view events are written durably to Cassandra and ClickHouse. An hourly reconciliation batch job recomputes deduplicated totals from ClickHouse, synchronizing MySQL display counters and resetting Redis keys to prevent counter drift.

DAG Scheduler and Workflow Orchestration

The workflow orchestrator coordinates complex task dependencies across distributed worker pools using durable state machines.

The video processing pipeline involves complex dependencies naturally structured as a Directed Acyclic Graph (DAG). Production architectures employ durable workflow engines like Temporal or cadence-based distributed orchestrators with built-in leader election, state checkpointing, and automatic worker failover rather than single-point-of-failure leaders. Independent tasks such as audio extraction, thumbnail generation, and multi-resolution chunk transcoding run concurrently across GPU workers, while the packager waits for all constituent video and audio tracks to complete. Every task is idempotent based on composite keys of video ID, rendition name, and chunk index. If a 1080p transcode task experiences a transient failure while 240p through 720p succeed, the orchestrator retries only the 1080p task up to 3 times via a dead-letter queue. If product policy permits graceful degradation, the platform publishes a baseline manifest containing the available renditions so viewers can begin streaming immediately, updating the master manifest once the 1080p task finishes.

Search Service (Elasticsearch)

Video search leverages inverted index architectures established in Search Engine, indexing rich textual metadata rather than raw video bytes.

• Indexed Documents: Inverted indices cover video titles, full descriptions, creator tags, channel identifiers, and timestamped speech-to-text transcripts.
• Query Capabilities: Supports full-text search evaluated via BM25 relevance scoring, n-gram prefix autocomplete suggestions, and edit-distance fuzzy matching for spelling mistakes.
• Asynchronous Synchronization: Metadata mutations committed to MySQL are captured via Debezium Change Data Capture (CDC), streamed through Kafka, and indexed into Elasticsearch within 2 seconds of publication.
• Tombstone Processing: When a video is soft-deleted in MySQL, CDC propagates a deletion event that purges the document from Elasticsearch, while playback authorization checks verify current database status.

Event Bus Design (Kafka)

The distributed event bus decouples producers from asynchronous consumer fleets and buffers high-volume traffic bursts.

# Kafka Event Bus Architecture for Video Ingestion and Telemetry
topics:
  video-uploaded:
    partitions: 64 # Partitioned by video_id: preserves sequential job dispatch per video
    retention_days: 7
    producers:
      - "Upload Service via MySQL Transactional Outbox + Debezium CDC (eliminates dual-write failure window)"
    consumers:
      - "DAG Scheduler / transcode worker fleet (initiates distributed transcode pipeline)"

  video-ready:
    partitions: 64 # Partitioned by video_id
    retention_days: 7
    producers:
      - "HLS/DASH packaging task (emitted only after all baseline renditions and manifests are durably committed in S3 and MySQL)"
    consumers:
      - "Video Metadata cache warmers"
      - "Elasticsearch indexer"
      - "CDN pre-warm worker"

  view-events:
    partitions: 512 # Partitioned by composite key (video_id:shard_bucket) to prevent viral video broker hotspots
    retention_hours: 24 # High-volume telemetry stream (2.5M events/sec peak)
    producers:
      - "Stateless Playback Beacon Ingestion API (client heartbeats sent every 10 seconds)"
    consumers:
      - "Apache Flink streaming job (deduplicates views, flushes approximate counts to Redis INCRBY, and writes durable records to Cassandra/ClickHouse)"
    deduplication: "Qualified view after 30 seconds watch time or 25% duration; unique view per (user_id, video_id) per 24-hour sliding window"

  search-index-updates:
    partitions: 32 # Partitioned by video_id
    producers:
      - "Debezium CDC connector on MySQL videos table (captures inserts, metadata updates, and deletion tombstones)"
    consumers:
      - "Elasticsearch synchronizer (asynchronously indexes or deletes documents within 2 seconds)"

operational_guarantees:
  replication_factor: 3
  min_insync_replicas: 2
  synchronous_path: "Upload negotiation -> S3 multipart direct upload -> atomic MySQL transaction (videos row + outbox_events row) -> HTTP 200/201"
  asynchronous_path: "MySQL CDC -> Kafka video-uploaded -> transcode pipeline -> S3 packaging -> video-ready -> search and CDN; playback beacons -> view-events -> Flink -> Redis and ClickHouse"