Design a Video Transcoding Pipeline

Segment-Based Parallel Transcoding: The Core Optimization

Parallelism comes from segmenting before encoding, allowing the orchestrator to fan out hundreds of small FFmpeg tasks instead of a handful of long ones.

Why split into segments? Without splitting: 1 video x 6 resolutions = 6 tasks at 20 minutes each = 20 min wall-clock. With splitting (10-second segments): 60 segments x 6 resolutions = 360 tasks at ~3 seconds each = ~30 seconds wall-clock with 40 workers.

GOP-aligned splitting: Must split at GOP boundaries (I-frame positions). FFmpeg: ffmpeg -i input.mp4 -c copy -f segment -segment_time 10 -reset_timestamps 1 segment_%03d.mp4. The -c copy flag ensures no re-encoding during split (fast, lossless).

Reassembly: FFmpeg concat demuxer. For HLS, segments ARE the final output: no reassembly needed! Just generate the .m3u8 playlist.

FFmpeg Command Breakdown

H.264 (CPU)

ffmpeg -i segment_005.mp4 -vf "scale=1280:720" -c:v libx264 -preset medium -crf 23 -profile:v high -level 4.0 -maxrate 3M -bufsize 6M -g 48 -sc_threshold 0 -an segment_005_720p.ts

H.265 (GPU NVENC)

ffmpeg -i segment_005.mp4 -vf "scale=1280:720" -c:v hevc_nvenc -preset p5 -rc:v vbr -cq 28 -maxrate 1.5M -bufsize 3M -g 48 -tag:v hvc1 segment_005_720p_h265.ts

Pipeline Orchestrator: Temporal/Step Functions

Kafka consumers alone can't express complex DAG dependencies. Temporal ⭐ provides DAG definition, per-step retries with exponential backoff, workflow state persistence, visibility dashboard, timeouts, and versioning. Each step has a retry policy with configurable initial interval, maximum interval, maximum attempts, and non-retryable errors.

@workflow
def transcode_pipeline(video_id, s3_key):
    probe_result = await probe_video(s3_key)
    segments = await split_video(s3_key, probe_result)
    resolutions = determine_resolutions(probe_result)
    transcode_futures = []
    for res in resolutions:
        for segment in segments:
            future = transcode_segment.async(segment, res, 'h264')
            transcode_futures.append(future)
    transcoded = await all(transcode_futures)
    manifest = await package_hls_dash(transcoded, video_id)
    await all(
        generate_thumbnails(s3_key, probe_result),
        extract_audio(s3_key),
        content_moderation(s3_key),
        generate_subtitles(s3_key)
    )
    await encrypt_drm(manifest, video_id)
    await update_video_status(video_id, 'ready')

Event Bus Design (Kafka)

Topic: video-uploaded
  Partitions: 128 (scale worker pools independently)
  Partition key: video_id (preserves per-video job ordering)
  Retention: 7 days (replay failed transcodes)
  Replication factor: 3, min.insync.replicas: 2

Producer: S3 event notification → API enriches with { video_id, s3_key, codec_hint, priority }
Consumer groups:
  1. orchestrator: start Temporal DAG (probe → GOP split → parallel transcode → package)
  2. status-updater: UPDATE videos SET status=processing|ready|failed in MySQL
  3. cdn-purger: invalidate CDN manifest on transcode-complete

Topic: transcode-progress (internal): per-segment completion for DAG resume
Topic: transcode-failed-dlq: poison jobs after 3 retries; alert ops

Sync path: presigned upload ACK < 200ms; processing is fully async
Async path: workers pull segment jobs from CPU/GPU pools; idempotent by segment_id
Lag alert: consumer lag > 300s → scale GPU pool