Design a Podcast Delivery Platform

1. Audio Processing Pipeline

We start with the audio processing pipeline because every downstream surface, including RSS syndication, native app playback, and dynamic ad insertion, depends on normalized, multi-bitrate files landing in object storage.

Creators upload raw audio (WAV, FLAC, MP3). The pipeline normalizes loudness to industry standards, trims silence, transcodes into multiple formats and bitrates, and generates waveforms, chapters, and optional transcripts before anything is published.

Step 1: Validate + Probe (FFprobe: extract duration, format, code, reject if >12 hrs or >2GB)
Step 2: Normalize Audio
  - Normalize loudness: target -16 LUFS (loudness standard for podcasts)
    ffmpeg -i input.wav -af "loudnorm=I=-16:TP=-1.5:LRA=11" normalized.wav
  - Silence trimming: remove > 3 seconds of silence at start/end
Step 3: Transcode to Multiple Formats/Bitrates
  - MP3 128 kbps: Universal compatibility (RSS feed reference)
  - AAC 128 kbps: iOS/Android native high quality
  - Opus 48 kbps: Incredible compression for modern clients (50% smaller than MP3!)
Step 4: Chapter Markers (Embed in ID3/M4A tags: { title, start_time, end_time })
Step 5: Generate Waveform (RMS amplitude per 100ms for custom seekbars, ~50KB JSON)
Step 6: Speech-to-Text Transcription (Whisper API, cost-optimized: only run for shows with >100 subs)

2. RSS Feed Service: The Core Distribution Mechanism

Podcasts are distributed via RSS. Every external aggregator (Spotify, Apple, Overcast) polls RSS feeds constantly.

Feed serving at scale:
  5M shows x polled every 15-30 minutes by 10+ aggregators = ~30M feed requests/hour
  
Strategy:
  1. Pre-generate RSS XML for each show → store in S3
  2. Serve via CDN with 15-minute TTL
  3. On new episode publish: regenerate feed XML → invalidate CDN cache
  4. Conditional requests: ETag/If-Modified-Since → 304 Not Modified (saves 90% bandwidth)
  
Stable RSS URL format: https://feeds.example.com/shows/{show_id}/rss

3. Playback Sync: Resume Across Devices

Syncs playback position dynamically, allowing a seamless transition from phone commute to desktop browser.

Sync mechanism:
  Client reports position every 30 seconds:
    POST /api/v1/playback/progress  { episode_id, position_seconds, speed }
  On opening:
    GET /api/v1/playback/progress/{episode_id}  → resumes from stored point

Storage: Redis
  Key: playback:{user_id}:{episode_id}
  Value: Hash { position, speed, duration, updated_at }
  TTL: 90 days (auto-cleanup old progress)

Scale: 30M DAU x update every 30 sec = ~100K writes/sec (easily handled by 10 Redis Cluster shards)

4. Dynamic Ad Insertion (DAI): The Revenue Engine

Rather than relying on static baked-in ads, Server-Side Ad Insertion (SSAI) stitches targeted commercial spots into audio streams dynamically at request time.

SSAI Splicing Flow:
  1. Creator marks ad breaks: { "breaks": [{"position": 0, "type": "pre-roll"}, {"position": 1200, "type": "mid-roll"}] }
  2. On request, Ad Decision Service evaluates demographics, frequency caps, and targets ads.
  3. Audio Stitching Service:
     - Splicing HLS segments dynamically on edge.
     - Segmented playlist: [segment_pre, ad_1, segment_mid, ad_2]
     - Allows pre-encoded segments cached on CDN separately. No real-time heavy CPU re-encoding.

Ad Impression Tracking:
  Client fires event when passing ad bounds: { ad_id, event: "impression|start|50%|complete" } → Kafka → ClickHouse

Event Bus Design (Kafka)

Topic: episode-uploaded
  Partitions: 64
  Partition key: show_id (serialize processing per podcast show)
  Retention: 7 days
  Replication factor: 3, min.insync.replicas: 2

Producer: Creator Service on S3 upload complete
  Event: { episode_id, show_id, s3_key, format, chapters[] }

Consumer groups:
  1. transcode-pipeline: normalize (-16 LUFS), transcode to MP3, AAC, and Opus, and generate waveform JSON
  2. rss-regenerator: rebuild RSS XML, purge CDN cache, and ping aggregator webhooks
  3. subscriber-notifier: push new episode notifications to subscribed listeners
  4. stt-pipeline: Whisper transcription and Elasticsearch full-text indexing

Topic: playback-events (play, seek, complete) for ClickHouse analytics and IAB metrics
Topic: ad-events (impression, 50%, complete) for monetization reporting
Topic: download-events for compliance filters in dynamic ad insertion

Sync path: upload ACK and episode status set to processing, with asynchronous publishing
DLQ: episode-uploaded-dlq after 3 retries