Design a Video Conferencing System (like Zoom)

Overview

Video conferencing has two distinct planes. The signaling plane manages session state, while the media plane carries real time RTP forwarding. Signaling does not carry video packets, and the SFU forwards media without decoding or encoding. The sections below cover join flow, NAT traversal, breakout rooms, PSTN bridging, topology choices, and network adaptation.

Signaling Server: Session Orchestration

Signaling is entirely separate from media transport. Clients open a persistent WebSocket to the signaling cluster for join and leave events, SDP offer and answer exchange, ICE candidate trickling, and meeting controls. At 20M signals/sec the signaling tier is stateful but lightweight because it never touches RTP packets. Redis holds ephemeral meeting state such as the participant list, SFU assignment, and TURN credentials with a short TTL that is refreshed while the meeting is active.

The join flow starts with a REST POST /join call. The response contains a signaling token, an assigned SFU endpoint, and ICE server configuration. The client then opens the WebSocket, sends a join message with its SDP offer, and receives an SDP answer through signaling. ICE candidates are exchanged in both directions until the media path is established over UDP/SRTP or a TURN relay.

NAT Traversal: ICE, STUN, and TURN

After signaling exchanges SDP, the client and SFU establish the media path through ICE. STUN provides server reflexive candidates so the endpoints can discover reachable public addresses. ICE connectivity checks select a viable candidate pair. When direct connectivity fails, especially behind restrictive or symmetric NAT, the client falls back to a TURN relay. TURN allocation state is kept with short lived credentials so relay access is scoped to the meeting.

The scaling concern is relay bandwidth. If NAT traversal fails for 30% of participants, TURN relays 15M streams at 3.5 Mbps each, which is about 52.5 Tbps through relay infrastructure, rounded to roughly 52 Tbps at the stated planning precision. Deploy TURN at edge PoPs, prefer SFU direct paths whenever possible, and scale relay capacity horizontally.

Breakout Rooms

Breakout rooms are logical submeetings spawned from a parent webinar. The host assigns participants to room buckets. Signaling updates each client's meeting_id and triggers an ICE restart on a new SFU shard. Parent meeting state is preserved in Redis so participants can be recalled atomically. Recording and chat are scoped to the active room. Returning to the main session is another signaling-driven SFU reassignment, not a full page reload.

Host: POST /api/meetings/{id}/breakout
  { "rooms": 4, "assignment": "manual" | "random", "duration_minutes": 15 }

Signaling broadcasts to each participant:
  { "type": "breakout_assign", "room_id": "br-2", "sfu_endpoint": "sfu-us-east-7...", "token": "..." }

On recall:
  { "type": "breakout_close", "return_sfu": "sfu-us-east-3...", "main_meeting_id": "m_123" }

State in Redis:
  breakout:{parent_meeting_id} → Hash { room_id → [user_ids], timer_expires_at }

PSTN Dial-In (SIP Gateway)

Phone participants cannot run WebRTC in a browser. They connect through a SIP/PSTN media gateway that bridges the PSTN leg to the SFU as a synthetic participant. The gateway decodes G.711 audio from the phone network, transcodes it to Opus, and injects it into the SFU media path. Dial-in numbers map to meeting PINs, and DTMF tones authenticate the caller. This is where MCU-style processing can appear in an SFU-first architecture. Only the phone leg needs transcoding rather than the entire meeting.

PSTN join flow:
  1. User dials +1-800-xxx-xxxx, enters meeting PIN via DTMF
  2. SIP gateway validates PIN → looks up meeting_id in PostgreSQL
  3. Gateway registers as synthetic participant "Phone User" on SFU
  4. Audio: PSTN (G.711 64 Kbps) ↔ gateway ↔ Opus ↔ SFU ↔ other participants

Capacity: ~5K concurrent PSTN legs per gateway cluster. Scale horizontally by region.
Latency budget: PSTN adds ~100-200ms one-way vs native WebRTC

Why SFU Over MCU and Mesh?

Topology choice drives both cost and latency. Mesh works for 1:1, but peer connections grow as N². MCU reduces each client to one upload and one download stream, but the server must decode and re-encode the participants. SFU forwards packets without transcoding, which allows per-subscriber quality selection.

SFU is attractive for group calls because the server forwards packets without decoding or encoding. Simulcast lets each receiver get an appropriate quality layer, and the SFU can forward selected streams rather than every available stream. In comparable workloads, this can use roughly 10x less media processing CPU than an MCU, although the exact ratio depends on codecs and meeting size.

Simulcast & SVC

Receivers on 3G and gigabit fiber cannot share one bitrate. Simulcast sends three independent encodes (180p / 720p / 1080p), and the SFU picks the right layer per subscriber. SVC achieves a similar outcome with a single layered bitstream. It can use bandwidth more efficiently, but it is harder to implement consistently across codec vendors.

Simulcast: Sender encodes 3 independent streams (High: 1080p @ 2.5 Mbps, Medium: 720p @ 1 Mbps, Low: 180p @ 150 Kbps). The SFU selects a layer for each receiver. An active speaker can receive High from the speaker and Low from others. Gallery view can receive Medium from all, and mobile on 3G can receive Low.

SVC: Single layered stream (base + enhancement layers). SFU drops upper layers for constrained receivers. Advantage: single encode, seamless quality transitions. VP9 and AV1 can support SVC. Product deployments vary by codec and platform.

SFU Architecture & Cascading

Each meeting pins to a primary SFU with a hot standby that already holds participant state. For global meetings, regional SFU nodes cascade streams so each video track crosses a region boundary exactly once. In the representative scenario below, this results in 30 Mbps of inter-region relay versus 870 Mbps if every participant connected directly.

Each meeting has a primary SFU and a hot standby. The standby receives the participant list and ICE candidates from signaling. On primary failure, signaling detects the heartbeat miss in < 3 sec, signals clients to reconnect to the backup, and clients perform an ICE restart in about 1 to 2 sec.

Multi-region cascading: A meeting with participants in US-East, EU-West, and APAC uses SFU-US-East, SFU-EU-West, and SFU-APAC. Each region fans out locally, so each stream crosses a region boundary once through an SFU to SFU relay. Bandwidth is 30 Mbps inter-region versus 870 Mbps if all participants connected directly, which is a 29x saving.

Network Quality Adaptation

Video quality must degrade gracefully before the call drops entirely. Clients monitor packet loss and RTT and then step down resolution when network conditions deteriorate. FEC adds redundant packets for recovery without retransmission. NACK requests retransmission from the SFU jitter buffer only when RTT stays under 100 ms.

Client monitors packet loss, RTT, and available bandwidth. Loss < 2% maps to 1080p 30fps. Loss 2-5% maps to 720p. Loss 5-10% maps to 360p 15fps. Loss > 10% maps to audio-only. FEC sends redundant packets (10% overhead) for recovery without retransmission. NACK requests retransmission from the SFU jitter buffer and is useful only if RTT < 100ms.

Large Meeting Optimization (1,000 participants)

A thousand WebRTC connections per client is impractical. Webinar mode keeps a small set of active speakers on the interactive SFU path, while passive viewers consume a CDN HLS stream with a 5 to 10 second delay. Viewers can be promoted to the interactive speaker role without changing the overall architecture.

Tiered architecture: Tier 1 Speakers (5-10) use full SFU send and receive video. Tier 2 Viewers receive the webinar stream through CDN HLS with a 5-10s delay and can be promoted to an interactive speaker role when needed. In the representative 1,000 participant case, keeping about 10 speakers on WebRTC leaves about 990 viewers on CDN delivery.