Interview Setup
Interview Prompt
Design Google Docs: real-time collaborative document editing where multiple users type simultaneously, see each other's cursors, and converge to the same document state. 50M DAU, 500K operations/sec globally.
Clarifying Questions (ask before designing)
| Question | Why it matters |
|---|---|
| OT or CRDT, and does offline editing matter? | OT needs central server ordering. CRDT can merge peer-to-peer on reconnect. Google Docs used OT, while Figma and Yjs use CRDT. |
| How many simultaneous editors per document is typical vs max? | 2 to 5 typical, 200 max. Above 200, switch to view-only mode or batch presence updates. |
| Must undo/redo work across collaborative edits? | Per-user undo stack is isolated from others' ops. Requires operational transform on the undo stack too. |
| What's the document size limit and how does it affect op log storage? | 50 KB text + 500 KB embeds average. 500 TB version history with delta compression over 30 days. |
Scope
In scope
- OT (Operational Transformation) vs CRDT
- Cursor position sync
- Version vectors
- WebSocket session management
- Conflict-free merges
- Undo/redo in collaborative context
Out of scope (state explicitly)
- Client desktop/mobile app implementation
- End-user file preview rendering for every format
- Building raw block storage hardware
Functional Requirements
Start by asking your interviewer which conflict resolution approach is expected, specifically OT or CRDT. Confirm offline editing, presence cursors, and version history scope for this round. The real-time transport patterns are covered in Network Protocols: HTTP, gRPC, WebSocket & DNS.
- Multiple users can simultaneously edit the same document in real time
- Each user sees a live cursor and selections of other collaborators
- Changes appear on all clients within 100 to 200ms of being made
- Full version history with the ability to view and restore any retained past version
- Commenting and suggestion mode (track changes)
- Offline editing with automatic conflict resolution on reconnect
- Rich text formatting: bold, italic, headings, lists, tables, images
- Permissions: owner, editor, commenter, viewer
- Document sharing via link with configurable access levels
- Export to PDF, DOCX, plain text
Non-Functional Requirements
Your interviewer will care most about convergence guarantees and sub-200 ms remote edit latency. Call out OT vs CRDT early because it determines whether the design needs a single ordering authority.
- Low Latency: Local keystrokes are reflected instantly. Remote changes become visible within 100 to 200ms
- Consistency: All clients must eventually converge to the same document state. A CRDT can provide strong eventual consistency when its merge and delivery assumptions hold
- Availability: 99.99%. Editing must continue even if some servers are down
- Scalability: Support 500M documents and up to 200 concurrent editors per actively coedited document
- Durability: Zero acknowledged edit loss under the stated failure model. An edit is accepted only after the operation log confirms durable quorum storage
- Conflict Resolution: Concurrent edits must merge automatically without user intervention
- Offline Support: Edits queued locally and synced on reconnect
- Security: Encryption in transit (TLS) and at rest, RBAC, audit logs
Capacity Estimations
Run this math before assigning document servers. Concurrent editors per actively coedited document and operations per second determine WebSocket fan-out, while snapshot frequency drives storage growth. Treat the 10M active-document assumption as open or viewed documents, because not every active document is simultaneously coedited. The separate 20M WebSocket assumption captures concurrently connected users.
| Metric | Calculation | Value |
|---|---|---|
| Total documents | Given (assumption documented in value) | 500M |
| DAU | Given (product assumption) | 50M users |
| Concurrently active docs | Given. Peak load assumption for open or viewed documents | 10M |
| Editors per actively coedited doc | Given. Typical collaboration assumption | 2-5 (up to 200 max) |
| Operations/sec (global) | From Operations/day ÷ 86400 (+ peak factor in value) | 500K |
| Avg document size | Given. Typical workload assumption | 50 KB (text) + 500 KB (images/embeds) |
| Storage (docs) | 500M x 550 KB | ~275 TB |
| Version history (30 days) | Given | ~500 TB (with delta compression) |
| WebSocket connections | Given. Concurrent editor connections are separate from the 10M active or viewed documents | 20M concurrent |
Architecture Diagram
In the room, ask about OT vs CRDT before drawing. This is the architectural fork that determines the conflict resolution design.
Walk your interviewer through the per-document server model first. For the OT path shown here, one collaboration server per active document owns operation ordering. Clients send operations over WebSocket. The server transforms concurrent edits, persists durable operations and snapshots, and broadcasts the resulting document state to connected peers.
Core Design Decision: OT vs CRDT
| Aspect | OT (Operational Transformation) | CRDT (Conflict-free Replicated Data Type) |
|---|---|---|
| How it works | Transform concurrent ops relative to shared server sequence | Each op carries unique IDs. Merge is commutative, associative, and idempotent |
| Single ordering server required? | Yes: central server assigns canonical order | No. Replicas can merge without one ordering authority. Servers can still provide relay and persistence |
| Offline support | Harder (ops must be rebased) | Natural (merge on reconnect) |
| Used by | Google Docs (original), SharePoint | Figma, Yjs, Automerge, Apple Notes |
| Design choice | Useful when a central ordering server is acceptable | Useful when offline editing and decentralized merging are priorities |
Collaboration Server: Why One Per Document
- Each actively coedited document is assigned to exactly ONE collaboration server
- Avoids distributed coordination for operation ordering on the OT path
- The collaboration server holds actively coedited document state in memory for fast operation processing
- If the server dies, another server loads the latest snapshot and replays the durable operation log
- ZooKeeper owns the document assignment lease. Gateways use the cached assignment or consistent-hash lookup to route to that server
Document Session Lifecycle
1. User opens doc → authenticate and authorize → WebSocket Gateway routes to assigned Collaboration Server
2. If doc not in memory → Load latest snapshot from PostgreSQL
→ Replay ops from Cassandra since snapshot
→ Build in-memory state
3. User types → Client generates op → Sends via WebSocket
4. Server transforms or merges → assigns seq_num → durably persists to the operation log → acknowledges → broadcasts
5. Periodically (every 100 ops or 30 sec) → Write snapshot to PostgreSQL
6. Last user leaves → After 5 min idle, evict doc from memoryComponent Deep Dives
OT vs CRDT: Merge Semantics
Start with this architectural fork because it determines how concurrent operations are ordered, merged, and synchronized offline. State the choice for rich text and explain the implications for offline support.
OT transforms concurrent operations against a canonical server order. It usually relies on a central collaboration server and requires rebasing for offline edits. CRDT operations carry identifiers scoped to a replica and causal metadata. Its merge rules support offline work without requiring a single ordering authority, although metadata and garbage collection add storage complexity.
Collaboration Server Memory Model
Per-document server memory and eviction policy matter at scale. Explain snapshot before eviction so a cold start can rebuild the document without replaying the full operation history.
In-memory: document tree + op log tail (last N ops for fast catch-up) On cold start: PostgreSQL snapshot + replay Cassandra ops since snapshot_index Hot doc (200 editors): batch ops every 50ms, and send presence updates every 100ms Eviction: LRU after 5 min idle, taking a snapshot before evict
Cursor & Presence Sync
Cursor positions are ephemeral presence updates rather than document operations, so they are not persisted to the operation log. Throttle to 10 updates/sec per user. Use CRDT-style position identifiers rather than raw offsets so cursors stay valid after concurrent inserts.
Permissions on the Real-Time Channel
Authenticate the user before opening the WebSocket and authorize the requested document role. Cache permissions in Redis for the stated 5 minute TTL, but invalidate the cache when sharing permissions change. The collaboration server checks editor permission before accepting a mutation. Suggestion mode stores proposed changes separately until an authorized editor accepts or rejects them.
Offline to Reconnect Merge
The client buffers operations in IndexedDB while offline. On reconnect, it sends buffered operations with client_seq and client_id. For OT, the server rebases those operations against the current document version and returns transformed operations. For CRDT, the peers exchange missing operations using causal metadata and merge them deterministically. The selected merge algorithm resolves ordinary text conflicts automatically, so users do not need a manual merge dialog.
API Design
WebSocket Protocol (bidirectional)
Use stable domain identifiers in the application model. The wire example below keeps the operation payload compact and uses integer positions as an OT-style illustration. A CRDT implementation should carry stable position identifiers rather than byte offsets. The server returns the acknowledgement only after the accepted operation reaches the configured durable write quorum.
type DocId = string;
type ClientId = string;
type UserId = string;
type DocumentOperation =
| { type: "insert"; pos: number; content: string; attrs?: Record<string, unknown> }
| { type: "delete"; pos: number; len: number }
| { type: "format"; pos: number; len: number; attrs: Record<string, unknown> };
interface OperationRequest {
type: "operation";
docId: DocId;
clientId: ClientId;
clientSeq: number;
baseServerSeq?: number; // required for OT, omitted for CRDT
ops: DocumentOperation[];
}
interface OperationAck {
type: "ack";
clientSeq: number;
serverSeq: number;
}
interface CursorUpdate {
type: "cursor";
userId: UserId;
positionId: string;
selectionStartId?: string;
selectionEndId?: string;
}// Client → Server: operation based on the last observed server sequence
{
"type": "operation",
"doc_id": "d_abc123",
"client_id": "c_789",
"client_seq": 42,
"base_server_seq": 100,
"ops": [
{ "type": "insert", "pos": 15, "content": "Hello", "attrs": {"bold": true} },
{ "type": "delete", "pos": 10, "len": 3 },
{ "type": "format", "pos": 5, "len": 10, "attrs": {"italic": true} }
]
}
// Server → Client (acknowledgement)
{ "type": "ack", "client_seq": 42, "server_seq": 105 }
// Presence updates
{ "type": "cursor", "user_id": "u_xyz", "cursor": { "position_id": "p_42", "selection": { "start_id": "p_42", "end_id": "p_50" } } }REST APIs
POST /api/docs → Create new document
GET /api/docs/{doc_id} → Get document
DELETE /api/docs/{doc_id} → Delete document (soft delete)
GET /api/docs/{doc_id}/history → Get version history
GET /api/docs/{doc_id}/version/{v} → Get doc at specific version
POST /api/docs/{doc_id}/restore/{v} → Restore to version v
POST /api/docs/{doc_id}/share → Update sharing permissions
POST /api/docs/{doc_id}/export?fmt=pdf → Export documentCommon Error Responses
400 Bad Request: invalid input, missing required fields, or malformed JSON payload 401 Unauthorized: missing or invalid authentication token or API key 403 Forbidden: authenticated caller lacks required permissions for this resource 404 Not Found: requested resource ID does not exist 409 Conflict: duplicate write or version conflict, retry with a unique idempotency key 422 Unprocessable Entity: syntactically valid request failed semantic business validation 429 Too Many Requests: rate limit quota exceeded, client should honor Retry-After header 500 Internal Error: unexpected server failure, retry safely with an idempotency key 503 Service Unavailable: downstream dependency is unavailable or overloaded, retry with exponential backoff
Data Model
Operation Log (Cassandra / ScyllaDB)
Table: operation_log Partition Key: (doc_id, seq_bucket) Clustering Key: seq_num (ASC) seq_bucket BIGINT -- floor(seq_num / 1,000,000) doc_id TEXT seq_num BIGINT user_id TEXT op_type TEXT -- 'insert' | 'delete' | 'format' op_data BLOB -- Serialized operation (protobuf) timestamp TIMESTAMP client_id TEXT client_seq INT # Idempotency key: (doc_id, client_id, client_seq)
Bucket the operation log by document and sequence range so a heavily edited document does not create an unbounded Cassandra partition. With 1M operations per bucket, a 1000 ops/sec hot document creates a new bucket roughly every 1000 seconds. Recovery starts at the snapshot sequence and reads only the required buckets.
Document Snapshots (PostgreSQL)
CREATE TABLE documents (
doc_id UUID PRIMARY KEY,
title TEXT,
owner_id UUID,
content_snapshot JSONB,
snapshot_seq BIGINT,
created_at TIMESTAMPTZ,
updated_at TIMESTAMPTZ,
is_deleted BOOLEAN DEFAULT FALSE
);Redis: Active Sessions
# Active document sessions
HSET doc:session:{doc_id} client:{client_id} '{"user_id":"u_xyz","cursor":"p_42","connected_at":"..."}'
# Permission cache (TTL 5 min)
HSET doc:perms:{doc_id} user:{user_id} "editor"
# Cached copy of the ZooKeeper document assignment lease
# Gateway refreshes this TTL while the document remains active
SET doc:server:{doc_id} "collab-server-17" EX 300Fault Tolerance
| Concern | Solution |
|---|---|
| Collaboration server failure | Failover uses ZooKeeper. The new server loads the latest snapshot, replays the operation log, and clients reconnect with exponential backoff |
| Conflict resolution | Correctly implemented OT or CRDT provides convergence, while the durable operation log prevents loss of acknowledged operations |
| Offline edits | Edits are buffered in IndexedDB and merged through the selected OT or CRDT algorithm on reconnect |
| Data durability | Cassandra with RF=3 stores acknowledged operations with quorum durability, and periodic snapshots are written to PostgreSQL for faster recovery |
| Hot document overload | Batch presence updates and operations, then switch to view-only mode above 200 editors |
| Duplicate operation delivery | The idempotency key combines doc_id, client_id, and client_seq. Server-side deduplication prevents a retried operation from being applied twice |
Collaboration Server Failure
1. WebSocket Gateway detects broken connection (heartbeat) 2. Gateway triggers failover: assigns doc to another collab server via ZooKeeper 3. New server loads latest snapshot from PostgreSQL 4. Replays all durable operations from Cassandra where seq_num > snapshot_seq 5. Clients reconnect (WebSocket auto-reconnect with exponential backoff) 6. Clients send their pending ops (buffered locally) 7. New server transforms and applies pending client ops Recovery time: < 5 seconds
Conflict Resolution (OT Deep Dive)
Scenario: User A inserts "X" at position 5, User B deletes at position 3.
With OT:
Server receives A's op first → applies insert("X", 5) → seq 101
Server receives B's op (based on seq 100) → must transform:
B's delete(3) is transformed against A's insert(5)
Since 3 < 5: delete(3) stays at 3
BUT A's insert(5) vs B's delete(3): since 5 > 3, adjust to insert(4)
Both clients converge to same state ✓CRDT Deep Dive
Using RGA (Replicated Growable Array): Each character = (uniqueID, value, isDeleted, parentID) UniqueID = (siteID, lamportClock), which is unique per replica and can be deterministically ordered Two inserts same parent: order by uniqueID (siteID tiebreak) Result: deterministic on ALL replicas with no single ordering server needed Tombstone GC: Deleted chars kept as tombstones Periodic compaction: if all replicas have seen delete, remove tombstone
Additional Considerations
Collaborative Undo/Redo
Maintain a per-user undo stack that records that user's intent rather than a copy of the entire document. Undo creates a new inverse operation against the current document state, so it does not erase edits made by other users. Permission checks still apply to the generated operation.
1. User A inserts "X" → append operation to A's undo stack 2. User B edits the same document → document state advances 3. User A presses Undo → create an inverse operation for A's insert 4. OT: transform the inverse against intervening operations before applying 5. CRDT: emit a new operation that reverts A's prior mutation while preserving later edits from others 6. Persist the undo operation in the normal operation log so version history remains durable 7. Redo emits another forward operation. Never restore an old snapshot wholesale
Offline Editing & Sync
1. User goes offline → all edits stored in IndexedDB/SQLite 2. Local CRDT state diverges from the server 3. User comes back online: a. Client sends all buffered ops to the server b. Server sends missing operations using the client's last seen sequence or causal state c. CRDT merge: both sides apply all missing operations, ensuring convergence under the CRDT's merge assumptions d. No manual conflict resolution is needed for ordinary text edits
Version History & Snapshotting
- Snapshot every 100 ops or every 30 seconds
- Operation log kept for 90 days for the stated compliance requirement. The 500 TB capacity estimate covers 30 days of version history
- To view version at time T: find latest snapshot before T, replay ops to T
- Delta compression between adjacent snapshots (~90% reduction)
Real-Time Presence at Scale
Problem: 200 users in a doc, each sending cursor updates at 10/sec Raw inbound updates: 200 × 10 = 2K cursor updates/sec Naive fanout: 2K × 199 other editors ≈ 398K outbound deliveries/sec Solution: - Cursor throttling (client-side): limit each user to one update per 100ms - Cursor batching (server-side): aggregate all 200 positions, broadcast every 100ms - Batched fanout: 10 batches/sec × 199 recipients = 1,990 outbound batch messages/sec - Message count reduction: roughly 200x compared with naive per-update fanout
Interview Walkthrough
- 25-minute cut
Skip arch50/arch75 depth unless staff.
- One collaboration server per active document (6 min)
- WebSocket for operations, with OT or CRDT for merging (7 min)
- Local keystroke instant. Remote op < 200ms (6 min)
- Version history via operation log compaction (6 min)
- Lead with the OT vs CRDT decision as the defining architectural fork. Explain the trade-off using offline requirements, metadata cost, and coordination complexity.
- Explain how concurrent edits converge: OT transforms ops against each other on a central server, CRDT merges ops commutatively without coordination.
- Use WebSocket for real-time operation delivery. Batch cursor and presence updates to avoid roughly 398K outbound cursor deliveries per second when 200 users each send updates at 10/sec.
- Persist an append-only op log plus periodic snapshots, replaying ops from the latest snapshot for version history and crash recovery, integrating patterns from Event Sourcing and CQRS.
- Address offline editing by buffering operations locally and synchronizing on reconnect. Use CRDT causal merge when CRDT is selected, or OT rebasing when OT is selected.
- Frame consistency using CAP Theorem and Consistency Models. Explain that the design remains available during partial failures while convergence and durable writes are preserved under the stated failure model.
- Mention tombstone compaction for CRDT storage growth (~30% overhead after heavy editing).
- Common pitfall: last-write-wins timestamps, where two users typing simultaneously will silently lose one user's edits.
Engineering Trade-offs
OT vs CRDT: The Core Collaborative Editing Decision
Collaborative editing balances convergence guarantees and offline support. OT vs CRDT is the defining trade-off.
Operational Transformation (OT): ✓ Compact representation (small operations) ✓ Works well for text (insert/delete are natural) ✗ A central server is required for canonical ordering ✗ Correctness is notoriously difficult to prove ✗ Complex to implement correctly for rich text CRDT: ✓ Convergence follows from the CRDT data type and its merge assumptions ✓ Can merge without a single central ordering server. Production systems may still use servers for relay and persistence ✓ Easier to reason about correctness ✗ Higher storage overhead (each character has metadata) ✗ Tombstones accumulate: ~30% of storage after heavy editing Design choice: explain both approaches, then choose based on offline requirements, metadata cost, and operational complexity.
Document Snapshot Strategy: Frequency vs Storage Cost
Alternative policy example: Snapshot every 1000 ops: Recovery: at most 999 ops to replay → acceptable (< 1 sec) Storage: manageable Version history checkpoints: - Every 1000 ops - On explicit "Version History" save (user-triggered) - On document sharing (ensure shareable state is snapshotted) Delta compression between snapshots: 60-90% storage reduction for documents with small incremental changes
Review
How helpful was this walkthrough?
Click a star to rate. We actively use this feedback to refine and update our system design content.
Discussion
Share your thoughts, ask questions, or help others.