This problem appears in multiple sheets. Depth expectations increase as you progress:
Interview Prompt
Design Design Dropbox / Google Drive (File Storage + Sync).
Clarifying Questions (ask before designing)
| Question | Why it matters |
|---|---|
| Which of these is highest priority: File chunking & deduplication, Delta sync (rsync), Conflict resolution? | Forces scope negotiation — senior candidates trim before drawing boxes. |
| What scale should we design for — DAU, QPS, data volume? | Drives every capacity decision; shows structured thinking. |
| What are the read vs write patterns on the critical path? | Determines caching, DB choice, and replication topology. |
| What consistency and durability guarantees are required? | Separates strong-consistency paths from eventual ones — a senior differentiator. |
Scope
In scope
- File chunking & deduplication
- Delta sync (rsync)
- Conflict resolution
- Block-level storage
- Metadata DB design
- Notification of changes across devices
Out of scope (state explicitly)
- Client desktop/mobile app implementation
- End-user file preview rendering for every format
- Building raw block storage hardware
Assumptions
These foundational concepts underpin the patterns used in this problem. Review them before deep-diving into component-level trade-offs.
- Upload files: Upload files of any type and size (up to 50 GB)
- Download files: Download any stored file
- Sync: Automatically sync files across multiple devices (desktop, mobile, web)
- File versioning: Maintain version history; rollback to previous versions
- Sharing: Share files/folders with other users via link or email (view/edit permissions)
- Offline access: Work offline; sync changes when reconnected
- Conflict resolution: Handle simultaneous edits of the same file by multiple users
- Notifications: Notify users when shared files are modified
- Folder structure: Support hierarchical folder organization
- Deduplication: Don't store duplicate files/chunks multiple times
- Durability: 99.999999999% (11 nines): files must NEVER be lost
- High Availability: 99.99%
- Consistency: Strong consistency for metadata (file exists or doesn't); eventual consistency for sync propagation
- Low Latency Sync: Changes propagated to other devices within seconds
- Bandwidth Efficiency: Minimize data transfer (send only changed portions of files)
- Scalability: 500M+ users, billions of files, exabytes of storage
- Security: End-to-end encryption, access control
| Metric | Calculation | Value |
|---|---|---|
| Users | Given (assumption documented in value) | 500M |
| Avg files per user | Given (typical workload assumption) | 200 |
| Total files | Given (assumption documented in value) | 100B |
| Avg file size | Given (typical workload assumption) | 500 KB |
| Total storage | 100B × 500 KB | 50 PB |
| With replication (3x) | Given (assumption documented in value) | 150 PB |
| DAU | Given (product assumption) | 100M |
| Files synced / day | Given | 1B |
| Uploads / sec | 1B ÷ 86400 | ~12K |
| Reads / sec | Given | ~50K |
| Avg file change (delta) | Given (typical workload assumption) | 50 KB (10% of file) |
Chunking: The Key Innovation
Why chunk files? Delta sync (if a 1 GB file changes slightly, only upload the changed chunks), Deduplication (identical chunks across files/users stored only once), Parallel upload (multiple chunks simultaneously), Resumable upload (restart from last successful chunk).
Chunking Algorithm: Content-Defined Chunking (CDC) (⭐): Fixed-size chunks have a problem: inserting 1 byte at the beginning shifts all chunk boundaries. CDC uses a rolling hash (Rabin fingerprint) to find content-defined boundaries that are stable: inserting data only affects the chunk where insertion happened. Hash each chunk with SHA-256 for content-addressable storage. Before uploading, check if chunk hash already exists → skip upload.
def chunk_file(data):
chunks = []
window_size = 48 # bytes
min_chunk = 256 * 1024 # 256 KB minimum
max_chunk = 8 * 1024 * 1024 # 8 MB maximum
target_chunk = 4 * 1024 * 1024 # 4 MB target
position = 0
chunk_start = 0
while position < len(data):
hash = rabin_hash(data[position:position+window_size])
chunk_size = position - chunk_start
if (hash % target_chunk == 0 and chunk_size >= min_chunk) or chunk_size >= max_chunk:
chunks.append(data[chunk_start:position])
chunk_start = position
position += 1
chunks.append(data[chunk_start:])
return chunksDesktop Sync Agent (Client-Side)
- File system watcher: Monitors local folder for changes (inotify/FSEvents/ReadDirectoryChangesW)
- On file change: Chunk file using CDC, hash each chunk, compare with known hashes, upload only new chunks to Block Server, send updated metadata to Metadata Service
- On remote change notification: Receive notification, fetch new metadata, compare chunk lists, download only missing/changed chunks, reconstruct file locally
Block Server
- Upload flow: Receive chunk data + SHA-256 hash → verify hash → compress (LZ4/zstd) → encrypt (AES-256) → upload to S3 → return chunk reference
- Download flow: Reverse: fetch from S3 → decrypt → decompress → verify hash → return to client
Metadata Service
- Manages file/folder tree, chunk references, sharing permissions, version history
- Database: PostgreSQL (ACID transactions for file operations)
- Cache: Redis (frequently accessed file metadata)
- On file update: Begin transaction, create new version entry, update chunk list, commit, publish file-changed event to Kafka
Notification Service
- Purpose: Notify other devices and shared users when a file changes
- WebSocket/Long Polling: Each connected client maintains a persistent connection
- Kafka consumer receives file-changed event → looks up all devices/users → push via WebSocket
- Offline devices: When reconnecting, query Metadata Service for all changes since last_sync_timestamp
Conflict Resolution
Dropbox approach (⭐): Save both versions: report.docx and report (Alice's conflicted copy).docx. User manually resolves. Simple, no data loss, user resolves.
Event Bus Design (Kafka)
Topic: dropbox_google_drive-events Partitions: 64 (scale consumers horizontally) Partition key: entity_id (user_id / order_id — preserves per-entity ordering) Retention: 7 days (compliance) or 24h (high-volume telemetry) Replication factor: 3, min.insync.replicas: 2 Producer: idempotent producer enabled (enable.idempotence=true) Consumer: consumer group "dropbox_google_drive-processors" - At-least-once delivery + idempotent handlers (dedup by event_id) - DLQ topic: dropbox_google_drive-events-dlq (poison messages after 3 retries) - Lag alert: consumer lag > 60s → scale workers Design Dropbox / Google Drive (File Storage + Sync): async side effects MUST NOT block the synchronous API response. Sync path: validate → persist source of truth → publish event → return 201 Async path: consumers update caches, indexes, notifications, aggregates
Upload File
POST /api/v1/files/upload/init
{
"file_name": "report.pdf",
"file_size": 10485760,
"parent_folder_id": "folder-uuid",
"chunk_hashes": ["sha256:abc...", "sha256:def...", "sha256:ghi..."]
}
Response: 200 OK
{
"file_id": "file-uuid",
"upload_id": "upload-uuid",
"chunks_needed": ["sha256:def..."], // only chunks not already stored (dedup!)
"upload_urls": {
"sha256:def...": "https://s3.presigned-url..."
}
}Upload Chunk
PUT {presigned_upload_url}
Content-Type: application/octet-stream
Body: [chunk binary data]Complete Upload
POST /api/v1/files/upload/complete
{
"upload_id": "upload-uuid",
"chunk_hashes_ordered": ["sha256:abc...", "sha256:def...", "sha256:ghi..."]
}Download File
GET /api/v1/files/{file_id}/download
Response: 200 OK
{
"chunks": [
{"hash": "sha256:abc...", "url": "https://s3.presigned...", "size": 4194304},
{"hash": "sha256:def...", "url": "https://s3.presigned...", "size": 4194304},
...
]
}Share File
POST /api/v1/files/{file_id}/share
{
"user_email": "alice@example.com",
"permission": "edit" // "view" or "edit"
}Get Changes Since
GET /api/v1/sync/changes?since=1710320000&cursor=abc123Common Error Responses
400 Bad Request: invalid input, missing fields, or malformed JSON 401 Unauthorized: missing or invalid auth token or API key 403 Forbidden: authenticated but insufficient permissions 404 Not Found: resource ID does not exist 409 Conflict: duplicate write or version conflict; retry with idempotency key 422 Unprocessable Entity: valid syntax but invalid business logic 429 Too Many Requests: rate limit exceeded; honor Retry-After header 500 Internal Error: unexpected server fault; retry with idempotency key 503 Service Unavailable: dependency down or overloaded; use exponential backoff
PostgreSQL: File Metadata
CREATE TABLE files (
file_id UUID PRIMARY KEY,
owner_id UUID NOT NULL,
name VARCHAR(256) NOT NULL,
parent_id UUID, -- parent folder
is_folder BOOLEAN DEFAULT FALSE,
size BIGINT,
mime_type VARCHAR(128),
current_version INT DEFAULT 1,
is_deleted BOOLEAN DEFAULT FALSE, -- soft delete
created_at TIMESTAMP,
updated_at TIMESTAMP,
UNIQUE (parent_id, name, owner_id), -- no duplicate names in same folder
INDEX idx_owner (owner_id),
INDEX idx_parent (parent_id)
);PostgreSQL: File Versions
CREATE TABLE file_versions (
version_id UUID PRIMARY KEY,
file_id UUID REFERENCES files,
version_number INT,
size BIGINT,
chunk_hashes TEXT[], -- ordered list of chunk hashes
modified_by UUID,
created_at TIMESTAMP,
UNIQUE (file_id, version_number)
);PostgreSQL: Chunks (Content-Addressable)
CREATE TABLE chunks (
chunk_hash VARCHAR(64) PRIMARY KEY, -- SHA-256
size INT,
storage_path TEXT, -- S3 key
reference_count INT DEFAULT 1, -- for garbage collection
compressed_size INT,
created_at TIMESTAMP
);PostgreSQL: Sharing
CREATE TABLE sharing (
share_id UUID PRIMARY KEY,
file_id UUID REFERENCES files,
shared_with UUID, -- user_id
permission ENUM('view', 'edit'),
shared_by UUID,
created_at TIMESTAMP,
UNIQUE (file_id, shared_with)
);S3: Chunk Storage
Bucket: file-chunks
Key: chunks/{sha256_hash_prefix_2chars}/{full_sha256_hash}
Value: encrypted, compressed chunk binary dataKafka Topics
Topic: file-events (created, updated, deleted, shared) Topic: sync-events (per-user sync notifications)
| Concern | Solution |
|---|---|
| Chunk loss in S3 | S3 provides 11 nines durability. Cross-region replication for DR |
| Metadata DB loss | PostgreSQL with synchronous replication to standby. Daily backups to S3 |
| Partial upload failure | Resumable uploads: client retries only failed chunks |
| Sync conflict | Conflicted copy created; no data loss |
| Client crash mid-sync | On restart, client compares local state with server → resync delta |
| Dedup reference counting | Chunk's reference_count decremented on file delete. Chunk deleted only when count = 0 (background GC) |
Specific: Data Integrity
- End-to-end checksums: SHA-256 computed client-side before upload → verified server-side after upload → verified again on download
- Encryption: Client-side encryption (optional, user holds keys) or server-side AES-256 with per-user keys managed by KMS
- Immutable chunk storage: Chunks are write-once, never modified → eliminates corruption risk
Bandwidth Optimization
- Delta sync: Only upload changed chunks (CDC ensures minimal chunks change)
- Compression: LZ4 (fast) or zstd (better ratio) applied to each chunk
- Deduplication: Same file uploaded by 1000 users → stored once. Cross-user dedup: hash-based. Storage savings: 30-50%
Storage Tiering
- Hot storage (SSD-backed S3): Files accessed in last 30 days
- Warm storage (S3 Standard): Files accessed in last 90 days
- Cold storage (S3 Glacier): Files not accessed in 90+ days
- Automatic tiering based on last access timestamp
Quota Management
- Free tier: 15 GB; Paid tiers: 100 GB, 2 TB, unlimited
- Track usage: user_storage_used = sum(unique chunks owned by user)
- Shared files: storage counted against the file owner, not the viewer
Team/Enterprise Features
- Admin console, audit log, DLP (scan files for sensitive content)
- Legal hold (prevent deletion under legal investigation)
- Single Sign-On (SSO) with SAML/OIDC integration
Real-Time Collaborative Editing (Google Docs Extension)
- Beyond file-level sync: character-level real-time collaboration
- Operational Transform (OT): Central server transforms concurrent operations
- CRDT: Each character has unique ID, merge operations are commutative and idempotent: no central coordination needed
Interview Walkthrough
- Separate metadata (file tree, permissions) from blob storage (chunks in object store) — interviewers expect this split on day one.
- Explain content-defined chunking (CDC) over fixed-size blocks: a 1-byte edit should upload one chunk, not the entire file.
- Use content hashes for deduplication — identical chunks across users and versions are stored once, saving 30–50% storage.
- Apply Consistent Hashing to map chunk hashes to storage nodes for even distribution and minimal reshuffling on scale-out.
- Describe the sync protocol: client lists local hashes → server returns missing chunks → client uploads deltas only.
- Address conflict resolution for concurrent edits on the same file — version vectors or last-modified timestamps with merge policies.
- Quantify bandwidth savings with Back-of-the-Envelope Estimation: CDC reduces a 100 MB re-upload to ~4 MB for a small edit.
- Common pitfall: uploading the full file on every save instead of block-level delta sync with hash-based dedup.
Fixed-Size Chunking vs Content-Defined Chunking (CDC)
Fixed-size chunks: inserting 1 byte at position 0 shifts all chunk boundaries: ALL chunks appear changed. CDC with Rabin fingerprint: boundaries are content-defined and stable: only 1 chunk changes. Savings in practice: For a 100 MB file with a 1 KB edit, CDC uploads ~4 MB (one chunk) vs 100 MB with fixed-size. 25x less bandwidth.
Why PostgreSQL for Metadata (Not MongoDB/Cassandra)?
File metadata access patterns require relational queries: list files in a folder with B-tree index on parent_id + name, UNIQUE constraint preventing duplicate names, ACID transactions for atomic version creation, relational JOINs for permission checks, recursive CTEs for folder tree traversal. MongoDB has no real transactions. Cassandra has no JOINs, no transactions, no unique constraints: terrible fit.
Conflict Resolution: OT vs CRDT vs Last-Write-Wins
Dropbox uses Last-Write-Wins with conflicted copies: simple, no data loss, user resolves manually, best for file-level sync. Google Docs uses Operational Transform for real-time character-level collaboration but requires central coordination. Figma uses CRDT with no central server needed but higher memory overhead.
Deduplication: Within-User vs Cross-User
Within-User dedup is always safe (same file uploaded twice → stored once). Cross-User dedup saves 30-50% storage but has privacy concerns (can infer another user has the same file). Dropbox's actual approach: cross-user dedup on chunk hashes, accepting the minor privacy trade-off for massive storage savings.
Upload Strategy: Whole File vs Chunked vs Streaming
Whole file upload: if 1 GB fails at 999 MB → restart from scratch. Chunked upload (⭐): resumable (chunk 47 fails, restart only chunk 47), parallel upload, dedup, delta sync. For a 100 MB file with 1 KB change: uploads ~4 MB (1 chunk) instead of 100 MB.
Staff interviews expect you to articulate how the system evolves under real growth — not jump straight to the final architecture.
Phase 1: MVP (0 to 100K users)
Monolith or minimal services proving core dropbox google drive flows. Optimize for shipping speed and correctness over scale.
Key components: Single region · Primary DB + Redis cache · Synchronous core path · Basic monitoring
Move to next phase when: p99 latency exceeds SLO or DB CPU sustained above 70%
Phase 2: Growth (100K to 10M users)
Split read/write paths, introduce async processing for non-critical work, add caching layers and horizontal scaling.
Key components: Read replicas or CQRS · Message queue for async work · CDN / edge caching · Service-level SLOs
Move to next phase when: Hot keys, fan-out bottlenecks, or ops toil from manual scaling
Phase 3: Scale (10M+ users)
Shard data plane, multi-region active-active or active-passive, formal DR runbooks, cost optimization.
Key components: Database sharding / partitioning · Multi-region replication · Auto-scaling + chaos testing · Dedicated platform/SRE ownership
Move to next phase when: Regional failure domain risk, compliance data residency, or linear cost growth unsustainable
SLOs & Error Budgets
| Metric | Target | Rationale |
|---|---|---|
| Core user-facing availability | 99.95% | Budget for planned maintenance + unplanned failures without user-visible outage. |
| p99 latency (critical path) | Problem-specific — state target early and tie to capacity math | Interview credibility comes from connecting SLO to architecture choices. |
| Error rate (5xx) | < 0.1% | Distinguishes transient blips from systemic failure requiring rollback. |
| Data durability | 99.999999999% (11 nines) for committed writes | Define which operations require fsync/quorum vs async replication. |
Incident Scenarios (2am reality)
| Scenario | How you detect | Mitigation |
|---|---|---|
| Primary database unavailable | Health check failures, connection pool exhaustion alerts, elevated 5xx | Failover to replica / promote standby; enable read-only degraded mode if writes impossible; queue writes if async path exists |
| Traffic spike (10× normal) | RPS anomaly alert, autoscaling lag, latency SLO burn rate | Rate limit non-critical endpoints; scale read path horizontally; pre-warm caches; shed load on expensive operations |
| Bad deploy causing elevated errors | Canary metric regression, error budget burn, deployment correlation | Automated rollback within 5 minutes; feature flag kill switch; maintain N-1 compatibility |
Cost Drivers (Staff lens)
- Egress bandwidth and CDN (often dominates media/data-heavy systems)
- Database storage + IOPS at scale (plan compaction, TTL, tiering)
- Compute for async pipelines (right-size workers, spot instances for batch)
- Managed service premiums vs operational headcount trade-off
Multi-Region & DR
Start single-region with cross-AZ redundancy. Add read replicas in secondary region for DR. Move to active-active only when latency SLO or data residency requires it — accept conflict resolution complexity explicitly.