Design Instagram (Photo Sharing + Social Features)

Post Service

The Post Service manages photo and video intake, persists initial metadata, orchestrates downstream processing, and authorizes deletion workflows without proxying large media payloads through application servers.

• Handles photo and video uploads along with post creation and deletion lifecycle events.
• Upload Flow:
  1. The client requests a pre-signed S3 upload URL from Post Service.
  2. The client uploads media binaries directly to S3, preventing large payloads from congesting application servers.
  3. The client sends post metadata including captions, tags, and location to Post Service.
  4. Post Service persists metadata to Cassandra with status = processing.
  5. Post Service publishes a post-created event to Kafka for the Media Processing Pipeline.
• Post Deletion Flow:
  1. When a user requests post deletion, Post Service verifies account ownership and authorization.
  2. Post Service marks the post record as deleted or tombstones it in Cassandra to prevent read hydration.
  3. Post Service emits a post-deleted event to Kafka.
  4. Background workers consume post-deleted to asynchronously remove the post ID from follower Redis timelines and delete the post document from Elasticsearch.
  5. An edge invalidation worker triggers an asynchronous CDN cache purge for the post's media URL patterns. Because CDN edge purge is asynchronous and takes time to propagate globally, logical deletion correctness relies on read-time tombstone and authorization checks that immediately suppress the post and block media hydration during cache propagation.

Media Processing Pipeline

The Media Processing Pipeline executes asynchronous transcoding, variant generation, and safety filters before content is eligible for distribution.

• Triggered by: The post-created Kafka event emitted after initial metadata persistence.
• Processing steps:
  1. Resize: Generate 4 standardized versions (150x150 thumbnail, 320x320 small, 640x640 medium, and 1080x1080 large).
  2. Apply filters: If the user selected an image filter, apply the transformation server-side or validate client-side pre-processing.
  3. Generate blurhash: Create a compact string hash representation for progressive placeholder rendering.
  4. Extract EXIF: Read GPS coordinates and camera metadata, and strip sensitive privacy data.
  5. Content moderation: Run automated machine learning models for NSFW and violence detection.
  6. Video processing: Transcode videos into multiple adaptive bitrate streams (360p, 720p, 1080p) using HLS or DASH and extract video thumbnails.
  7. Publishing transition: When all variants and moderation checks pass, atomically update the post record to status = published in Cassandra, warm CDN edge caches, and emit a post-published event to Kafka via transactional outbox or CDC.
• Technology: AWS Lambda for parallel image transformations, FFmpeg for video encoding, and dedicated GPU clusters for computer vision moderation.
• Output: Multiple resized assets stored back in S3 with corresponding CDN URLs generated.

Fan-Out Service

The Fan-Out Service consumes published post events and distributes chronological timeline pointers into Redis caches according to creator audience size.

• Consumes from Kafka post_events topic specifically for post-published events (never post-created). Fan-out is triggered exclusively by this event after media processing is complete.
• Normal users (< 50K followers): Using an illustrative ~50K threshold (tuned dynamically in production based on worker capacity and queue lag), the worker fetches the author's follower list from the MySQL social graph and executes ZADD for post_id into each follower's feed:{user_id} Redis sorted set (scored by created_at timestamp), maintaining a rolling cap of 500 entries.
• Celebrities (≥ 50K followers): The worker skips push fan-out entirely. Posts remain in the author's user_posts Cassandra partition and are pulled dynamically at read time.
• Deletion handling: On receiving a post-deleted event, the worker executes ZREM to evict post_id from follower Redis timelines asynchronously.
• Consumer idempotency: Redis ZADD and ZREM operations are naturally idempotent under Kafka at-least-once delivery, ensuring duplicate event replays cause no timeline corruption.

Feed Service (Hybrid Fan-Out)

Feed Service coordinates bounded candidate retrieval, celebrity timeline pulls, read-time authorization filtering, and dynamic ML ranking before hydrating the final timeline response.

• Adopts the hybrid approach established in News Feed System and Twitter Timeline.
• Read-Time Assembly Pipeline:
  1. Bounded Candidate Retrieval: Fetches pre-computed candidate post IDs from the viewer's Redis sorted set feed:{user_id} (scored by creation timestamp). Candidate retrieval is bounded to at most 500 entries, so latency remains effectively constant with respect to total feed history.
  2. Celebrity Pull: Queries the user_posts table for the latest ~50 posts from each followed celebrity (accounts at or above the illustrative ~50K threshold).
  3. Visibility and Authorization Filtering: Evaluates candidate posts against active social graph permissions, dropping stale pointers from recent unfollows, account blocks, post deletions, and private accounts. Stale Redis pointers are harmless because they never bypass this authorization gate.
  4. ML Ranking at Read Time: Passes the combined candidate pool (~500 posts) to the ML Ranking Service, which scores and orders posts dynamically based on viewer affinity and engagement velocity.
  5. Hydration and Response: Hydrates the top 50 ranked post IDs with full post metadata, author profiles, and CDN media URLs, returning a cursor-paginated response to the client.
• Feed ranking signals: The machine learning model re-ranks candidate posts using multiple signals:
  • Relationship strength, measuring interaction frequency and bi-directional user-author interaction history between viewer and author.
  • Post recency, applying exponential decay to older content.
  • Engagement velocity, tracking like and comment acceleration within the initial 30-minute window.
  • Content format affinity, evaluating preferences across static photos, video clips, and carousels.
  • Historical viewer behavior and topical engagement clusters.

Story Service

Story Service manages ephemeral multimedia sharing with automatic 24-hour product expiration enforced through database TTLs, application-level timestamp validation, and asynchronous object lifecycle policies.

• Ephemeral content lifecycle: Stories maintain a strict 24-hour product visibility window. Cassandra enforces primary row expiry with default_time_to_live = 86400, while story records carry an explicit expires_at timestamp.
• Read-time validation: Story Service and feed requests evaluate expires_at on every read, immediately rejecting expired content regardless of storage or cache propagation states.
• Storage management: Media assets reside in dedicated S3 prefixes where S3 lifecycle policies make objects eligible for asynchronous deletion after 24 hours.
• CDN delivery and authorization: CDN media URLs require non-expired signed access, ensuring edge caching never serves logically expired content after the 24-hour window.
• Pre-fetching: Story thumbnails and initial frames for followed creators are pre-fetched by the client application on launch to eliminate playback buffering.
• Story tray cache: Redis sorted sets (story_tray:{user_id}) store pre-computed lists of followed creator IDs with active stories ordered by latest publication timestamp. The tray is asynchronously updated when stories are published or expire, and is periodically rebuilt as a recovery mechanism. Story Service acts as the authoritative enforcement point by validating expires_at on every read, while mobile clients may also defensively hide expired items.

Explore Service

Explore Service surfaces discovery content from accounts the user does not currently follow by combining graph clustering, topical embeddings, and engagement velocity.

• Discovery objective: Introduce relevant content outside the follower graph to drive discovery and creator growth.
• Candidate generation:
  1. Collaborative filtering identifies posts liked by users with similar historical consumption profiles.
  2. Content-based filtering pairs posts that share hashtags, topics, or location clusters with user affinity vectors.
  3. Engagement velocity surfaces rapidly trending posts exhibiting anomalous like and comment acceleration.
• Architecture implementation:
  • Offline pipelines run Apache Spark jobs to cluster user embeddings and pre-generate candidate pools per interest category.
  • Online ranking models score hundreds of candidate posts against real-time user context within a strict 100-millisecond latency budget.
  • Ranked Explore grids are cached in Redis with short expiration windows to balance personalization freshness with computation cost.

Social Service (Follow, Like, Comment)

Social Service handles core engagement interactions by coordinating atomic cache operations with asynchronous durability pipelines.

• Follow operations: Persists edge relationships to MySQL as the source of truth for the social graph, refreshes Redis follower sets, and publishes a follow-event to Kafka for downstream timeline invalidation.
• Like interactions: Executes an atomic check-and-set in Redis using SADD liked:{post_id} {user_id}, increments the counter with INCR like_count:{post_id}, and emits a Kafka event for asynchronous Cassandra persistence and notification triggers.
• Comments: Writes comment payloads directly to Cassandra using post_id as the partition key and comment_id as the clustering key for chronological ordering, while publishing an event to Kafka for the Notification Service.
• Deduplication: The Redis set for liked:{post_id} naturally prevents duplicate likes, ensuring that retried network requests remain idempotent.
• Anti-spam protection: Applies rate limiting capped at 20 comments per minute per user alongside machine learning spam classifiers that analyze comment text prior to ingestion.

Notification Service

Channel adapters isolate provider-specific retry logic and fan out push notifications across external mobile gateways and in-app activity feeds. For deep architectural mechanics, review the Notification System.

• Consumes notification events from Kafka topics including post-published, like-events, follow-events, comment-events, and mention-events.
• Delivery channels: Dispatches mobile push notifications through Apple Push Notification service (APNs) and Firebase Cloud Messaging (FCM), while updating an internal inbox for the in-app Activity tab.
• Consumer idempotency: Deduplicates push notifications using Redis SETNX on event_id or time-windowed post_id:user_id keys to prevent duplicate pushes under Kafka at-least-once delivery.
• Aggregation and batching: Collapses multiple engagement events on the same post over a short duration into a single summary notification, such as stating that multiple users liked your photo rather than dispatching dozens of individual pushes.
• Activity feed storage: Stores notification timelines in Cassandra partitioned by user_id and ordered by event timestamp to power the user activity view.
• Rate limiting: Enforces a ceiling of at most one push notification per post within a 5-minute window for identical event types to prevent notification fatigue.

Search Service (Elasticsearch)

Search Service powers typeahead queries and exploratory discovery across user accounts, hashtags, and geographic locations using distributed inverted indexes.

• Indexed entities: Indexes user profiles (username, display name, and bio), hashtags (post count and real-time velocity metrics), and geographic locations (place name and coordinates).
• Query features: Supports edge n-gram prefix autocomplete for usernames, fuzzy spell correction, and trending hashtag lookups.
• Consumer idempotency: Elasticsearch consumers ingest post-published events to index hashtags and locations using post_id as the document ID for idempotent upserts, and delete the corresponding document upon receiving post-deleted events.
• Ranking logic: User searches prioritize verified badges, follower volume, and mutual connections. Hashtag searches prioritize total post count combined with recent trending velocity.
• Data synchronization: Changes to profile metadata flow through Kafka change data capture (CDC) pipelines, allowing Elasticsearch consumer workers to update index documents asynchronously.
• Architectural constraint: Post captions are deliberately excluded from full-text search indexing because discovery on Instagram relies primarily on hashtags, Explore recommendations, and location tags rather than open text retrieval.

ML Ranking Service

The ML Ranking Service runs on the read serving path to score merged candidate posts retrieved from Redis timeline caches and celebrity pulls within low latency bounds.

• Scores approximately 500 candidate post IDs sourced from the user's Redis feed cache (scored by creation timestamp) combined with real-time celebrity pulls.
• Read-time scoring: Batch-scores candidate posts with ONNX runtime on CPU within a strict 100-millisecond latency budget before passing the top 50 to hydration.
• Model features: Evaluates relationship strength, post recency, engagement velocity, content format (photo, video, or carousel), and historical topic preferences.
• Service consumers: Serves both Feed Service for personalized home feeds and Explore Service for discovery candidate scoring.
• Fallback resilience: Automatically falls back to a simple reverse-chronological sort if the ranking cluster experiences downtime or latency spikes.

Analytics Pipeline

Analytics runs on a separate service level objective from the hot serving path, aggregating telemetry and engagement events without placing load on transactional databases.

• Consumes all Kafka event streams into Apache Flink for real-time windowed aggregations before sinking data into ClickHouse.
• Idempotency: Flink windowing deduplicates incoming events by event_id before database insertion.
• Computed metrics: Tracks post engagement rates, story completion percentages, follower growth curves, and creator analytics dashboards.
• Downstream consumers: Supplies feature pipelines for Explore ranking models, enriches content moderation heuristics, and powers advertising attribution.

Event Bus Design (Kafka)

The event bus decouples producers from consumers, buffers traffic spikes, and coordinates independent parallel consumer groups across asynchronous processing stages.

topics:
  post_events: "post-created (triggers media pipeline), post-published (triggers fan-out, search indexing, and notifications), or post-deleted (triggers cache and index eviction)"
  story_events: "story-created or story-expired"
  like_events: "like or unlike interactions on posts"
  follow_events: "follow, unfollow, or block events for relationship graph updates"
  comment_events: "new comment or comment deletion on posts"
  notification_events: "batched mobile push alerts and in-app activity triggers"

post_events_configuration:
  partitions: 128
  partition_key: "author_id (guarantees strict per-creator ordering across Kafka partitions)"
  retention_period: "7 days"
  replication_factor: 3
  min_insync_replicas: 2

producer_settings:
  idempotence: true # enable.idempotence=true — prevents duplicate Kafka records caused by producer retries
  payload_schema:
    event_id: "UUID"
    event_type: "post-created | post-published | post-deleted"
    post_id: "int64"
    author_id: "UUID"
    status: "processing | published | deleted"
    media_urls: "string[]"
    created_at: "timestamp"

consumer_groups:
  media_pipeline:
    consumes: "post-created"
    action: "Resize variants, generate blurhash, and run moderation, then on success, atomically mark published in DB and emit post-published"
  fan_out_worker:
    consumes: "post-published"
    action: "ZADD post_id to follower Redis feeds for accounts below the illustrative ~50K celebrity threshold"
  search_indexer:
    consumes: "post-published and post-deleted"
    action: "Upsert or delete hashtags and locations in Elasticsearch inverted indexes"
  notification_worker:
    consumes: "post-published, like_events, and comment_events"
    action: "Dispatches batched mobile push notifications and activity feed updates"
  analytics_pipeline:
    consumes: "All post_events, like_events, and story_events"
    action: "Streams events to Apache Flink for windowed aggregations and sinks to ClickHouse"

consumer_idempotency:
  delivery_guarantee: "At-least-once delivery, requiring all consumers to implement idempotent execution"
  fan_out_workers: "Redis ZADD is inherently idempotent on duplicate event replay"
  search_indexer: "Elasticsearch writes use post_id as document ID for idempotent upserts"
  notification_worker: "Deduplicates alerts using Redis SETNX on event_id or post_id:user_id time windows"
  analytics_pipeline: "Flink windowing deduplicates incoming events by event_id before database insertion"

pipeline_paths:
  sync_upload_path: "Client uploads to S3 -> Post Service persists post(status=processing) -> emit post-created -> return 201 Created"
  async_publishing_path: "Media pipeline finishes -> atomically update status=published in DB -> emit post-published event via transactional outbox/CDC -> fan-out, search, and notification consumers process in parallel"
  deletion_path: "Authorize delete -> canonical tombstone/delete in DB -> emit post-deleted event -> async workers remove feed pointers, delete search index docs, and trigger async CDN purge while read-time checks suppress content during propagation"
  dead_letter_queue: "Route to post-events-dlq after 3 retries, and alert when consumer lag exceeds 60s"