Design a Notification System (Push, Email, SMS)

Notification Service (Ingress API Gateway)

The API layer serves as the single ingress point for all internal microservices. It validates request payloads, verifies recipient preferences and quiet hours, renders localized templates, and fans out messages to channel-specific Kafka topics before returning an immediate 202 Accepted response.

• Validation: Enforces schema correctness, verifies user existence, and confirms template availability.
• Preference Filtering: Queries cached user preferences to suppress channels the user has disabled for that notification category.
• Rate Limiting: Executes token-bucket checks in Redis to enforce per-user channel thresholds (such as a maximum of 10 push notifications per hour).
• Quiet Hours Evaluation: Converts the current timestamp to the user's local timezone and defers non-critical notifications until the quiet window concludes.
• Idempotency Enforcement: Checks the request_id against Redis using a SET NX command with a 24-hour expiration to prevent duplicate enqueueing.

Template Rendering Pipeline

After preference and quiet-hours checks pass, the API layer resolves the corresponding template_id and user locale from PostgreSQL. It substitutes dynamic parameters (such as customer name and tracking links) to produce channel-tailored payloads: push headers under 256 characters, full responsive HTML email MIME bodies, and segment-optimized SMS text.

Pre-rendering templates at the API layer allows downstream channel workers to operate as lightweight delivery agents that never execute secondary database lookups.

User Preferences and Suppression Service

User preferences and global suppression lists reside in PostgreSQL to guarantee strong consistency, ensuring marketing opt-outs take effect immediately without violating regulatory compliance standards.

• Granular Settings: Configurable per notification category (marketing, transactional, social) and per channel (push, email, SMS).
• Suppression Registry: Maintains bounced email addresses, unsubscribed phone numbers, and invalidated push tokens to prevent repetitive failed deliveries.

Kafka Message Bus and Topic Topology

Kafka isolates high-volume marketing traffic from critical alerts and buffers transient downstream provider slowdowns. Topics are partitioned by user_id to preserve strict chronological ordering for each individual recipient.

• Dedicated Channel Topics: push_notifications, email_notifications, sms_notifications, and notification_status.
• Durable Asynchronous Decoupling: Configured with a replication factor of 3 and min.insync.replicas=2, supporting a 7-day retention period. Kafka guarantees at-least-once message delivery to worker pools, while final handset delivery depends on provider responses and retries.
• Blast Isolation: If an external email provider experiences latency, only the email topic accumulates lag while push notifications and SMS alerts proceed uninterrupted.

Push Notification Worker Pool

Push workers manage persistent HTTP/2 connections to Apple Push Notification service (APNs) and Firebase Cloud Messaging (FCM), minimizing handshake latency across high-throughput bursts.

• Connection Pooling: Reuses warm HTTP/2 multiplexed connections to avoid per-message TLS handshake overhead.
• Token Invalidation Handling: When APNs returns an HTTP 410 Gone or FCM returns a NotRegistered error, workers flag the device token as invalid in PostgreSQL to suppress future attempts.

Email Worker Pool

Email workers ingest pre-compiled MIME payloads from Kafka and dispatch them through primary and secondary delivery providers with automated circuit breakers.

• Multi-Provider Routing: Routes bulk traffic through AWS SES and transactional alerts through SendGrid, failing over automatically if error rates rise.
• Deliverability Guardrails: Enforces SPF, DKIM, and DMARC record signing while managing dedicated IP warmup schedules to safeguard domain reputation.

SMS Worker Pool

Because SMS carries significant per-segment carrier costs, SMS workers apply strict rate caps and dynamically route messages to the most cost-effective provider by destination country code.

• Regional Routing: Directs domestic North American traffic through Twilio and international deliveries through regional aggregators like Sinch or MessageBird.
• Segment Optimization: Validates character encodings (GSM-7 vs UCS-2) to avoid accidental multi-segment billing overhead.

Delivery Tracker and Telemetry Service

Delivery Tracker processes worker dispatch receipts and asynchronous provider webhooks to record lifecycle milestones in Cassandra and ClickHouse.

• State Transitions: Tracks progression across QUEUED, SENT, DELIVERED, READ, FAILED, and BOUNCED, distinguishing between successful provider acceptance and confirmed handset delivery based on provider capabilities.
• Storage Split: Writes notification records to Cassandra for fast user-facing inbox lookups and streams analytics events to ClickHouse for delivery funnel dashboards.

Event Bus Configuration Reference

The Kafka event bus schema and consumer group topology structure message flow across partitions and dead-letter queues:

Topic: push_notifications
  Partitions: 128
  Partition key: user_id (preserves per-user delivery order)
  Retention: 7 days
  Replication factor: 3, min.insync.replicas: 2

Topic: email_notifications
  Partitions: 64
  Partition key: user_id
  Retention: 7 days

Topic: sms_notifications
  Partitions: 32
  Partition key: user_id
  Retention: 7 days

Topic: notification_status
  Partitions: 32
  Partition key: notification_id
  Retention: 30 days (delivery webhook audit trail)

Producer: Notification Service (executes validation, deduplication, and template rendering)
  Event schema: { notification_id, request_id, user_id, channel, template_id, priority, payload, created_at }

Consumer groups (per channel pipeline):
  1. push-workers: Persistent HTTP/2 connection pooling to APNs and FCM
  2. email-workers: SendGrid and AWS SES with automated circuit-breaker failover
  3. sms-workers: Twilio and Nexmo with regional country routing
  4. status-tracker: Streams delivery receipts into Cassandra and ClickHouse
  Dead Letter Queue: {channel}_notifications-dlq after 3 retries, with lag alerts above 60s

Synchronous Hot Path: Validate request -> Redis deduplication on request_id -> Enqueue to channel topics -> Return 202 Accepted
Asynchronous Delivery Path: Channel worker pools deliver to providers -> Delivery Tracker processes provider webhooks