Design a Log Aggregation and Search System (like Splunk / ELK)

PII Redaction: Preventing Sensitive Data in Logs

Developers frequently log sensitive customer details by accident, including email addresses, authentication secrets, Social Security numbers, and credit card credentials. Storing sensitive data in plaintext can violate security and privacy requirements such as GDPR and CCPA. Retrospective cleanup across petabytes of indexed data can be expensive and slow, so prevention at the source is mandatory.

Defense in Depth Solution:

• Layer 1 (Code Linting): Reduce accidental PII logging through CI rules that flag typical sensitive parameter names and known logging anti-patterns.
• Layer 2 (Agent Redaction): Local agent executes high speed regex sanitization matches:
  • Email regex: [a-zA-Z0-9._%+-]+@[a-zA-Z0-9.-]+\.[a-zA-Z]{2,}
  • SSN regex: \d{3}-\d{2}-\d{4}
  • Credit Card regex: \d{4}[- ]?\d{4}[- ]?\d{4}[- ]?\d{4}
• Layer 3 (Tokenization Vaults): Replace identified PII parameters with reversible transaction tokens (such as tok_39d8c4). The tokenization service controls key access through managed key storage and role based authorization, so individual operators do not hold the underlying cryptographic keys.

Access Control and Tenant Isolation

Logs often contain sensitive production data, so search and live tail access must be authenticated and authorized independently of the ingestion path. The service injects tenant, environment, and service scope from the authenticated identity rather than trusting equivalent values supplied only in the query, and it audits sensitive log access. Encryption in transit and at rest is mandatory, while agent side redaction remains the first line of defense for secrets and regulated fields.

Structured vs Unstructured Logging: Why JSON Wins

Unstructured logs require complex and fragile Grok or regular expression processing pipelines that break easily when log formats evolve:

Unstructured raw log line:
2026-03-14 10:00:00 ERROR [payment-service] User 12345 payment failed: timeout after 5000ms

Structured JSON log line:
{"timestamp":"2026-03-14T10:00:00Z","level":"ERROR","service":"payment-service","user_id":"12345","message":"payment failed","event_id":"evt_abc123","error":"timeout","duration_ms":5000,"trace_id":"abc123"}

Why JSON wins:

1. No application specific regex parsing: Structured fields avoid runtime Grok or regex extraction in the indexer for fields that are already typed, while JSON parsing itself remains necessary.
2. Type preservation: Numeric values (such as duration_ms) remain queryable numbers rather than strings.
3. Compression friendly: JSON structures repeating identical field keys are highly deduplicated by block compression algorithms like LZ4 and ZSTD.

Log, Trace, and Metric Correlation: The Observability Triangle

Tracing a user request failure across multiple microservices is challenging without system-wide correlation.

Unified Troubleshooting Workflow:

1. The application SDK injects trace_id and span_id from OpenTelemetry tracing contexts into every structured log event payload.
2. When an error log alert triggers, engineers click the attached trace_id to open a distributed waterfall trace view.
3. After pinpointing the specific database call causing latency, responders navigate directly to the database metrics dashboard for root cause analysis.

{
  "level": "ERROR",
  "message": "payment timeout",
  "trace_id": "abc123",
  "span_id": "def456"
}

Cost Optimization: Calculations & Levers

At 4.3 TB/day compressed logical volume, keeping one year of data on hot SSD at the stated $0.10/GB-month price costs approximately $1.88M / year before index and replica overhead. Applying an aggressive tiering lifecycle architecture reduces the storage-only estimate to approximately $451K / year under the stated illustrative prices:

Annual Storage Cost Model:
Assumptions:
  - 4.3 TB/day compressed logical log volume
  - Storage prices are stated per GB-month
  - Excludes Elasticsearch index overhead, replicas, compute, requests, and egress

  - Hot tier (2 days):       4.3 x 2 x 1000 x $0.10 x 12 = $10,320/year
  - Warm tier (28 days):     4.3 x 28 x 1000 x $0.03 x 12 = $43,344/year
  - Cold tier (335 days):    4.3 x 335 x 1000 x $0.023 x 12 = $397,578/year
  -------------------------------------------------------------------------
  Tiered total: ~$451,242/year

Pure hot storage:
  4.3 x 365 x 1000 x $0.10 x 12 = ~$1,883,400/year

Estimated reduction: ~76% through architectural tiering.

Additional Cost Levers

• Intelligent Sampling: Retain 100% of ERROR and WARN logs, but sample only 10% of high volume INFO and DEBUG lines. This cuts total volume by approximately 70% only when INFO and DEBUG together account for about 77.8% of baseline traffic, so the actual reduction must be measured from production mix.
• Heartbeat Filtering: Filter out continuous, redundant health check probe logs at the host agent level before they traverse the network.

OpenTelemetry Logs & Trace Correlation

Shipping logs with trace_id and span_id fields via the OpenTelemetry log bridge allows engineers to jump seamlessly from an individual log event to the full distributed transaction trace explored in Distributed Tracing. Sampling verbose DEBUG lines aggressively while retaining all ERROR events with complete context ensures that correlating logs to metric exemplars closes outage investigations significantly faster than search queries alone.

Related Problems and Core Concepts

Log aggregation and real time observability intersect directly with several platform design problems:

For deep-dive query execution, shard allocation, and cluster operations in search engines, explore Elasticsearch Search Cluster. Distributed trace propagation and context bridge mechanisms are analyzed in Distributed Tracing, while automated incident routing is detailed in On Call Escalation System. For time series telemetry and high throughput streaming pipelines, see Distributed Metrics Aggregation and User Analytics Pipeline.

To master the architectural concepts underpinning this design, review Kafka Architecture and Guarantees, Indexing and Query Optimization, Observability and Distributed Tracing, Stream Processing Basics, and System Design Interview Patterns.

Interview Walkthrough

• 25-minute cut

  Skip arch50 and arch75 depth unless interviewing for a staff-level role.

  • Diagram the pipeline from host agent to Kafka buffer to indexer workers and Elasticsearch, explaining each component's backpressure role (5 min)
  • Quantify ingestion scale at 1,000,000 lines/sec and 4.3 TB/day compressed, justifying the Kafka buffer and ILM storage tiers (6 min)
  • Mandate structured JSON logging with low cardinality index fields configured at the edge (5 min)
  • Detail hot, warm, and cold ILM: active logs on SSD, aged indices on HDD, and long-term retention on object storage (5 min)
  • Staff level: multiline stack trace assembly and edge PII redaction in the collector pipeline (4 min)
• Trace the ingestion path: application runtime to local host agent with buffering and redaction, through Kafka and indexer workers, down to hot, warm, and cold storage tiers.
• Mandate structured JSON logging across services because unstructured Grok parsing breaks on format drift and introduces parsing latency.
• Redact PII at the host agent layer using regex detection and replacement before logs leave the node, because retrospective cleanup across petabyte archives can be expensive and slow.
• Implement progressive storage tiering: 2 days on hot SSD Elasticsearch, 28 days on warm HDD, and 335 days on cold object storage for an estimated 76% reduction in storage cost under the stated pricing assumptions.
• Inject trace_id from OpenTelemetry contexts into every log record to enable one click navigation from error logs to distributed trace waterfalls and metrics.
• Apply intelligent sampling by retaining 10% of INFO and DEBUG logs while preserving 100% of ERROR and WARN events, reducing ingest volume by approximately 70% when INFO and DEBUG comprise about 77.8% of baseline traffic.
• Quantify scale: 500,000 to 1,000,000 logs/sec produces 500 MB/s to 1 GB/s ingestion bandwidth, requiring local agents to batch and compress before network transfer.
• Address the critical pitfall of retaining all logs in hot Elasticsearch indefinitely, which costs approximately $1.88M annually under the stated compressed volume and $0.10/GB-month storage assumption, before index and replica overhead.