Kafka Architecture and Guarantees

Kafka is the answer when multiple consumers need to replay the same event stream — not when you need a simple task queue. We think in topics, partitions, offsets, and consumer groups.

Kafka is not a task queue — it is a replayable log. Think "who needs to read this stream, in what order, and for how long?" before drawing a broker box. Contrast with concept #17 Message Queues (consume-once tasks) and problem #32 Distributed Message Broker (full broker design interview).

Kafka fits when multiple consumers need the same durable, replayable stream — not for simple task queues. Three lenses:

Walk write and read paths as interview steps. Step 1 — Produce: producer batches records, picks partition via key hash. Step 2 — Replicate: leader appends to log; ISR followers sync before acks=all. Step 3 — Consume: consumer group polls batches, tracks offset per partition. Step 4 — Commit offset: after processing, commit — at-least-once by default. Step 5 — Scale: add partitions and consumers up to partition count; state ordering is per-partition only.

The Partition Model

Each topic consists of one or more partitions distributed across the cluster. Partitions allow Kafka to scale horizontally:

Partition keys, ISR, and pull-based consumption are the characteristics we cite on the whiteboard:

• Message record shape: Each record has a value (payload), optional key (partition routing), timestamp, and headers (metadata). Ordering is by offset within a partition, not by timestamp.
• Partition-Level Ordering: Messages are ordered strictly within a single partition, not globally across the topic.
• Partition key routing: Messages with the same key land on the same partition, preserving order per entity (e.g., per match, per user, per order).
• High Throughput via Sequential Writes: Kafka appends to partition logs sequentially, which maps well to disk and OS page cache.
• Zero-Copy Message Transfer: Data is copied directly from disk to network sockets without crossing into user-space memory.
• Offset-based progress: Consumers track their read position per partition and commit offsets back to Kafka. On restart they resume from the last commit (at-least-once by default).
• Decoupled Pull Protocol: Consumers request batches when they are ready, preventing memory exhaustion.

Throughput and replay cost operational complexity and eventual consumer lag — we compare:

Hot partitions, consumer lag, and rebalance pauses — we lead with mitigations before diagrams:

These HLD problems map to Kafka when replay, fan-out, or per-entity ordering matter:

Choose Kafka over SQS/Rabbit when replay, multiple independent consumers, or high-volume ingest dominate:

Kafka connects to event sourcing, stream processing, WAL durability, and message broker design problems:

• #17 Message Queues Fundamentals — task queues vs durable replayable logs; when RabbitMQ/SQS beats Kafka.
• #19 Event Sourcing & CQRS — append-only event store and projection pipelines.
• #20 Stream Processing Basics — Flink/Kafka Streams consumers on top of the log.
• #23 Write-Ahead Log (WAL) — same sequential-durability idea inside databases.
• #32 Distributed Message Broker (problem) — end-to-end broker HLD with partitions, ISR, and compaction.

ISR & Durability Mathematics

Kafka achieves partition reliability via replication. The leader replica handles producer writes and consumer reads (by default). Followers that are caught up with the leader form the ISR (In-Sync Replicas) set; lagging replicas drop out of acks=all quorum:

To balance write speed against durability risk, tune the producer's acks configuration:

• acks=0: Producer fire-and-forget. Zero durability guarantee.
• acks=1: Returns success the instant the leader broker writes it locally. Risk: leader dies before replication.
• acks=all: Leader writes locally and waits for all active ISR followers to sync. Full durability, higher write latency.

Log Compaction Internals

Normally, Kafka purges logs by age (TTL) or log size limit. But for key-value streams (like database records captured via CDC), we only care about the latest state of a key. Log Compaction periodically garbage-collects old keys, keeping only the final updated version per key:

Zero-Copy Reads & OS Page Cache

Traditional brokers read file contents into kernel buffer, copy them to user-space application memory, copy them back to kernel socket buffer, and then to network card. Kafka bypasses this entirely using the sendfile system call. The OS transfers data directly from the kernel page cache to the network NIC, eliminating CPU overhead and context switches.

ZooKeeper vs KRaft Consensus

Historically, Kafka relied on ZooKeeper to manage cluster states, partition assignments, and broker membership. Under KRaft (modern Kafka), consensus is integrated directly into Kafka brokers using a customized Raft-based metadata log. This eliminates the external coordination dependency, allows instant cluster boots, and supports millions of partitions.

Retention vs Compaction: Know Your Stream Type

Kafka offers two log lifecycle modes — choosing wrong breaks downstream consumers:

• Time/size retention (delete): Old segments drop after N days or N GB. Correct for clickstreams, metrics, and audit logs where history beyond the window is worthless.
• Log compaction: Keeps the latest record per key forever; tombstones with null values delete keys. Correct for CDC changelog topics where consumers rebuild current state from the log.

Event sourcing pitfall: Never enable compaction on an append-only event-sourcing topic where every state transition must be preserved — compaction garbage-collects intermediate events and destroys the audit trail. Pair event sourcing with infinite retention (delete policy disabled) or external snapshot + archive, not key compaction.