Design an LLM Chat Application (ChatGPT)

1. Streaming via Server-Sent Events (SSE)

Never wait for a full completion before responding to the user. Server-Sent Events stream tokens incrementally as the GPU generates them, reducing perceived latency even when full generation takes 10 seconds or longer. For transport details, see Network Protocols (HTTP, gRPC, WebSocket).

Waiting for a 500-word response to fully generate on the GPU before sending it to the client takes 10+ seconds, creating an unacceptable user experience. We use Server-Sent Events (SSE) (or WebSockets) to stream chunks to the client as soon as the GPU produces them. SSE fits this server to client stream because it uses ordinary HTTP and does not require a bidirectional protocol. The client should detect disconnects, and the server can support resumable streams with event IDs plus a short replay buffer.

HTTP/1.1 200 OK
Content-Type: text/event-stream
Cache-Control: no-cache
Connection: keep-alive
X-Accel-Buffering: no

id: chunk_1
data: {"id": "msg_1", "object": "chat.completion.chunk", "choices": [{"delta": {"content": "The"}}]}

id: chunk_2
data: {"id": "msg_1", "object": "chat.completion.chunk", "choices": [{"delta": {"content": " capital"}}]}

id: chunk_3
data: {"id": "msg_1", "object": "chat.completion.chunk", "choices": [{"delta": {"content": " of"}}]}

id: chunk_4
data: [DONE]

2. Context Manager & Prompt Builder

The inference model is stateless between requests, so the Context Manager reconstructs the prompt from persisted conversation state on each turn and enforces the model context window budget. It should fetch the required recent messages plus any compact summary or retrieved older context rather than blindly replaying the entire history.

The model computes each request from the tokens supplied in that request and does not persist conversation memory by itself. To provide conversational memory, the server fetches the recent message slice, system instructions, and any retained summary or retrieved context, then assembles that state into the prompt before sending it to the model.

[
  {"role": "system", "content": "You are a helpful AI assistant. Limit answers to 1 sentence."},
  {"role": "user", "content": "Hi"},
  {"role": "assistant", "content": "Hello! How can I help?"},
  {"role": "user", "content": "What's the capital of France?"}
]

If the history exceeds the model context window (such as 8K or 128K tokens), the Context Manager must intervene. It can apply truncation with a sliding window, summarize older blocks into a dense memory block, or use retrieval augmented generation to fetch only semantically relevant past messages using techniques covered in Vector Embeddings and Approximate Nearest Neighbor Search. A strong production design can combine these techniques by retaining a compact summary while retrieving specific older turns only when needed. Summarization can run asynchronously, with a synchronous fallback to recent history when a fresh summary is unavailable. The invariant is prompt_tokens + max_output_tokens ≤ model_context_limit, with explicit headroom for system instructions, tool metadata, and other request overhead.

3. The Inference Engine & GPU Optimization ⭐

Production inference requires specialized runtimes such as vLLM or TensorRT-LLM rather than raw PyTorch scripts. Continuous batching and paged KV cache management improve GPU utilization and reduce memory fragmentation.

Continuous Batching (Iteration-Level Scheduling)

In traditional request level batching, the GPU waits for the longest request in the batch to finish before accepting new requests.Continuous Batching ejects finished requests at the exact token iteration they finish and inserts new requests instantly. This generally improves GPU utilization and throughput by keeping the scheduler supplied with runnable work.

KV Cache & PagedAttention

The KV Cache stores the internal self attention state tensors, Keys and Values, for previously generated tokens so they do not have to be recomputed for every new token. However, KV cache memory is highly dynamic and can fragment physical VRAM.PagedAttention, as implemented by vLLM, manages KV cache blocks in pages so logical blocks need not occupy one contiguous physical VRAM region. This reduces fragmentation and can increase effective batching capacity, with the actual gain depending on model architecture, sequence lengths, memory budget, and workload.

4. Model Router & Prefix Caching

Prefix aware routing directs follow up messages to a GPU worker that still holds the reusable prompt prefix in VRAM, reducing prefill work by up to 60% in favorable multi turn cases. Similar local context caching patterns power advanced code agents such as our AI Coding Assistant (Cursor).

A specialized L7 router sends requests to inference workers. Advanced routers implement prefix aware routing: when a large reusable prefix, such as a 10,000 word document, is known to be cached, the router prefers the worker holding that prefix KV state. The unchanged prefix can avoid repeated prefill work, but newly appended user turns still require their own prefill and decode steps. Load aware fallback prevents a hot prefix from concentrating traffic on one worker.