Design an API Gateway (Kong / Envoy)

1. The Data Plane (Asynchronous I/O)

Request flow crosses four concerns in sequence: terminating TLS, authenticating, rate limiting, and routing to a healthy upstream, with each step implemented as an inline plugin on the event loop proxy core. The sections below connect asynchronous I/O architecture, zero downtime configuration propagation, distributed rate limiting, stateless JWT authentication, circuit breaking, and the edge security layers that sit in front of the gateway.

An event loop architecture with epoll or kqueue handles thousands of concurrent connections per worker thread rather than a thread per connection model, which quickly exhausts CPU through context switching.

To achieve high throughput (10,000+ requests/sec per node), gateways like Nginx (which Kong is based on) or Envoy do not use a thread per connection model (like older Apache or Tomcat servers), which would exhaust CPU and RAM via context switching overhead. Instead, they use an Event Loop architecture (epoll/kqueue) with non-blocking I/O. A single worker thread handles thousands of concurrent connections efficiently, dramatically reducing memory overhead.

2. Dynamic Configuration (Control Plane) ⭐

Dynamic configuration via xDS streams or etcd watches lets gateway nodes receive validated routing snapshots without restarting. The proxy atomically swaps the active configuration in memory, while worker generations can drain existing connections before retiring old state.

If an engineering team deploys a new microservice, the gateway needs to know the new route. If we have 50 gateway nodes, restarting them to update a YAML config file drops active connections and causes an outage. Modern gateways solve this by separating the control plane from the data plane. The control plane stores durable configuration, validates it, and publishes versioned snapshots through xDS or an equivalent distribution channel. Data plane nodes consume those snapshots locally instead of querying PostgreSQL on the request path.

# Versioned configuration snapshot published by the Control Plane
version: 42
route:
  name: payment-service-route
  path: /api/v1/payments/*
  methods:
    - GET
    - POST
  backend:
    service: payment-service
    port: 8443
  plugins:
    - rate-limiting
    - jwt

# The Control Plane distributes this validated snapshot through xDS.
# Data Plane nodes atomically swap the active snapshot in memory.
# Existing connections continue draining on the previous snapshot.

3. Distributed Rate Limiting

Global rate limiting requires shared state across gateway nodes when the quota must be exact. Redis with Lua provides atomic Token Bucket decisions for that strict mode. Optional local token buckets can reduce latency when the policy accepts bounded global overshoot and asynchronous reconciliation.

Rate limiting mitigates application abuse and request floods. Volumetric DDoS is primarily handled at the CDN or anycast WAF edge. For globally enforced limits, the gateway shares Redis state across nodes so an attacker cannot bypass a limit simply by reaching another node.

Optimization (Local Caching vs Accuracy): A synchronous Redis round trip for every request can add 1 to 2 ms of latency. A local token bucket with a preallocated per node budget can reduce that cost, with asynchronous reconciliation to Redis. This trades exact global enforcement for bounded overshoot, so it is appropriate only for policies that explicitly accept approximate global enforcement.

4. Authentication (Stateless JWT)

Stateless JWT validation at the gateway avoids a database round trip on every request by verifying signatures offline using cached JWKS keys and validating the issuer, audience, expiry, not before, and required scopes or roles. Refresh JWKS on an unknown key identifier and allow key overlap during key rotation.

Making a database call to the User Service to validate a session token for every API request creates unnecessary dependency and latency. Instead, the gateway uses stateless JSON Web Tokens (JWT).

Because the JWT is cryptographically signed with asymmetric keys such as RS256, the gateway can verify the token's authenticity offline using cached JWKS keys. It trusts only validated claims such as user_id and roles. The gateway first strips any client supplied identity headers, then injects its own authenticated identity headers over the mTLS protected upstream connection. Backend services still enforce authorization based on the trusted identity and resource policy.

5. Circuit Breaker Pattern

Circuit breakers prevent a failing upstream from exhausting the gateway connection pool and causing cascading failure across the entire API platform.

If the "Order Service" goes down, the gateway might queue up thousands of requests waiting for it to respond, eventually exhausting the gateway's own connection pool and causing a catastrophic cascading failure across the entire API platform.

• The Circuit Breaker tracks failure rates, such as 50% failures occurring in 10 seconds.
• If the threshold is crossed, the circuit Opens. The gateway instantly rejects new requests to the Order Service with 503 Service Unavailable without even trying the network call, giving the backend time to recover.
• After a timeout, it shifts to Half-Open, allowing a small percentage of test requests through. If they succeed, it Closes the circuit. Failed test requests reopen it.

6. mTLS Upstream & WAF at the Edge

Defense in depth spans three layers. The edge WAF blocks OWASP attacks, bot floods, and volumetric traffic. Gateway plugins enforce authentication, request limits, and circuit breaking. mTLS and workload identity protect the gateway to service path. Rate limiting can fail open where endpoint policy permits it, while authentication failures fail closed.

mTLS (service to service):
  - The edge security layer filters public traffic before the gateway.
  - The gateway routes to upstream microservices over mTLS using SPIFFE/SPIRE or an equivalent workload PKI.
  - The gateway validates the workload identity in the certificate against the expected service identity.
  - Backend services accept gateway traffic only over the authenticated channel.
  - This limits lateral movement if a pod is compromised.

WAF / edge protection:
  Layer 1: Anycast CDN/WAF
    OWASP Top 10 rules, SQLi/XSS blocking, geo-blocking, bot scoring, and volumetric DDoS absorption.
    If the WAF terminates TLS, it re-encrypts traffic to the gateway. Otherwise the gateway terminates client TLS.
  Layer 2: Gateway plugins
    IP rate limiting, JWT validation, request size caps, and circuit breaking.
  Layer 3: Backend
    Resource authorization remains in the backend service. The gateway authenticates the caller but does not replace resource authorization.

Rate limiting policy:
  - Strict global limits use an atomic Redis decision.
  - Latency sensitive endpoints may use bounded local token buckets with asynchronous reconciliation when the policy permits controlled overshoot.
  - Availability oriented endpoints may fail open on Redis outage. Security sensitive endpoints may fail closed.