Design a Cryptocurrency Exchange

Order Lifecycle

The Order Service validates and reserves funds in Redis before the matching engine ever sees the order, meaning a failed reserve returns immediately without polluting the book. Settlement consumers apply PostgreSQL ledger updates idempotently from trade events.

1. User submits order: POST /api/orders {pair:BTC_USDT, side:buy, price:50000, qty:0.5}
2. Order Service:
   a. Validate: user authenticated, pair exists, qty > min
   b. Reserve funds: Lua script in Redis → USDT_available -= 25000
   c. If Redis reserve fails → reject immediately
   d. Persist order (status=open) to PostgreSQL
   e. Route to matching engine for this pair

3. Matching Engine (single-threaded, in-memory per pair):
   a. Insert order into order book (TreeMap of price levels)
   b. Match against opposite side: buy $50K vs best ask $49,950 → MATCH at $49,950
   c. Generate trade event. Write to WAL before returning.
   d. Publish trade event to Kafka

4. Post-Trade Settlement (async from Kafka):
   a. Update balances, deduct fees (maker 0.1%, taker 0.15%)
   b. All inside PostgreSQL transaction

5. WebSocket broadcast: order book update, trade feed, user events

Matching Engine Deep Dive

Price-time priority means the best bid meets the best ask at the maker's price, and FIFO queueing within a price level prevents queue-jumping. Single-threaded execution per pair eliminates lock contention and makes cancel-during-match semantics deterministic.

Data structure: Two TreeMaps (Red-Black Trees) per pair
  Bids: sorted by price DESC (highest first), then time ASC
  Asks: sorted by price ASC (lowest first), then time ASC

Price-Time Priority: same price → earlier order fills first (FIFO)

Single-threaded: ONE thread per trading pair
  No locks needed → predictable µs latency
  100 pairs yields 100 independent threads

WAL: Every match written to append-only file BEFORE publish
  On crash → replay WAL → rebuild order book state

Performance: 100K+ matches/sec per pair (in-memory, single-threaded)

Deposit & Withdrawal Flow

Deposits wait for chain confirmations before crediting, so a reorg deeper than the threshold reverses pending credits. Withdrawals drain the hot wallet first, while large amounts queue for cold-wallet multi-sig with human review on anomaly signals.

DEPOSIT:
  1. Generate HD wallet address (unique per user per chain)
  2. Blockchain Watcher detects incoming transaction
  3. Wait for N confirmations (BTC:3, ETH:12, SOL:32)
  4. Credit user: UPDATE balances SET available = available + amount
  5. Edge case: chain reorg → reverse credit if tx disappears

WITHDRAWAL:
  1. User submits withdrawal (2FA required)
  2. Deduct from available balance (PG transaction)
  3. Hot wallet service signs transaction (HSM multi-sig 2-of-3)
  4. Broadcast to blockchain
  5. Monitor for confirmation
  6. If hot wallet balance < withdrawal amount: queue and trigger cold-to-hot replenishment

Race Conditions

Double-spend, cancel-during-match, and withdraw-while-trading are the three classic failure modes. Redis Lua and PostgreSQL SELECT FOR UPDATE split available vs reserved balances so concurrent orders cannot oversell the same coins.

1. Double-Spend: User has 1 BTC, submits two sell orders simultaneously
   Solution: Redis Lua atomic check-and-decrement (single-threaded, no race)
   PostgreSQL: authoritative backup with SELECT FOR UPDATE

2. Order Cancel During Matching:
   Solution: Single-threaded engine → cancel queued behind match
   If order partially filled before cancel: fill matched, cancel remainder

3. Withdrawal After Balance Used in Trade:
   Solution: Withdrawal deducts from available, trade from reserved (different balance pool)
   PostgreSQL transaction ensures atomicity

Blockchain Reorg Handling

The system waits for N confirmations before crediting deposits (such as 3 confirmations or approximately 30 minutes on Bitcoin, and 12 confirmations or approximately 3 minutes on Ethereum). The Blockchain Watcher compares block hashes continuously. If a reorg is detected deeper than N blocks, the system reverses pending credits. The exchange never permits users to trade against unconfirmed deposits.

Event Bus Design (Kafka)

Trade events partition by trading pair to preserve per-book ordering, allowing settlement, WebSocket broadcast, and audit consumers to replay from the same durable log. WAL fsync precedes Kafka publish so a crash never loses a matched trade that was ACKed to the client.

Topic: trade-events
  Partitions: 128 (partition by trading_pair to preserve per-pair ordering)
  Retention: compacted by trade_id (compliance; replay rebuilds order book state)
  Producers: Matching Engine immediately after WAL append (before ACK to client)
  Payload: {trade_id, pair, price, qty, buyer_id, seller_id, maker_order_id, taker_order_id}
  Consumers: Settlement Service, WebSocket broadcaster, audit log, market data feed

Topic: order-events
  Partitions: 64 (partition by order_id)
  Events: placed, partially_filled, cancelled, expired
  Consumers: Order Service state sync, user notification WebSocket

Topic: wallet-events
  Events: deposit_confirmed, withdrawal_submitted, withdrawal_broadcast, balance_adjusted
  Consumers: Blockchain Watcher, AML monitoring, Proof-of-Reserves aggregator

Match path: in-memory match → WAL fsync → publish trade-events → return fill to client
  Settlement (balance update, fees) consumes trade-events asynchronously