Design a Distributed Lock Manager

Approach 1: Single Redis Node (Fastest but Weakest)

ACQUIRE:
  SET lock:{name} {owner_id} NX EX {ttl_seconds}
  NX = only set if Not eXists (atomic), EX = expire after ttl_seconds
  Returns "OK" if lock acquired, or nil if lock is already held by someone else

RELEASE: (must be executed atomically using Lua script)
  if redis.call("GET", KEYS[1]) == ARGV[1] then
      return redis.call("DEL", KEYS[1])
  else
      return 0  -- not the lock holder, so abort deletion
  end

Approach 2: Redlock Algorithm (Distributed Redis Quorum)

Redlock Setup across 5 Independent Redis Nodes:
1. Record initial start timestamp
2. Attempt SET key uuid NX PX ttl on all 5 nodes, preferably in parallel to reduce elapsed time
3. Count successful responses: if >= 3 nodes respond OK and elapsed time < TTL, lock is held
4. If majority quorum is not reached, dispatch DEL commands to all 5 nodes immediately

Fault Tolerance: Can still form a 3-node quorum after 2 host failures, assuming the remaining nodes are reachable and the algorithm's timing assumptions hold

Approach 3: ZooKeeper-Based Lock (Strongest Consensus Correctness)

Lock path: /locks/resource-name/

Algorithm:
1. Create an ephemeral sequential znode under /locks/{resource}/
2. Query all existing child znodes in the path
3. If your znode owns the lowest sequence number, you hold the lock
4. Otherwise, place an event watch on the znode with the next-lower sequence number
5. When that watched znode is deleted, receive event notification and recheck sequence
6. To release: delete your assigned znode
7. If client process crashes: ZooKeeper session expires and auto-deletes the ephemeral znode

Approach 4: etcd-Based Lock (Modern Raft Alternative)

// Grant a 30-second TTL lease
lease, _ := client.Grant(ctx, 30)

// Put key with lease attachment, which auto-expires if heartbeats cease
_, err := client.Put(ctx, "/locks/resource", "owner-id", clientv3.WithLease(lease.ID))

Redlock vs ZooKeeper vs etcd: When to Use Which

Selecting the appropriate coordination backend requires matching system safety requirements with latency and operational constraints.

• Redis SET NX: Single-node execution delivers sub-millisecond latency, but lacks fencing token generation while asynchronous replica replication risks losing active locks during failover. Suitable strictly for advisory coordination.
• Redlock: Gathers quorum across 5 independent Redis master nodes to survive up to 2 host outages. However, timing anomalies like clock drift and long GC pauses can still cause dual holders, requiring mandatory pairing with storage-layer fencing tokens.
• ZooKeeper and etcd: Consensus backed engines where ephemeral znodes or lease objects auto release upon session loss. Their ordered metadata can be used to derive fencing tokens, but the protected storage engine must enforce those tokens.

Fencing Tokens (Critical for Correctness)

When the token source is strictly ordered and the protected storage engine enforces it, fencing tokens protect against stale writes during timing anomalies. Without storage enforcement, a stale lock holder whose lease expired during a pause can still issue corrupting writes to persistent storage.

Execution Without Fencing Tokens:
  1. Client A acquires lock with 30s TTL
  2. Client A enters a 60s stop the world garbage collection pause
  3. Lock TTL expires at T=30s
  4. Client B acquires the released lock and writes updated state to storage
  5. Client A wakes up, mistakenly believes it still holds the lock, and overwrites Client B's update

Execution With Fencing Tokens:
  1. Lock service issues monotonic fencing token 33 to Client A
  2. Client A pauses, and the lock expires at T=30s
  3. Client B acquires lock and receives monotonic fencing token 34
  4. Client B writes to storage with token 34, causing storage to record last_seen_token = 34
  5. Client A wakes up and attempts to write with token 33
  6. Storage server evaluates token (33 <= 34) and rejects Client A's stale write

Lock Granularity and Hold Time

Lock granularity directly dictates system concurrency and partition resilience, requiring careful trade-offs between contention overhead and deadlock risk.

• Coarse-Grained Locking: Locking an entire database table or resource collection simplifies conflict detection but severely constrains throughput.
• Fine-Grained Row Locking: Locking individual entity rows enables high concurrency, but requires strict global key acquisition ordering to avoid deadlocks when transactions touch multiple rows.
• Short TTL with Watchdog Heartbeats: Provisioning short lease durations (10 to 30 seconds) coupled with background heartbeat renewal limits the blast radius and recovery delay if a holder crashes.