Design Live Likes & Reactions

1. Toggle Reaction Race Conditions ⭐

If a user rapidly taps Like, Unlike, and Like again, out of order packet delivery could corrupt the count. We use action_id and a monotonic action_seq to preserve user intent instead of relying on packet arrival order. The service treats the accepted sequence at the user and content serialization boundary as authoritative, including across multiple devices. The same sequence also makes retries safe:

# Rapid client action sequence: Like, Unlike, Like within 100ms

Accepted chronological intent:
  - T=0ms: LIKE has action_seq=1 and becomes the current reaction.
  - T=30ms: UNLIKE has action_seq=2 and removes the current reaction.
  - T=60ms: LIKE has action_seq=3 and becomes the current reaction again.
  Result: Net heart count is +1.

Network out of order delivery:
  - Each mutation carries a unique action_id and a monotonic action_seq.
  - If action_seq=3 arrives before action_seq=1 or action_seq=2, Redis applies 3 and ignores the older actions.
  - A retry with the same action_id or a stale action_seq becomes a no-op.
  - Redis set membership is not sufficient for ordering user intent. The action sequence is the authoritative ordering signal.

Multiple devices:
  - The service serializes mutations at the user/content boundary and uses the latest accepted action_seq.
  - Ties are resolved at the server serialization point, and the accepted sequence is persisted with the current reaction state.

2. Viral Hot Keys (1M+ events on a single stream) ⭐

When 1M users react to a single stream, a single content counter becomes a Redis hot shard. We reduce command volume with batching and distribute the remaining work across sharded sub-counters:

# Scaling hot keys under extreme load (1M reactions/sec on a single content_id)

Solutions:
  1. In-Process Aggregation (Reaction Service):
     - Buffer reactions in service memory for 100ms.
     - Send one batched counter delta with cumulative counts (for example, +342).
     - Reduces 1M individual Redis commands to ~200 batched Redis operations/sec across 20 service nodes.
     - Batching alone still targets the same Redis shard, so combine it with counter sharding when one content_id is hot.
     - The same hot content can also be promoted to a multi partition Kafka key so event ingestion does not serialize on one partition.

  2. Sharded Sub-Counters:
     - Split the single content counter into N shard keys:
       reaction_count:{cid:0}, reaction_count:{cid:1}, ..., reaction_count:{cid:7}
     - Choose the shard with hash(user_id) % 8.
     - Keep matching reaction state and counter keys for a shard on the same Redis Cluster slot.
     - Query: SUM across all 8 sub-counters.

  3. Local Redis Sidecars with Write Behind:
     - Accept writes into local host Redis instances.
     - Background tasks sync accumulated deltas to the central cluster.
     - Treat this as an optional optimization because a host failure can lose unflushed state unless another durable path has already acknowledged the mutation.

3. Redis Deduplication Set Memory Exhaustion ⭐

Storing 50M user IDs creates large memory pressure, and Redis set overhead makes the raw 800MB UUID payload estimate optimistic. We solve the memory bottleneck with an advisory Bloom filter and exact verification. Durable Cassandra state uses bounded content shards so a mega stream does not create one oversized partition:

For a mega stream, the durable current-state table uses a larger shard count such as N=128, while Redis serving counters may use a smaller operational shard count such as 8. The two shard counts serve different bottlenecks.

# Deduplication memory optimization for 50M unique reactors

Challenge:
  - Storing 50M UUID payloads alone at 16 bytes each requires ~800 MB per stream.
  - Actual Redis set memory is higher because of object and hash-table overhead.
  - Bloom filtering reduces the auxiliary membership footprint, but it does not replace the exact
    current reaction state needed to process type changes and removals. That exact state is a
    separate memory consideration for mega streams.

Solutions:
  1. Bloom Filters:
     - Allocate approximately 15 bits per element (~93 MB total).
     - This is more conservative than a 1% false positive target when configured with an appropriate number of hash functions.
     - Treat the Bloom filter as an advisory front gate. A negative result can skip an exact lookup,
       while a positive result triggers an exact current-state check before rejecting a mutation.
     - For mega streams, keep exact Redis reaction state for the active-user working set and use
       Cassandra for cold-user verification when that state is evicted. If the durable state is still
       catching up, defer or retry rather than deciding from the Bloom result alone.
     - Never use a Bloom positive as proof that a user has already reacted because false positives are possible.

  2. TTL Cleanup:
     - Apply 24 hour expiration to live stream deduplication keys after the event ends.

  3. Split-Tier Verification:
     - Maintain exact in-memory reaction state only for active live users when memory permits.
     - Query Cassandra for durable current reaction state when Redis state is unavailable.
     - Keep the serving state and Bloom membership key separately so reducing Bloom memory does not
       accidentally remove the exact state required for an active user's type change.