Design a Workflow Orchestration Engine (like Temporal/Cadence)

How Durable Execution Works: The Core Innovation

Durable execution via event replay is Temporal's core innovation. Walk through replay first, then explain saga compensation, human approval through Signals, and workflow versioning. Draw the server components after the replay model is clear.

// Traditional approach (stateless service):

  function processOrder(order) {
    payment = chargeCard(order)         // Process crashes here
    shipment = createShipment(order)    // Never reached
  }
  // On crash: partial state remains, requiring manual DB recovery

// Temporal approach (durable execution):

  function processOrder(order) {
    payment = workflow.executeActivity(ChargeCard, order)     // Step 1
    shipment = workflow.executeActivity(CreateShipment, order) // Step 2
  }
  // On crash: Temporal replays history, returns the recorded ChargeCard result,
  // and resumes execution at step 2 without re-running the completed activity}

HOW THIS WORKS:
  1. Worker replays event history:
     Event 1: WorkflowExecutionStarted
     Event 2: ActivityTaskScheduled (ChargeCard)
     Event 3: ActivityTaskCompleted (result: {payment_id: "pay_123"})
     Workflow execution paused, with history recorded up to here
  2. Worker re-executes deterministic code: 
     - the recorded ChargeCard result is consumed from history
     - createShipment emits the NEW command
  3. Worker returns command: ScheduleActivityTask(CreateShipment)

CRITICAL RULE: Workflow code must be DETERMINISTIC
  No random(), no direct wall-clock reads, and no direct I/O calls
  Use Temporal workflow APIs for time and signals. All external side effects execute through registered activities

Event History (Event-Sourced State)

The event history is the authoritative workflow state. Separate execution metadata can still support indexing, task management, and leases. Show a concrete event sequence so your interviewer sees replay mechanics rather than an abstract concept.

Every workflow execution is an append-only event log:

  Event 1:  WorkflowExecutionStarted     {input: {orderId: "42"}}
  Event 2:  WorkflowTaskScheduled
  Event 3:  WorkflowTaskCompleted        {commands: [ScheduleActivity(ChargeCard)]}
  Event 4:  ActivityTaskScheduled        {activityType: "ChargeCard"}
  Event 5:  ActivityTaskStarted          {attempt: 1}
  Event 6:  ActivityTaskCompleted        {result: {paymentId: "pay_123"}}
  Event 7:  WorkflowTaskScheduled
  Event 8:  WorkflowTaskCompleted        {commands: [ScheduleActivity(CreateShipment)]}
  Event 9:  TimerStarted                 {duration: 30 days}
  ...
  Event 42: TimerFired
  Event 43: WorkflowTaskCompleted        {commands: [CompleteWorkflow]}
  Event 44: WorkflowExecutionCompleted   {result: {status: "success"}}

This history is the authoritative logical workflow state. Execution metadata may still live in separate tables for indexing, leases, and task management.

2.5 PB History Retention and Archival

If the 10,000 workflow starts per second peak rate were sustained, history storage would become the dominant cost. Continue-As-New, archival, and retention controls are mandatory operational requirements.

If the 10,000 workflow starts/sec peak rate were sustained, event history grows ~86 TB/day, reaching ~2.5 PB over a 30-day retention window. Continue-As-New starts a fresh run with the same Workflow ID when active history becomes too large, carrying forward the required state and keeping replay bounded. Archival moves completed histories older than 7 days to S3 or GCS object storage, while the visibility index retains search metadata. Cassandra tuning requires increasing compaction throughput with ConcurrentCompactors, separating history and visibility keyspaces, and monitoring SSTable count per shard. Without proactive history bounding, archival, and retention management, history storage cost dominates the entire platform budget.

Saga Pattern (Distributed Compensation)

Saga compensation belongs directly in workflow code as standard try/catch blocks rather than being scattered across message queues, because orchestration wins when rollback paths must remain auditable. See Distributed Transactions: 2PC vs Saga and Microservices Patterns: Saga and Outbox.

// Problem: Book a trip = Flight + Hotel + Car Rental (3 services)

Temporal Saga:
  function bookTrip(trip) {
    flight = workflow.executeActivity(BookFlight, trip)
    try {
      hotel = workflow.executeActivity(BookHotel, trip)
    } catch (e) {
      workflow.executeActivity(CancelFlight, flight)  // compensate
      throw e
    }
    try {
      car = workflow.executeActivity(RentCar, trip)
    } catch (e) {
      workflow.executeActivity(CancelHotel, hotel)    // compensate
      workflow.executeActivity(CancelFlight, flight)  // compensate
      throw e
    }
    return {flight, hotel, car}
  }

Why better than choreography (event driven):

• Compensation logic stays in one workflow definition instead of being scattered across event consumers
• Temporal can retry compensation activities even when a worker crashes
• Compensation failures remain visible and can be retried or escalated without losing workflow history

Activity Idempotency (At-Least-Once Delivery)

Activities execute at least once by design. Idempotency keys must remain stable across retries rather than incorporating attempt numbers, which would otherwise cause duplicate operations.

Workflow start idempotency:
  1. Choose a stable workflow_id for the business operation (for example order-42).
  2. Start the workflow with an explicit workflow-ID reuse/conflict policy.
  3. If the client retries the same start request, reuse the same workflow_id so the server does not create duplicate business workflows.

Activity idempotency:
  Problem: Temporal guarantees at least once activity execution. If a worker crashes after the downstream side effect commits but before activity completion is recorded, Temporal may retry the activity.

Idempotency Key Pattern:
  1. Generate a stable key before calling the activity. NEVER include attempt_number because every retry must reuse the same key:
     key = sha256(workflow_id + run_id + activity_id)
  2. Pass key to downstream: POST /charge { idempotency_key: "abc123" }
  3. Downstream stores (key -> result) for TTL=24h or for the maximum retry/recovery window, whichever is longer
  4. Duplicate request returns the prior result with no duplicate side effect

Non-retryable errors:
  HTTP 400: non_retryable (retrying will not resolve the client error)
  HTTP 503: retryable with exponential backoff (transient failure)

Visibility Events (Kafka Fan-out)

Kafka is used for visibility fan-out rather than workflow durability. The append-only workflow history is the authoritative source of truth, so be careful not to conflate messaging queues with orchestration state. Learn more about queuing mechanics in Message Queue Fundamentals.

Topic: workflow-visibility-events
  Partitions: 64 (partition by workflow_id)
  Events: WorkflowExecutionStarted, Completed, Failed, TimedOut, Cancelled
  Retention: 30 days (ops dashboards + billing meter)
  Producers: Temporal server components after the corresponding history append
  Consumers: observability stack, SLA alerting, cost attribution

Topic: activity-task-events
  Purpose: External operational telemetry only, because worker task dispatch still uses Temporal task queues
  Partitions: 128 (partition by task_queue)
  Events: ActivityTaskScheduled, Started, Completed, Failed, TimedOut
  Consumers: worker autoscaling, retry alerting, retry policy tuning

Note: Temporal's source of truth is append-only event history in Cassandra or PostgreSQL,
  not Kafka. Kafka fans out visibility signals to external systems only.

Start workflow: Client -> Frontend -> history write -> return run_id (< 100ms target)
  Workers poll task queues. Activity side effects run outside workflow code and are isolated from deterministic workflow replay