Design a Distributed Job Scheduler (Quartz / Airflow)

1. Job Store (CRUD API)

The Job Store serves as the durable source of truth where every submission lands before scheduling. It validates the job payload, enforces submission idempotency using a client key, persists metadata to PostgreSQL, computes the target execution window, and inserts the job ID into the Redis sorted set timer bucket when the occurrence falls within the upcoming 24 hour window.

2. Time Bucket Strategy (Redis ZSETs) ⭐

Partitioning scheduled jobs into granular time buckets prevents schedulers from scanning millions of future records on every tick. Each bucket is implemented as a Redis sorted set with the key timer:bucket:{minute_timestamp}:{shard_id}. In the baseline deployment, shard_id is 0. At scale, the shard suffix distributes timer buckets across Redis nodes. Job IDs are scored by their exact epoch execution second, so the scheduler can query only the active bucket instead of performing expensive full table scans.

3. Schedule Manager (Scanner Loop) ⭐

The Schedule Manager executes on a leader node elected through etcd or ZooKeeper. Every second, the scanner loop queries the current and previous minute buckets with ZRANGEBYSCORE to capture due jobs and boundary stragglers. It atomically claims each due job with ZREM, acquires a secondary dispatch lock, records the dispatched state and timestamp in PostgreSQL, and publishes the execution event to Kafka. A reconciliation loop republishes stale dispatched jobs when publication was not confirmed.

4. Job Dispatcher (Priority Based Routing)

Priority based dispatch isolates workloads into dedicated Kafka topics for critical, high, normal, and low execution tiers. This prevents heavy batch workloads from causing head of line blocking for interactive jobs and enables independent autoscaling of specialized worker pools.

5. Worker Pool & Executor Lifecycle

Workers subscribe to priority topics through dedicated consumer groups. Each worker follows a structured execution lifecycle to maintain safety during consumer rebalancing and process crashes.

1. Consume the job payload from the designated priority Kafka topic.
2. Acquire a bounded distributed execution lease with a fencing token to prevent concurrent runs if Kafka rebalances partitions.
3. Perform a database idempotency check by reading the current status and skipping execution if the record is already marked as completed.
4. Update the database status to running and record the execution start timestamp.
5. Execute the actual payload by invoking the target HTTP endpoint, script, or containerized task.
6. Upon successful completion, conditionally transition the job to completed, release the execution lease, trigger callbacks, and notify dependent DAG nodes. Only the successful state transition may decrement dependency counters.
7. Upon execution failure, increment the retry counter and schedule a future retry using exponential backoff with jitter, or push the task to the dead letter queue if retries are exhausted.

6. Recurring Job Scheduler

The recurring job engine calculates future executions from the cron schedule rather than from the actual completion time of the previous run. It accounts for daylight saving time transitions in the configured timezone, applies the recurring job's misfire and concurrency policies, writes the next execution instance to PostgreSQL, and places it in a Redis timer bucket only when the execution window falls within the upcoming 24 hours. Cancelling a recurring definition marks it inactive and removes its future timer entries. Any already running instance follows the configured execution policy.

7. DAG Dependency Resolver ⭐

The dependency resolver coordinates multi stage workflows where child tasks wait for upstream parent jobs to finish. When a job with dependencies enters the system, edge records persist in the database and an atomic Redis counter initializes to the number of prerequisite parents. Each successful parent completion decrements the child counter. When the counter reaches zero, all prerequisites are satisfied and the child job can be dispatched immediately.

8. Dead Letter Queue (DLQ)

Jobs that exhaust their configured retry budget are routed to the jobs.dead_letter topic. A dedicated dead letter consumer logs the complete error stack trace, increments failure metrics, and triggers on call alerts for critical workloads. Administrative interfaces allow engineers to inspect failed payloads, modify faulty parameters, and re-enqueue tasks into active processing queues.

9. Callback Service

The callback service delivers execution outcomes to upstream caller systems asynchronously via HTTP webhooks. It uses a stable callback delivery ID derived from the completed execution, performs three retries with exponential backoff, and sends that ID with every attempt so receivers can deduplicate repeated webhook deliveries. If callback attempts fail completely, the underlying job remains safely marked as completed because webhook delivery is decoupled as a best effort operational notification.

10. Effectively Once Execution

True end to end exactly once behavior across arbitrary external side effects requires transactional coordination. The scheduler instead targets effectively once side effects by combining at least once message dispatch with three defense layers: atomic ZREM claims on the Redis timer bucket, distributed dispatch locks using monotonic fencing tokens, and worker execution locks coupled with database idempotency checks that skip processing when records are already marked completed.

Event Bus Design (Kafka)

The event bus decouples the schedule manager from worker execution pools and absorbs sudden dispatch bursts. Topics are partitioned by priority to maintain strict workload isolation.

# Kafka Topic Architecture and Partition Topology
topics:
  jobs.critical:
    priority: 1
    partitions: 16
    consumers: "dedicated critical worker pool"
    use_cases: ["billing", "fraud alerts", "payment reconciliation"]

  jobs.high:
    priority: 2
    partitions: 32
    consumers: "shared high priority workers"

  jobs.normal:
    priority: 3
    partitions: 64
    partition_key: "job_id"
    consumers: "default worker pool"
    producers: "Schedule Manager after atomic ZREM claim and dispatch record"

  jobs.low:
    priority: 4
    partitions: 32
    consumers: "idle only batch workers"

  jobs.dead_letter:
    partitions: 8
    producers: "Worker Pool after max retries are exhausted"
    consumers: ["DLQ inspector", "on-call alerts", "manual retry console"]

configuration:
  replication_factor: 3
  min_insync_replicas: 2
  producer_acks: "all"
  producer_idempotence: true
  dispatch_path: "Redis ZREM claim -> PG status=dispatched with dispatched_at -> Kafka publish, while stale dispatched jobs are reconciled and republished idempotently"
  execution_path: "worker consume -> exec lease -> run payload -> PG completed/failed"