sql-job-queue

Use a jobs table plus a job_deps edge table for the dependency DAG, a ready-set query to find runnable jobs, and an atomic lease transaction to claim them. A tick loop scans, ranks by fair-share, leases, dispatches, and reclaims stale leases — no broker required.

FOR UPDATE SKIP LOCKED lets a worker lock and claim a ready row while skipping rows already locked by other workers, so concurrent claims never pick the same job. It works on PostgreSQL and MySQL 8.0+ with InnoDB, but is a syntax error on MySQL 5.7.

No, with_for_update is a silent no-op on SQLite because SQLite has no row-level locks — it does not error, it just does not lock. Use BEGIN IMMEDIATE to take the database write lock at transaction start so the claim serializes across workers.

MySQL defaults to REPEATABLE READ isolation, so a claim SELECT reads a snapshot taken at transaction start and can miss rows committed afterward. Set READ COMMITTED for the claim transaction so each statement sees freshly committed ready rows.

Workers renew a lease with a heartbeat while running; a dead worker stops renewing, so its lease expires. A reclaim sweep each tick returns expired-lease jobs to the ready state, while a separate per-dispatch timeout bounds hung-but-alive jobs.

Use a broker like Celery or RabbitMQ for short fire-and-forget tasks with static enqueue and multi-machine fan-out. A DB-backed queue fits long-running stateful jobs with dependency-driven readiness on a single box where the database is the only durable store.