GenDurable.Scheduler (gen_durable v0.1.2)

Copy Markdown View Source

Per-queue feeder + executor loop (spec §6). Backpressure-driven: it claims work from the database into a small in-memory buffer, then spawns at most concurrency supervised Tasks at a time, draining the buffer as slots free.

Aggressiveness knobs

Throughput is not capped by poll_interval — a finished Task immediately refills from the buffer (no round-trip) or picks a fresh batch. The poll timer only governs how promptly newly-inserted work is discovered while idle. The knobs let a deployment trade DB chatter, latency, and fairness:

  • concurrency — max Tasks running at once (the executor width).
  • prefetch — extra rows to claim and hold in the buffer beyond the running slots. 0 (default) means no over-fetch: identical to picking exactly the free slots. Raising it batches picks and absorbs completion bursts, at the cost of cross-node fairness and priority freshness (claimed rows are invisible to other nodes). Buffered rows are heartbeated, so they never go stale regardless of depth — depth is decoupled from lease_ttl.
  • min_demand — batch gate: don't pick unless at least this many slots are free (so picks come fat, not one row at a time). Ignored when the queue is fully idle, to avoid starvation.
  • poll_interval — base idle poll. max_poll_interval — backoff ceiling: an empty pick on a fully idle queue doubles the interval up to this cap, then a non-empty pick (or any in-flight work) snaps back to the base. This is the lever that cuts idle DB load. (LISTEN/NOTIFY is banned, so polling is the only discovery path — the point is to poll adaptively, not constantly.)

Buffered and in-flight rows are both executing + leased; the heartbeat (one batched UPDATE per tick over buffer ++ in_flight) keeps every claimed row alive. If this scheduler dies, the whole claimed set waits one lease_ttl for the reaper — so deeper buffers mean a larger crash blip, bounded by the (short) TTL, not by the buffer depth.

partition_key serialization (spec §6) is handled per-job at execution time: a session-level advisory lock is taken on a checked-out connection held for the whole step, and released after the outcome commits. If the lock is contended, the row is returned to runnable and skipped.

Summary

Functions

Returns a specification to start this module under a supervisor.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

start_link(opts)