GenDurable.Poke (gen_durable v0.2.15)

Copy Markdown View Source

The poke transport: how "runnable work exists NOW" reaches schedulers instead of leaving discovery to the poll timer. Configured per instance via the :poke engine option:

  • :local (default) — poke the caller's node only. Zero moving parts beyond the per-instance emitter; other nodes discover the work on their next poll.
  • :cluster — poke every node's schedulers of the queue over Erlang distribution. Membership rides an OTP :pg scope, so only nodes that actually run the queue are reached. Without distribution it degrades to :local.
  • {:redis, url_or_opts} — publish over Redis Pub/Sub, for clusters without Erlang distribution. Requires the optional :redix dependency. The caller's node is poked directly (no Redis round-trip, and a Redis outage cannot lose local pokes); the publish carries the origin VM's token so subscribers skip self-originated messages.
  • :postgres — publish over Postgres LISTEN/NOTIFY, for clusters that share a Postgres but have neither Erlang distribution nor Redis. No extra dependency (reuses the repo's Postgrex). The caller's node is poked directly; other nodes receive a NOTIFY on the instance channel, tagged with the origin VM token so a node drops its own message. A subscriber (PgListener) holds a dedicated Postgrex.Notifications connection and turns foreign notifications into local pokes.
  • :none — never poke anyone, ever. No emitter, no listener, no cross-node traffic; the poll interval is the sole discovery mechanism. For deployments that accept poll latency, or where NOTIFY/distribution/Redis are unwanted.

Out-of-band emission

Every poke is emitted out of band: dispatch/2 and dispatch_rows/2 are async casts to a per-instance Emitter process, so the caller (an insert, a signal wake, a batched outcome flush) returns immediately having done zero transport work on its hot path. For :postgres this is not just tidy — it moves the NOTIFY off the insert's commit path, where it would otherwise serialize commits fleet-wide on the database-global notify-queue lock.

The Emitter coalesces per queue over a short in-VM window (@window_ms): a burst/stream of pokes for one queue fans out at most once per window per node. This send-side window replaces the old Redis SET NX PX distributed dedup lock — fan-in is now bounded by node count (each node emits ≤ 1 broadcast per queue per window), not by the insert rate.

Besides inserts, pokes announce every engine-driven wake: a signal flipping a parked row, a fan-out's freshly-inserted children (in their queues), and a parent whose join the last child just completed. Delivery is best-effort in every mode — a lost poke costs one poll interval, never correctness. The poll remains the discovery floor for what a poke cannot see (retry backoffs, the reaper's wakes, remote events under :local) — and is the only discovery mechanism under :none.

A poke only wakes an idle scheduler — the idle → work transition it exists for. A scheduler with work in flight drops it and rediscovers new work on its next task completion (or poll), so a fan-out never becomes N nodes all picking on every insert. That receive-side gate composes with the Emitter's send-side window.