Running on Multiple Nodes

Copy Markdown View Source

Baton is a library, not a service: its OTP application supervises no processes, and everything durable — the DAG, results, stats, checkpoints, completion claims — lives in Postgres. "Connecting to Baton" is connecting to the database, so any node running your application with the same Oban instance and repo can insert and execute workflows. There is no Baton singleton to overwhelm and no Baton-side clustering to configure.

That safety is by construction, and it rests on a handful of invariants the host application must keep. This guide collects them.

Why multi-node already works

Every cross-node coordination point is either a Postgres atomic claim or a best-effort latency optimization with a timer fallback:

  • Exactly-once completion — the workflow-finished announcement is an INSERT ... ON CONFLICT DO NOTHING on workflow_completions (Baton.Completion). Every node may race to announce; exactly one wins.
  • Idempotent sweeps — each node runs its own Baton.Plugin inside its own Oban instance. The sweeps cancel by job id and announce through the same atomic claim, so N copies are harmless, and first runs are randomly staggered so nodes don't synchronize.
  • Cross-node wake-ups — Baton.Dispatch nudges queues through Oban.Notifier (Postgres LISTEN/NOTIFY under the default notifier), so a step completing on node A wakes a parked dependent's queue on node B. A lost notification costs only latency: the parked snooze timer and Oban's Stager are the correctness fallback.
  • Node-local caches are only caches — Baton.ResultCache is content-addressed and node-local; a miss or a cold node costs a read, never correctness.

Failover

A node dying mid-step is handled by three layers:

  1. Oban.Plugins.Lifeline (host config) rescues jobs the dead node left executing. Without it, orphans sit until manual intervention — run it.
  2. The result-first idempotency guard — Baton.Worker checks for an already-stored result before performing, so a rescued job whose previous attempt died after the LLM answered and the result was written does not pay for the call again.
  3. Batch checkpoints — a batch step's provider batch id is checkpointed in workflow_nodes.checkpoint, so any node can resume polling a batch another node submitted.

The threshold ordering invariant

Three durations must stay ordered, cluster-wide:

longest step timeout  <  stale_executing_threshold_seconds  <  Lifeline rescue_after

If the stale threshold undercuts a step's real runtime, Baton.Check treats a live dep as dead and cancels its dependents; if rescue_after undercuts the stale threshold, Lifeline re-runs jobs the stale check already wrote off. Baton.Config warns about the first violation at boot.

Rate limiting across nodes

Oban queue limits are per node: queues: [llm: 8] on three nodes is 24 concurrent LLM jobs. The queue limit is a coarse concurrency ceiling, not the budget — the budget is Baton.RateLimiter, and for a common cluster-wide limit the implementation must be backed by shared storage:

  • Postgres-backed buckets work unchanged — every node's acquire/3 debits the same rows, so the provider sees one combined budget no matter how many nodes submit. (Redis or any other shared store works the same way.)
  • ETS/GenServer-backed limiters do not — each node enforces its own copy of the limit, and the provider sees N× the intended traffic.

Also size the database connection pool for the fleet: every node brings its own Ecto pool, and nodes × pool_size must stay under Postgres's max_connections with room for Oban's own connections.

Event listeners must be leader-gated

Baton.Events broadcasts through Phoenix.PubSub, which delivers to every subscriber on every node. A dashboard rendering those events N times is fine; a process that writes in response to them — reconciling state, inserting follow-up work — will duplicate the write once per node unless exactly one node acts. Gate such writers on a cluster-wide leader check, e.g. Oban's own peer election:

def handle_info({:workflow_completed, _} = event, state) do
  if Oban.Peer.leader?(Baton.Config.oban_name()) do
    act_on(event)
  end

  {:noreply, state}
end

Check leadership at the write, not at init/1, so leadership can move between nodes without restarting the process. And remember PubSub's default PG2 adapter rides distributed Erlang — if your nodes aren't clustered (libcluster, DNSCluster, …), broadcasts stay node-local and a leader-gated listener on a non-leader node sees nothing.

Checklist

  • [ ] All nodes run the same release (same step modules, same Baton version) against the same Postgres and Oban instance name.
  • [ ] Oban.Plugins.Lifeline is enabled, with rescue_after above stale_executing_threshold_seconds, which is above the longest step timeout.
  • [ ] The configured Baton.RateLimiter uses shared storage (Postgres, Redis) — not ETS or process state.
  • [ ] Per-node queue concurrency × node count is an acceptable ceiling, and total DB connections fit max_connections.
  • [ ] Distributed Erlang is actually connected in production if anything subscribes to Baton.Events across nodes.
  • [ ] Every process that writes in response to a Baton.Events broadcast is leader-gated at the write site.