Operations

What actually happens in production, and the knobs you own.

Delivery guarantees, stated plainly

Belay is at-least-once. A worker that dies after doing the work but before acking will have the job re-run — with the crucial mitigation that memoized steps make the re-run skip everything the first run recorded. Two windows to know about:

• Crash orphans: a killed node's running jobs sit unreachable until their lease expires (lease_ttl, default 30s), then any node's sweeper reclaims them for retry. That's the worst-case added latency for a crash.
• Zombie acks: a worker cut off from the database long enough to lose its lease might come back and try to ack; acks are fenced by attempt number and rejected. You'll see a stale ack warning — that's the fence working.

Shutdown

On SIGTERM, producers stop claiming first, running jobs get shutdown_grace (default 15s) to finish, and whatever survives is killed — its lease expires and another node picks it up. Set shutdown_grace just under your platform's kill timeout.

Dispatch latency

Agent workloads are bursts of short tasks, so dispatch overhead is the product. Belay layers three mechanisms — each optional layer only buys latency; polling remains the correctness floor:

1. Local pokes (always on): inserts, releases, and completions poke producers on the same node instantly, and across BEAM nodes when you run distributed Erlang.
2. Adaptive polling (always on): producers poll at busy_poll (25ms) while work flows and decay to poll_interval (500ms) when idle — burst latency without idle database load. Poke storms coalesce into single claim rounds.
3. pg_notify accelerator (opt-in, notifiers: [:local, :postgres]): wake-ups ride the database itself, for fleets that share Postgres but not an Erlang cluster. One dedicated listen connection per node (auto-reconnecting; point listen_url: past any transaction pooler); payloads are a queue name or job id, far under NOTIFY's limits.

Measured insert→result round trips (bench/run.sh, stock settings, local Postgres 16, M1):

Honest caveats, with receipts: Postgres serializes NOTIFY-issuing commits on a global lock (the Recall.ai outage class; ~190k vs ~350k TPS with/without it in pgsql-hackers benchmarks — present through PG 18, and PG 19's fix covers only the listener-wake side). Belay stays far from that zone by coalescing to one pg_notify per queue per insert batch/completion, and the accelerator is droppable at any time — [:local] keeps the adaptive-polling latencies above. The pooler rule: pg_notify through PgBouncer is fine; only the listen connection needs a direct line (listen_url:). Notifications are at-most-once by design, which is exactly why the polling floor is non-negotiable. await_result wakes on result notifications and otherwise re-checks on a 5ms→200ms backoff. One structural advantage worth naming: workers are BEAM processes in your running app — there is no sandbox cold-start tier between dispatch and execution at all.

Sizing the loop

Throughput levers: chunks and adaptive limits

Chunk workers turn N claimed jobs into one worker invocation — one bulk INSERT, one batch-priced embeddings call:

use Belay.Worker, queue: :embeddings, chunk: [size: 100, gather_ms: 500]

The producer gathers claimed jobs per worker up to size, waiting at most gather_ms for stragglers (full chunks dispatch immediately). Gathered jobs are already leased, so a crash mid-gather reclaims them like any other crash; the gather window is clamped to half the lease TTL. See Belay.Worker for the run_chunk/1 contract including per-job partial failure.

Adaptive concurrency scales a queue's per-node limit with load:

queues: [ingest: [limit: [min: 2, max: 50]]]

Saturated claim rounds double the limit toward max; an idle queue decays back to min. Leaderless — each node adapts its own limit, and global_limit, rate limits, and partition fairness still bound the fleet exactly. Scale changes emit [:belay, :queue, :scale].

For policy-driven scaling (queue depth thresholds, business hours, an operating agent watching costs), drive Belay.Queues.put/3 from your own logic — it is runtime CRUD reconciled by every node. The bundled MCP server does not currently expose queue-definition CRUD.

Retention

Terminal jobs are pruned with their steps and events by the sweeper:

{Belay,
 ...,
 retention: [succeeded: 86_400, failed: 7 * 86_400, cancelled: 7 * 86_400],
 signal_ttl: 7 * 86_400}

Set a state to :infinity to keep it forever (and own the table growth). Incomplete jobs are never pruned.

The dashboard

One child spec, zero dependencies:

{Belay.Dashboard, belay: MyApp.Belay, port: 4004, token: System.fetch_env!("DASH_TOKEN")}

Live queue tiles (with each queue's limits and rates), a filterable job list, a drawer with the full journal - steps with costs, events, errors, children - a rendered workflow DAG, and retry/cancel/signal/steer actions (mutations require either a dashboard token or the same authorizer: contract as the MCP server; tokenless dashboards are read-only). It binds 127.0.0.1 by default and speaks plain HTTP; front it with your proxy for remote access.

Day-2 tooling

Belay.stats(MyApp.Belay)
#=> %{"ai" => %{"ready" => 12, "running" => 2}, "default" => %{"succeeded" => 1043}}

Belay.list_jobs(MyApp.Belay, state: :failed, limit: 20)
Belay.retry_job(MyApp.Belay, job_id)
Belay.cancel_job(MyApp.Belay, job_id)      # immediate when parked, cooperative when running
Belay.pause_queue(MyApp.Belay, :ai)    # local producer stops claiming
Belay.steps(MyApp.Belay, job_id)       # the journal, costs included
Belay.events(MyApp.Belay, job_id)      # the emitted stream
Belay.Replay.dry_run(MyApp.Belay, job_id)  # what did it actually do?

The same inspection surface is exposed to AI assistants and scripts through mix belay.mcp. Mutations are disabled unless you pass --authorizer MyGuard or explicitly opt in with --allow-mutations.

Telemetry

Events: [:belay, :job, :start | :stop | :exception] with durations and job metadata. For one-line structured logs:

Belay.Telemetry.attach_default_logger(:info)
# belay job=812 worker=MyApp.Agent queue=ai state=succeeded attempt=1 duration=8143ms

Postgres notes

• The claim path is FOR UPDATE SKIP LOCKED over partial indexes; the migration creates every index the engine relies on.
• LISTEN/NOTIFY is never load-bearing. The optional accelerator uses it; correctness never does. PgBouncer transaction pooling, RDS Proxy, and serverless Postgres are all fine (see Dispatch latency for the one direct-connection rule when you enable the accelerator).
• Retention is a DELETE ... LIMIT batch per sweep; autovacuum handles the rest at moderate scale. At very high throughput, shorten retention before reaching for anything exotic.
• Partitioned claims are exact: per-key allowances are computed with a window-function ranking inside the claim transaction, under the queue's advisory lock — heavy key skew cannot starve minority keys.
• Measured throughput (bench/throughput.exs): ~416 trivial jobs/s end-to-end on a laptop with 3 worker processes and unbatched acks; each job costs a claim and an ack transaction. Batched acking is the known post-1.0 lever for multiples beyond that.