Changelog

Copy Markdown View Source

2.0.0 — 2026-08-01

Fixes for everything found by a post-1.0 adversarial review of the whole library. Every item ships with a regression test that fails on 1.0.0.

No database migration is required. The schema and wire contract are unchanged, so 1.0.x and 2.0.0 nodes can run against the same database during a rolling upgrade (but see the note at the end of the breaking list).

This is a major release because four fixes change the behaviour of setups that work today. Three of them are secure-by-default or don't-corrupt-silently decisions that cannot be made without a break; each has a one-line escape hatch.

Breaking changes

  1. The dashboard only answers to hostnames you allow. A loopback-bound dashboard (the default) now serves only localhost, 127.0.0.1, and [::1]. Reaching it under any other name — an /etc/hosts alias, or a same-host reverse proxy that forwards the original Host (nginx's proxy_set_header Host $host) — returns 403 with a message naming the option.

    Why: comparing Origin to the request's own Host cannot distinguish a genuine localhost request from one aimed at the dashboard by a name an attacker controls, so it never established which host was asked for.

    Upgrade: name your hosts, or opt out.

    {Belay.Dashboard, belay: MyApp.Belay, allowed_hosts: ["belay.internal"]}
{Belay.Dashboard, belay: MyApp.Belay, allowed_hosts: :any}   # 1.0.0 behaviour

    Dashboards bound anywhere other than loopback are unaffected by default — that is the documented "behind a proxy you trust" deployment, and it keeps 1.0.0 behaviour unless you set the option.

  2. GET /api/jobs refuses out-of-range parameters. ?limit= above 500, below 1, or non-numeric now returns 400; before_id and parent_id must parse as integers. Previously limit was unbounded, so one request could read an entire jobs table into memory.

    Why not clamp instead: quietly returning 500 rows to a caller that asked for more looks like the end of the result set, which is a data-loss bug in whatever is reading the endpoint. A refusal is visible.

    Upgrade: page with limit ≤ 500 and before_id.

  3. Belay.Workflow.insert/2 raises when a member is dropped. A member whose unique: key collides with an existing incomplete job used to be skipped silently, returning {:ok, jobs} with that name missing — its dependents then stayed held forever and status/2 never reported done. It now raises ArgumentError and inserts nothing.

    Upgrade: if you combine unique: with workflow members and expect collisions, catch the error, or scope the key so members cannot collide.

  4. Running jobs now get :shutdown_grace on the way down. The option was documented and parsed but consumed by nothing, so jobs were killed mid-step on every deploy and burned an attempt via lease expiry. Shutdown now waits for in-flight jobs, up to the configured grace (default 15s).

    Upgrade: nothing, unless your orchestrator's termination grace is shorter than Belay's — raise it, or set shutdown_grace: 0 to keep 1.0.0 behaviour.

Two smaller ones: Belay.Crons.put/5 now rejects a module that is not a Belay.Worker (it previously accepted it and then stopped crons from firing), and during a rolling upgrade a 2.0.0 node writing a dynamic cron with budget:/unique: options produces a row 1.0.x nodes cannot read. In practice no such row exists, because writing one on 1.0.0 raises — that is one of the bugs fixed here.

Fixed

  • Maintenance could stop running. A job whose kind no longer resolved to a loaded module — a worker renamed or removed in a deploy, or a row written by another SDK — could prevent lease reclamation, the parent-wake backstop, and retention pruning from completing. Such jobs now retry on the default schedule and fail normally with the real error journaled.
  • Signals delivered to a job's workflow scope did not wake it. await/3 reads a priority list of scopes but parks on one, and delivery only matched that one, so Belay.signal(inst, "wf:<id>", name) woke nothing once the job had parked — and with no deadline, that job never ran again. Delivery now matches every scope the await reads.
  • A timed-out await could never park again. Await fields survived non-await outcomes, so after a timeout was retried every later attempt returned {:error, :timeout} immediately until max_attempts ran out. Terminal rows no longer advertise a live await either.
  • get_signal was non-deterministic on PostgreSQL. A signal name present in more than one scope returned whichever row the planner reached first, which also diverged from the Memory adapter. Scope priority is now explicit.
  • Wrong or rotated encryption keys burned every attempt silently. Decryption ran outside the outcome path, so the job was reclaimed as an opaque lease_expired instead of failing with the real error. One undecryptable job also no longer affects the chunk it was gathered into.
  • The MCP server now validates tool arguments and cannot be taken down by a bad one. Arguments are checked against the advertised inputSchema, and failures return an isError result instead of ending the session.
  • Dynamic manual: true queues were claimed automatically. Only static config was filtered, so a runtime manual queue got producers on every node.
  • Dynamic crons with budget: or unique: options crashed on PostgreSQL while working on Memory.
  • 500 responses from the dashboard no longer include internal exception messages; they log the detail with a reference instead.
  • Two executors for the same job on one node no longer cause the live one to be killed; a zombie executor's emit/2 can no longer fail the live attempt; and Belay.Txn.insert/3 no longer raises inside the caller's transaction when a windowed unique key dedupes across a window boundary.
  • Clearer errors for unique: [within: ...] and misconfigured encryption keys, which previously raised FunctionClauseError and Protocol.UndefinedError from inside the error path itself.
  • The Oban migrator carries meta and tags across instead of dropping them (and still works against tables that predate those columns).
  • verify/traces.exs had been failing partway through since the pre-1.0 cancel → cancel_job rename.

Added

  • :allowed_hosts on Belay.Dashboard — the Host values it will answer to, which is what actually pins the dashboard to the names you expect.
  • :max_result_bytes — reject job results above a size. Unset by default; oversized results are logged either way. Step values keep their own cap.
  • mix belay.migrate — the task two schema errors already told users to run.
  • verify/signal_scopes/, a Lockstep model of the signal/await scope protocol: the 1.0.0 delivery rule loses the wake at iteration 1, matching the read set without locking it still races, and the shipped fix survives 400 explored schedules.

Changed

1.0.0 — 2026-07-25

First stable release. The public API and wire contract are now frozen for the 1.x series.

Changed

  • The public API is frozen and machine-checked. Public functions now have documentation and typespecs, with a regression test enforcing both. Belay.get_job/2 consistently returns {:error, :not_found} for a missing job; await_result/3 propagates that result immediately.
  • The wire contract is stable. Schema migration revision 3 introduces belay_meta.contract_major, separating protocol compatibility from additive migration bookkeeping. PostgreSQL storage checks compatibility at startup and emits an actionable failure before producers start.
  • Migrations are deploy-safe. migrate!/1 now applies all pending revisions atomically under a transaction-scoped advisory lock, so multiple nodes may invoke it concurrently.
  • PostgreSQL support is explicit. PostgreSQL 14–18 are supported; CI runs the PostgreSQL suite on 14, 16, and 18 across the Elixir/OTP compatibility matrix.
  • Package verification now tests the artifact users receive. CI unpacks the Hex tarball, compiles it, and builds its docs. The formal-verification overview required by ExDoc is included without shipping the nested verification project.

  • Queue claims now emit [:belay, :queue, :claim, :start | :stop | :exception] telemetry spans with queue, demand, and claim-count metadata.

  • Documentation now consistently names steer_job/3 and reports the current suite counts.

1.0.0-rc.6 — 2026-07-23

Changed

  • Breaking (API naming). Two job-control functions gained the _job suffix so the family is consistent and matches Oban. Belay.cancel_job/2 and Belay.steer_job/3 replace the former cancel and steer. Update call sites; behaviour is unchanged. This is a deliberate pre-1.0 rename to freeze a consistent surface before the stable release.

1.0.0-rc.5 — 2026-07-21

Launch-prep release: formal verification layer, chunk workers, adaptive concurrency, the Oban migrator, dashboard v2, and the fixes surfaced by a pre-launch adversarial review.

Added

  • Formal verification layer (verify/). A TLA+ model of the durable-execution core (journaling, budgets, cooperative cancel, crash/reclaim, settlement), generated from source + execution traces and exhaustively checked by TLC — 187,975,659 distinct states (depth 49), zero violations. Validated by mutation: reverting any of the three known production bugs in the model (the budget crash window; the cancel-request clear; the operator-retry semantics) yields a TLC counterexample matching the real-world failure step for step.

  • Schedule-explored wake protocol (verify/wake_protocol/). The parent-wake protocol under Lockstep controlled concurrency: the pre-fix count-gated protocol loses the wake on a found, saved, replayable schedule (PCT iteration 1); the shipped unconditional protocol survives every explored interleaving without the sweeper reconciler — evidence the reconciler is a backstop, not load-bearing.

  • SCHEMA.md — the wire contract. The Postgres schema specified as a versioned protocol: tables, state machine, annotated SQL for every operation, advisory-lock discipline, the two soak-learned race rules, and a conformance path (the soak driver verifies any foreign worker SDK by reading the database). Groundwork for Python/TypeScript SDKs as thin contract implementations rather than rewrites.

  • Belay.Codec — cross-language value envelope. Step values and job results now decode as Erlang term format (leading byte 131, written by Elixir) or UTF-8 JSON (written by any other SDK); tested with foreign-written JSON rows replaying through the engine.

  • Chunk workers. chunk: [size: n, gather_ms: t] + run_chunk/1: the producer gathers claimed jobs per worker and runs them as one invocation — one bulk INSERT instead of hundreds, one batch-priced embeddings call instead of a hundred singles. Per-job outcome maps retry only the failed jobs; gathered jobs are already leased, so crashes mid-gather reclaim cleanly. Full chunks dispatch without waiting for the gather window.

  • Adaptive per-queue concurrency. limit: [min: a, max: b] scales each node's limit up while claim rounds come back saturated and decays it when the queue idles — leaderless, like the burst-poll cadence it mirrors, and exactly bounded by global_limit/rate/partition limits. Policy-driven scaling stays a recipe: drive runtime Queues.put/3 from application code.

  • Dashboard v2. KPI strip with live throughput and spend-rate sparklines (trailing-window snapshots persisted client-side), queue rows with limit-utilization bars (adaptive ranges shown as min–max), a two-line job list with args previews, durations, per-job cost badges and state-chip filters, and a restyled journal-timeline drawer. The spend KPI rides the same single stats scan the counts already paid for (queue_stats now returns spend sums); job summaries gained max_attempts, input_preview, spent_usd_micros, started_at. ?sse=0 renders a static snapshot for screenshot tooling.

  • mix belay.migrate_oban — move an Oban installation's pending work in one command: dry-run analyzer (state census, per-worker port verification, executing-row warnings), faithful conversion (schedules, retry counts, errors, priority), --map renames, and idempotent re-runs via meta.migrated_from_oban_id. History stays put by design — the guide documents the archive pattern instead.

Fixed

  • Dashboard and MCP mutations now fail closed. A tokenless dashboard and the MCP server are read-only unless an authorizer or explicit mutation opt-in is configured. Dashboard writes reject query-string credentials, cross-origin requests, non-object/malformed JSON, bodies over 1 MiB, large header sets, and ambiguous request framing. Bearer-token comparison uses a constant-time digest check.
  • PostgreSQL URLs now match deployment reality. Percent-encoded credentials/database names, Unix-socket hosts, connection query options, and secure sslmode values are parsed and tested; unsupported ambiguous TLS modes fail with a configuration error instead of silently weakening a connection. Explicit child-spec options retain precedence over URL values.
  • Queue pause/resume is synchronous. The API now returns only after the producer has changed state, closing the race where a caller could pause a queue and immediately observe one more claim.
  • Zombie acknowledgements have a dedicated proof and dual-adapter regression. A stale attempt cannot commit after lease expiry and reclaim; the focused TLA+ model completely checks the fence, while its no-fence mutant produces the expected Claim(1) → Expire(1) → Claim(2) → Ack(1) counterexample.
  • Operator retry now clears a pending cancel and respects workflow dependencies. Found by extending the TLA+ model with a Retry action: (1) retry left cancel_requested set, so a cooperatively-cancelled job was un-retryable forever — ready, claimed, instantly re-cancelled, in a loop — and a retry racing a stale cancel was silently defeated; (2) retry sent workflow members straight to ready, so a cascade-cancelled dependent could run (and succeed) while its dependency sat failed. Retry now clears the flag (the operator's later intent wins) and re-holds workflow members through a settlement pass under the workflow lock — released if deps are satisfied, re-doomed if not. Both adapters, both confirmed by failing tests against the real engine before the fix.
  • Budgets are now enforced before every step execution, not only after. The 7-hour endurance soak (99,004 jobs, 4,978 worker kills, 13 Postgres restarts) caught 6 jobs paying for one step past their budget: the check ran only after journaling a new step, so a crash between journaling the over-budget step and acking the failure let the next attempt replay past the journal and execute one more paid step. The runner now pre-flights the budget against durable spend before running any step body (no extra queries — it reuses the row the cancel check already fetches). A deterministic regression test pins the crash window.
  • Cancel requests now survive crashes and retries. The adapter- equivalence property test (below) caught the Postgres ack path clearing cancel_requested on reclaim/retry while Memory preserved it — silently losing a user's cancellation if the worker died before honoring it. The flag now persists across all non-terminal transitions on both adapters; the wire contract states it normatively.

1.0.0-rc.4 — 2026-07-20

The "better at every dimension" release: the embedded dashboard, transactional enqueue, runtime CRUD, encryption, and exact partitioned claims.

Added

  • Belay.Dashboard — an embedded web dashboard with zero dependencies (hand-rolled HTTP over gen_tcp, single-file UI, SSE live updates): queue tiles with limits and live counts, filterable job list, a journal drawer (steps with costs, events, errors, children), a rendered workflow DAG, and retry/cancel/signal/steer actions. Tokenless dashboards are read-only; writes require a token or the same pluggable authorizer as the MCP server.
  • Belay.Txn — transactional enqueue inside your own Postgrex or Ecto transaction (duck-typed over query!; still no Ecto dependency). With the :postgres notifier, the wake-up is issued via pg_notify inside the transaction, so it delivers exactly on commit and never on rollback.
  • Runtime queue and cron CRUD — Belay.Queues.put/delete/list and Belay.Crons.put/delete/pause/resume/list, persisted in the database, validated eagerly, reconciled by every node's QueueSync (producers now live under a DynamicSupervisor); dynamic entries override static config by name. Leaderless, like everything else.
  • Encrypted inputs — use Belay.Worker, encrypted: true plus encryption: [key: {mod, fun, args}]: AES-256-GCM envelopes at rest, plaintext only inside the executing process; schemas validate before encryption; replay decrypts transparently.
  • Shared view serializers (lib/belay/view.ex) so the dashboard and MCP describe jobs identically.
  • bench/throughput.exs — measured ~416 trivial jobs/s end-to-end on a laptop (3 worker processes, unbatched acks).

Changed

  • Partitioned claims are now exact. Per-key allowances are computed with a window-function ranking inside the claim transaction (under the queue advisory lock), replacing the bounded over-fetch heuristic — heavy key skew can no longer starve minority keys (regression-tested with 30:1 skew).

1.0.0-rc.3 — 2026-07-20

Dispatch-latency release: agent workloads are bursts of short tasks, so insert→result overhead is the product. Measured on stock settings (bench/run.sh): ~9ms p50 same-node, ~11ms p50 / ~25ms p99 across unconnected OS processes with the new notifier, ~50ms p50 on adaptive polling alone (vs ~250ms average before).

Added

  • Notifier layer (Belay.Notifier): wake-ups as accelerators, never load-bearing. :local (registry + :pg, always on) and opt-in :postgres — pg_notify pokes and result notifications across fleets that share Postgres but not an Erlang cluster, with a dedicated auto-reconnecting listen connection per node and channel names scoped per database. If the channel is down, latency falls back to the polling floor; correctness never depends on NOTIFY.
  • Adaptive burst polling: producers poll at busy_poll (default 25ms) while claiming work and decay exponentially to poll_interval when idle; poke storms coalesce into single claim rounds.
  • Fast await_result: wakes on result notifications; otherwise re-checks on a 5ms→200ms backoff instead of a fixed 200ms.
  • MCP authorizer hook: mix belay.mcp --authorizer MyGuard (or authorizer: on Belay.MCP.serve/2) gates the mutating tools (retry/cancel/signal/steer) behind authorize(tool, args) — the mount point for capability-token systems (e.g. Legant) supervising operating agents.
  • bench/ — reproducible latency benchmark across the three topologies.

Fixed

  • Soak/bench harnesses now clean up respawned workers by pattern (stray workers were exhausting Postgres connections).

1.0.0-rc.2 — 2026-07-20

Hardening release driven by the chaos soak harness (soak/), which runs a mixed workload across multiple worker OS processes under kill -9 and full Postgres restarts, then verifies thirteen invariants. The passing report lives in soak/REPORT.md.

Fixed

  • Parent wake-up races in dynamic children. Under READ COMMITTED, the last two children acking concurrently could each see the other as incomplete and both skip the parent's $children signal, parking the parent forever. Fixed in layers: children now signal unconditionally on every terminal ack; signal delivery and awaiting-parking serialize on a per-scope advisory lock; and the sweeper re-readies any parent awaiting $children whose children are all terminal — so even an unknown residual ordering degrades to a sweep-interval delay, loudly logged, never a stuck job.
  • Database-outage resilience. Producers, the lease keeper, the sweeper, and the cron scheduler now rescue storage failures and skip the cycle with a warning instead of crash-looping the supervision tree; restart budgets are lenient. A full Postgres restart mid-load is survived with claim rounds skipped and no losses.
  • Spawn idempotency across the insert/journal gap. Children carry always-scoped unique keys derived from (parent, spawn name, index), and the id list is rebuilt from those keys — a crash between inserting children and recording the spawn step can no longer duplicate them.

Added

  • soak/ chaos harness (workers, driver with ledgered expectations, kill/restart orchestration, invariant verification, report generation).
  • Third uniqueness scope: unique: [key: k, scope: :always].
  • Sub-second worker timeouts: timeout: {n, :millisecond}.

1.0.0-rc.1 — 2026-07-20

First release candidate. Everything below ships open, Apache-2.0.

Engine

  • Claim/lease/ack execution with attempt fencing; expired leases reclaimed cluster-wide in seconds, stale acks rejected.
  • One-rule scheduling: a job is claimable when ready_at is due — scheduled work, retry backoff, and snoozes share it; there is no staging step and no leader election anywhere.
  • Poll-first dispatch with in-cluster pokes; no LISTEN/NOTIFY dependency.
  • Graceful shutdown: producers stop claiming first; running jobs get shutdown_grace; the cluster reclaims the rest.
  • Per-worker execution timeout:; queue pause/resume at runtime.
  • Constraint-backed unique jobs: unique: "key" (while incomplete) and unique: [key: k, within: seconds] (per window); duplicates return the existing job flagged duplicate?: true.
  • Retention pruning per terminal state, cascading steps and events; signal TTLs; rate-window cleanup.
  • Leaderless cron with per-slot dedup via a unique index.
  • Admission control per queue: global_limit (live-leased counting), sliding-window rate limits, per-key partition fairness.

The agent layer

  • Durable steps (Belay.step/4): memoized per job with cost columns — retries replay past completed work.
  • Budgets (budget: [usd:, tokens:]): jobs fail with :budget_exceeded the moment accumulated step costs cross the cap.
  • Signals (await/3, signal_job/4): park at zero cost, wake instantly, deadline timeouts; steering (steer/3 → steering/1) injects guidance into running jobs; cooperative cancellation at step boundaries.
  • Dynamic children: spawn/3 and spawn_many/3 (replay-safe, memoized), await_children/1, map_children/5 fan-out/fan-in — agents grow their own DAGs at runtime.
  • Workflows: DAG dependencies in the held state, released and cascaded transactionally inside the completing job's ack; ignore: policies.
  • Batches: transparent sugar over workflows with an any-outcome on_complete callback.
  • Durable event streams: emit/2 + live subscriptions + offset replay.
  • Replay debugging: Belay.Replay.dry_run/2 re-runs job code against the recorded journal, side-effect-free, with precise divergence reports.
  • Token-resource rate limits: shared resource buckets with estimated admission and post-hoc true-up via debit/3.
  • Input schemas: insert-time validation raising at the call site.

Operability

Storage

  • Behaviour with two adapters: deterministic in-memory (test/simulation reference) and Postgres (Postgrex, hand-written SQL, clock passed as a parameter). The same 64-test suite runs against both.