# Operations Generate the PostgreSQL installation through Igniter or the deterministic Mix task: ```sh mix igniter.install ash_onetime --repo MyApp.Repo mix ash_onetime.gen.migrations --repo MyApp.Repo mix ecto.migrate ``` The migration creates authoritative idempotency claims, nonce claims, response payload partitions, collision constraints, cleanup functions, and deletion guards. Claim parents may be unpartitioned or hash-partitioned by `operation_hash`; response payloads use date partitions plus a default partition. All identifiers are quoted and every operation keeps the operation hash in its logical key. Run bounded cleanup manually or with the optional Oban worker: ```sh mix ash_onetime.prune --repo MyApp.Repo --batch-size 500 ``` Cleanup deletes only claims strictly past their replay/retention horizon, keeps processing external recovery points, removes response bytes atomically with completed claims, and drops only empty expired payload partitions after obtaining the required lock. Batch size and prefix are explicit. Schedule `AshOnetime.Oban.CleanupWorker` only if Oban is installed. Abandoned external `processing` claims — committed recovery points whose peer effect never settles and that no retry ever recovers — are immortal under cleanup (it skips them) and the delete guard (it forbids deleting them). Left unbounded this is a storage denial of service. `AshOnetime.Store.reap(target, batch_size, abandonment_seconds)` is an opt-in, bounded reaper that deletes such claims past a separate, much longer abandonment horizon. A claim is reaped only when it is older than both `abandonment_seconds` and a hard 1-day floor and past its own retention horizon (all re-enforced by the delete guard), so an in-flight or in-retention recovery point is never removed. `abandonment_seconds` must be at least the 1-day floor. Schedule it far less frequently than cleanup (recoverability becomes bounded by `max(retention, abandonment)`), and size the horizon well beyond any legitimate in-flight window. Run the reaper manually or with the optional Oban worker, mirroring cleanup: ```sh mix ash_onetime.reap --repo MyApp.Repo --abandonment-seconds 1209600 ``` `--abandonment-seconds` defaults to 604800 (7 days) and must be at least the 86400-second (1 day) floor; `--batch-size` defaults to 500. Schedule `AshOnetime.Oban.ReapWorker` only if Oban is installed, on its own far slower cadence than `AshOnetime.Oban.CleanupWorker`. Before reaping, observe the backlog with `AshOnetime.Store.processing_backlog(target)`, which returns `{:ok, %{processing_count: n, oldest_age_seconds: seconds}}` (`oldest_age_seconds` is `nil` when no recovery points are in flight) — watch it grow to detect abandonment accumulation and to size the abandonment horizon above the oldest legitimate in-flight claim. Existing deployments installed before the reaper existed must, in a manual migration, `CREATE OR REPLACE` the `ash_onetime_guard_idempotency_delete` function with the new body (the attached trigger keeps pointing at the same name), `CREATE` the `ash_onetime_reap_idempotency` function, and `CREATE` the `ash_onetime_idempotency_claims_processing_index`. Copy those three definitions from a freshly generated install migration (`mix ash_onetime.gen.migrations`), changing the guard's `CREATE FUNCTION` to `CREATE OR REPLACE FUNCTION`. Until they are applied a reap attempt fails closed against the old guard. ### Forward response partitions The `ash_onetime_response_payloads` table is range-partitioned by month. The install migration generates a fixed window (install month through +12). Cleanup drops only past, empty named partitions; the `_default` partition is excluded from drop enumeration. Once retention exceeds the generated window, payloads route to `_default` and are never dropped — silently defeating bounded retention. Forward monthly partition creation keeps the window ahead of retention: ```sh mix ash_onetime.roll_partitions --repo MyApp.Repo --months 6 ``` `--months` (default 3, 1..24) sets how many forward months to ensure exist. The roll is idempotent and concurrency-safe (advisory-locked, bounded `lock_timeout`), so concurrent or overlapping runs are safe. Schedule `AshOnetime.Oban.PartitionWorker` (or a cron cadence) ahead of your retention horizon — missing it causes new writes to route to `_default`, so the roll is more time-sensitive than cleanup/reap. Existing installs that predate the roll or have crossed the window should generate and run the forward migration once: ```sh mix ash_onetime.gen.roll_forward --repo MyApp.Repo --months 18 mix ecto.migrate ``` It adds the `response_partition` index, back-fills the elapsed+forward partitions, and drains past-retention payloads stranded in `_default` (a claim-scoped delete; the delete guard removes the payload). Within-retention stranded payloads are left — a retry under a new key is a new execution, and the rolled-forward partitions now route future inserts correctly. **Gap recovery:** `roll_partitions` only creates partitions ahead of the current month — it does not back-fill. If the worker/cron is down for a month or longer, that month's within-retention payloads strand in `_default` (which cleanup never drops). Re-running `mix ash_onetime.gen.roll_forward` is the recovery procedure: its drain is idempotent and removes any past-retention payloads that aged out while the roll was down. An exhausted `PartitionWorker` (`max_attempts: 3` discards persistent failures) leaves the same gap — monitor for discarded jobs and re-run the forward migration when the worker has been down across a month boundary. A context-multitenant tenant prefix must be 1..63 bytes — PostgreSQL truncates identifiers at 63 bytes (NAMEDATALEN), so a longer prefix could route two tenants to the same schema. Both admission and cleanup reject an out-of-range prefix (admission fails closed with `:missing_prefix`) rather than truncating. Configure an optional completed-response cache with `config :ash_onetime, :cache, MyCache` and a bounded `:cache_timeout`. Implement `AshOnetime.Cache`; treat every value as untrusted. The default `AshOnetime.Cache.None` needs no application configuration or supervision tree. `AshOnetime.Plug` can copy configured headers into `conn.private.ash_onetime.untrusted`; it does not verify them. Trusted verification remains inside the protected action. Telemetry events are `[:ash_onetime, event]`, where event is `:admission`, `:conflict`, `:replay`, `:fingerprint_mismatch`, `:verification`, `:encoding`, `:cache`, `:cleanup`, `:reap`, `:external_recovery`, `:store_uncertainty`, or `:untracked_execution`. The `:reap` event carries a `:claims_reaped` count for each bounded reaper run. Metadata is exactly `strategy`, `resource`, `action`, and `result_class`; measurements are only `duration` or `count`. Raw keys, scopes, tokens, fingerprints, payloads, signatures, resolver identities, exceptions, and store results are never telemetry fields. ### DPoP replay fence operational characteristics A `:one_time_nonce` protection with `commit: :independent` (the DPoP §11.1 replay fence, ADR-0003) routes its claim through the `claim_committed` worker, which spawns a process that opens its own transaction on its own connection. Two operational characteristics apply, inherited from the external-effect path but now exercised per-request rather than per-external-effect: - **Connection-pool pressure.** The worker needs a checkout from the same pool while the caller's `before_action` still holds one — effectively +1 connection per in-flight protected request. Under `pool_size N` concurrent protected actions, the `(N+1)`th blocks waiting for a checkout; under sustained burst this surfaces as latency and, at the limit, a `:checkout_unavailable` rejection (which fails closed — the request is rejected, never admitted unsafely). Size the pool for the expected concurrency of DPoP-protected endpoints. - **30s worker timeout.** If the worker stalls (slow query, lock contention on the unique index, pool pressure), `before_action` blocks for up to 30s before the worker times out and returns `:dispatched_unknown`, which surfaces as a `:store_uncertainty` telemetry event and a typed store error. Both timeout and a genuine disconnect fail closed; an operator cannot distinguish them from telemetry alone, but neither can cause a silent re-admit. The fence's fail-closed posture means neither characteristic can reopen the replay gap the fence exists to close — the worst case is rejection under stress, never a double-spend. ## Key rotation Token verification resolves material by the signed key identifier. Add a new key before using it to sign. Retain every old verification key until the last token it signed is outside `max_age + clock_skew`, then remove it. An unknown key identifier fails verification; never fall back silently to a different key. The supported runtime is Elixir `~> 1.20` (verified on 1.20.2) and Erlang/OTP 29 with Ash `>= 3.29.3`, AshPostgres 2, and PostgreSQL 18. Release checks include the full suite, mutation matrix, warnings-as-errors documentation, exact Hex archive inspection, and an unpacked zero-configuration consumer, all run on the runtime pinned in `.tool-versions`; see [CONTRIBUTING](../CONTRIBUTING.md).