Operations

Run one Engine per whole store. Production :sync requires an explicitly validated local Linux filesystem with truthful file and directory sync, advisory locking, and exclusive rename semantics. validated_filesystem: true is the operator's attestation, not a filesystem probe or device certification. Never silently downgrade to :write; production builds reject that mode. macOS :write is for development and carries no power-loss promise. A release must be built for its target OS/architecture with Elixir 1.20, OTP 29, and a C11 compiler at build time; the deployed release runs its included native helper without Mix or a compiler.

Initialize and inspect

Set a dedicated absolute data_dir in the consuming application's runtime configuration, then initialize a genuinely new store once before supervising the Engine:

MIX_ENV=prod mix tay.storage.init --data-dir /srv/tay/jobs --durability sync --validated-filesystem
MIX_ENV=prod mix tay.storage.inspect --data-dir /srv/tay/jobs --durability sync --validated-filesystem

Initialization is not an idempotent “ensure” or repair command. Normal startup is existing-only; a missing lock, STORE marker, or required history is an error. Only an explicitly intended empty pre-existing root permits --bootstrap-existing. Never delete or replace a lock to bypass contention. Offline inspection acquires the existing lock and performs complete physical and Event-v1 semantic validation without activation. It returns aggregate diagnostics, not jobs or a reusable recovery ticket. Use mix help tay.storage.inspect for independent replay budgets.

Explicit initialization is the default operational model (initialize: :never). Embedding applications that need first-start creation, such as containerized deployments, may opt into initialize: :if_missing on Tay.child_spec/1. The native ownership layer initializes only when it creates the final storage root; every pre-existing root remains existing-only and proceeds through normal full recovery. This option is initialization only, never repair or a fallback after recovery failure.

Local executor access

An Engine normally creates its Unix socket automatically outside data_dir. The final automatic directory is private (0700), and the socket defaults to 0600. An explicit path or TAY_SOCKET_PATH overrides discovery; prepare the parent ownership and permissions yourself. Tay refuses a regular file at the socket path and removes only a verified stale socket. Grant access only to trusted local processes: a peer can submit, inspect, cancel, and execute tasks. Bound connection, frame, task, result, and error limits for the workload. See the protocol contract for the wire format and result lifetime.

Queue and lifecycle controls

Use name: when an Engine has a nondefault name. Pause is volatile and stops new claims, not already-started effects. Drain closes new insertion/claim admission and succeeds after active tasks and their outcomes settle; queued jobs may remain. A timeout leaves the Engine draining. Stop normally drains; forced stop is an explicit interruption, not successful drain. Restart creates a fresh ownership, recovery, projection, and execution generation:

:ok = Tay.pause_queue(:default)
:ok = Tay.resume_queue(:default)
:ok = Tay.drain(timeout: 30_000)
:ok = Tay.stop(timeout: 30_000)
:ok = Tay.restart(timeout: 30_000)

An unknown outcome after a submitted insert or control call does not prove no change. Reconcile by stable job ID, original definition, status, or a new explicit control decision; do not automatically allocate another job ID. Worker effects are at-least-once. Cancellation fences the durable execution token before best-effort termination but cannot undo an effect already made.

Live inspection and dashboard

Tay.jobs/1 returns public Tay.Job views in deterministic newest-first insertion order. It accepts state, queue, exact worker-key (worker/workers), partial worker-key (worker_contains), ID, limit, and opaque cursor filters. Pages are capped at 100 results. Each response includes the exact matching total_count plus opaque previous_cursor, next_cursor, and last_cursor values for bidirectional and final-page navigation. The first page uses no cursor. Cursors are tied to their filters. Concurrent inserts and state changes produce a weakly consistent live view, so refresh after administrative actions or engine restart.

Tay.stats/1 and Tay.queues/1 use reconstructable runtime counters rather than scanning retained history. The dashboard included in the tay Hex package mounts these APIs in a host Phoenix application. It is not an authentication boundary; protect its router scope with the host application's administrator pipeline.

Store-v2 automatic compaction

The per-Engine policy is enabled when compaction: is omitted. It uses finite terminal retention and conservative sealed-history estimates. Optional periodic semantic checkpoints use the same crash-safe Store-v2 epoch publication; they never serialize private ETS implementation details. Store-v2 candidate construction runs online; only the final fenced publication switch closes admission internally. Calls arriving during that short switch wait for the new generation within their configured caller timeout instead of receiving an immediate unavailable. Initial Store-v1 adoption still drains and recovers because that format has no safe online catch-up boundary. Reclaim-policy thresholds are conjunctive. Retention compaction is deferred while active jobs exist. A due periodic checkpoint may run with active jobs and additionally requires at least one new durable tail event. Cooldown uses the later of the verified manifest capture time and post-publication recovered activation time. Restart conservatively extends cooldown rather than shortening it by construction time. Backward clocks conservatively defer.

Configuration (milliseconds except the explicit retention unit):

compaction: [
  enabled: true,
  terminal_retention: {:hours, 24},
  check_interval: 60_000,
  checkpoint_interval: 0,
  min_interval: 3_600_000,
  min_sealed_segments: 1,
  min_reclaimable_bytes: 16_777_216,
  dead_ratio_threshold: 0.25
]

checkpoint_interval: 0 disables periodic checkpoints. A positive millisecond value checkpoints only after at least one new durable tail event. The evaluator runs at the smaller of check_interval and checkpoint_interval. Automatic checkpoints capture the Engine's active ETS projection and disposable terminal projection at an exact durable sequence frontier; they do not reconstruct that state by replaying the old journal. Concurrent mutations that arrive after the capture are appended to the candidate tail. The temporary ETS snapshot is released as soon as the publisher has consumed it. The new CURRENT is published only after sync and independent verification, and restart replays the canonical snapshot followed by that tail. The durable sequence frontier—not a wall-clock timestamp—decides which mutations are already covered. The first checkpoint of a Store-v1 root performs the existing safe migration path, which briefly drains execution because Store v1 has no online catch-up boundary; subsequent Store-v2 checkpoints prepare online.

Unknown/duplicate keys, unbounded/nonpositive timers or byte/count limits and invalid ratios fail configuration. Timers/counts fit unsigned 32 bits; bytes fit signed durable time/integer range. Bounded hours use the shared manifest rule 1..2_562_047_788_015, not a separate configuration duration syntax. A manual zero-retention request is a one-shot full terminal-history purge: it preserves every non-terminal job and persists the minimum future retention of one hour. Ratios are positive finite numbers at most one. Automatic configuration rejects infinity.

Terminal jobs are removed from the hot ETS job projection as soon as they finish and remain queryable through a disposable disk-backed inspection index. Active-job and terminal-history counts are informational and never reject admission or trigger maintenance. max_state_bytes protects active payload data with one conservative in-memory byte budget calculated from the job's flat BEAM heap shape, runtime machine-word size, retained canonical encoding, and binary payload bytes. The former max_state_nodes option is accepted when valid but ignored for 1.x configuration compatibility. Aggregate terminal state and queue counters remain in memory. The append-only Store remains authoritative for both projections and rebuilds them on restart. When the last active job becomes terminal, Tay schedules an idle full collection for the Engine and Writer. This releases transient heaps and binary references grown by a burst; allocator/VM baseline memory is not forcibly returned below what the Erlang runtime still needs.

For release-candidate compatibility, the former max_jobs and max_terminal_jobs options are accepted when valid but ignored.

Time retention is eligibility, not an exact-time deletion guarantee: terminal jobs expire at terminal_at <= captured_at - duration, but actual work waits for sealed-history thresholds, cooldown, healthy ownership, successful execution drain and sufficient additional disk. The estimator retains at most 128 oldest hour buckets, conservatively counting only fully expired hours (up to one extra hour of estimation delay). Small/idle stores may retain terminals indefinitely below thresholds. Live, scheduled, available, executing, retryable and unsettled jobs are protected. Expired IDs are not remembered: get/retry/cancel return not-found, and a later submission with that ID is a new job.

A successful compaction makes its terminal_retention the durable runtime setting stored with Store-v2. On subsequent recovery that value takes priority over the startup configuration or standalone environment. Running compaction from the Dashboard therefore both applies and persists the selected retention.

compaction: false disables the policy child/timer; terminal history may then grow indefinitely and the operator assumes storage management. Manual work remains available:

{:ok, stats} = Tay.compact(timeout: 900_000)
{:ok, stats} = Tay.compact(terminal_retention: {:hours, 0})
{:ok, stats} = Tay.compact(terminal_retention: {:hours, 48})
{:ok, stats} = Tay.compact(terminal_retention: :infinity)

Manual/automatic/stop/restart share one fixed operation slot; a concurrent administrative operation returns capacity/busy rather than queuing or forcing a switch. operation_slot means one of those operations is already running; it is not a RAM or terminal-history capacity failure. Supervisor shutdown cancels the evaluator's timer and asks the owner to cancel construction before CURRENT. Once CURRENT is in flight, it waits for verification/reconciliation before retiring Engine and defers reclamation. No source bytes are deleted to obtain candidate headroom. Payload-free aggregate events are structured Logger debug metadata under :tay_compaction; enable debug logging to observe evaluation/gate/defer/start/completion/failure events.

On Store v2, candidate construction runs in the background. The public status is state: :compacting, phase: :preparing; admission, dispatch, and execution remain live, and committed mutations are journaled for candidate catch-up. The final fenced publication reports phase: :switching and closes admission while the old tail is synchronized, catch-up is appended, the candidate is revalidated, and CURRENT is atomically replaced. The expensive candidate construction remains online, but switch duration still depends on retained history and filesystem validation throughput. The Engine process is retained. The first Store-v1 to Store-v2 adoption instead reports state: :migrating and uses the existing drain/recovery path because Store v1 has no safe online catch-up boundary. The qualification suite covers finite workload measurements and the documented operational limits.

Cold backup and restore

Stop and fence every Engine using the source before copying. The native tools hold its existing exclusive lock throughout inventory, hashing, and copy; they refuse a live owner. A cold copy is whole-store bytes, not a live snapshot, semantic validation, repair, or a proof that the backup contains the latest acknowledged history. Keep the SHA-256 catalog outside both stores. The source, destination parent, and catalog parent must exist; destination and output catalog must not. Do not run source and restored copies concurrently: they share a STORE_ID.

MIX_ENV=prod mix tay.storage.backup \
  --source /srv/tay/jobs \
  --destination /srv/backups/tay-2026-09-13 \
  --catalog /srv/backups/tay-2026-09-13.json \
  --durability sync --validated-filesystem

MIX_ENV=prod mix tay.storage.restore \
  --source /srv/backups/tay-2026-09-13 \
  --destination /srv/tay/restored \
  --verify-catalog /srv/backups/tay-2026-09-13.json \
  --catalog /srv/backups/tay-restored.json \
  --durability sync --validated-filesystem

MIX_ENV=prod mix tay.storage.inspect \
  --data-dir /srv/tay/restored --durability sync --validated-filesystem

For development copies use --durability development on backup/restore and --durability write on inspection. Strict copying is Linux-only on a supported validated persistent local filesystem. The tools reject symlinks, hard-linked source files, unsafe aliases, unexpected file types, changed ancestors/source identities, missing/extra archive files, malformed catalogs, checksum mismatch, and exhausted file/byte budgets. Defaults are 100,000 inventory entries (including segments) and 10 GiB of file bytes; override only for a separately qualified workload.

The destination is a new private 0700 sibling staging directory with 0600 files. STORE is copied last. In strict mode every copied file is fsynced, followed by the segments and store directories, exclusive no-replace publication, ancestor syncs, then exclusive external catalog creation and its syncs. A failure preserves the source and any partial staging or published destination. A complete-looking destination without a successful catalog is not acknowledged backup success; never overwrite it or infer a recoverable prefix. Restore always verifies the selected backup's catalog before creating destination data. An older valid backup explicitly accepts lost later jobs and possible repeated external effects. Inspect and activate it in a fresh Engine generation before admitting work.

Incident boundary

On corruption, incomplete history, unsupported semantics, or an uncertain storage operation, stop admission and preserve the entire store and namespace. Do not truncate, pad, delete, rename-aside, skip records, or reuse sequence space. Distinguish ownership contention and operational budget refusal from bad bytes. Increase a budget only for a new full inspection of unchanged data. There is no automatic repair, retention/compaction, or live backup. Protect plaintext args, stores, catalogs, and backups with deployment access controls.