Production Operations

This guide describes the host responsibilities for running Spectre in a real product. Spectre provides deterministic runtime transitions; it does not replace application-level authorization, persistence, queues, or observability.

Release status

0.2.0 is the first vNext core release. Runtime invariants are tested and the suite exceeds 90% line coverage, but public APIs may still change in a minor 0.x release. Pin a compatible minor version and read CHANGELOG.md before upgrading. The exact compatibility boundary is listed in PUBLIC_API.md.

Before deploying an upgrade, run Spectre.Foundation.Conformance against representative durable backups and verify every compiled Definition/Manifest pair. Umbrella applications and satellite suites should also run Spectre.Stack.Conformance against the complete installed package set. These gates complement, rather than replace, restart and side-effect idempotency tests. See Foundation Conformance.

State persistence

Use a durable state adapter for multi-process, multi-node, or restart-safe conversations. Prefer compare-and-set persistence with the current revision:

defmodule MyApp.AgentState do
  @behaviour Spectre.State.Store

  @impl true
  def load(input, opts) do
    conversation_id = Keyword.fetch!(opts, :conversation_id)
    MyApp.AgentStates.fetch(conversation_id, input)
  end

  @impl true
  def persist(state, expected_revision, _input, _agent, _opts) do
    MyApp.AgentStates.compare_and_set(
      state.conversation_id,
      expected_revision,
      Spectre.State.Codec.encode!(state)
    )
  end
end

Treat {:error, :stale_state} as a concurrency signal. Treat {:error, {:persistence_ambiguous, reason, result}} as an uncertain commit and reconcile it from durable state before retrying external work. Spectre produces that ambiguous result not only when an adapter explicitly returns {:error, {:ambiguous, reason}}, but also when an invoked state callback raises, exits, throws, is killed, times out, or returns a malformed result. Any of those failures can happen after the store committed.

With a strict synchronous persistence journal, compare-and-set can succeed before the audit append fails. That result is returned as {:error, {:persistence_journal_failed, reason, committed_result}}. A Session retains committed_result; request-scoped callers must do the same or reload the durable state. Do not retry the old revision as though no write occurred.

Canonical Instance checkpoints

Spectre.State.Store persists conversational state. Work, Vigil, external controller, control, event, consent, and receipt sections belong to the Instance's complete canonical checkpoint and use the separate Spectre.Instance.CheckpointStore compare-and-swap boundary.

Configure a durable checkpoint store for any operational loop that must survive process or node loss. If a write can have committed before an adapter timeout or crash, return {:error, {:ambiguous, reason}}. Spectre fences further automatic writes until Spectre.reconcile_checkpoint/2 loads and validates durable state. Monitor checkpoint_status/1, and use flush_checkpoint/2 as a deployment or graceful-shutdown barrier.

Checkpoint schema 4 includes the current Definition Activation, retained Run continuations, Definition lifecycle and event records, and private Skill-state branches. Configure the same durable Spectre.Definition.Store on restart so Spectre can re-read every pinned Definition and verify its closure. The bundled in-memory Definition Store is not valid beside a durable Checkpoint Store.

Before upgrading an existing checkpoint namespace to 0.2.3, implement migrate_instance_key/5 atomically and drain old owners. A local Registry is not a distributed ownership guarantee; multi-node deployments must also configure a linearizable Spectre.Instance.Owner lease adapter. See Migrating to 0.2.3.

Before upgrading live checkpoint namespaces to 0.2.5, quiesce schema-3 writers. Once schema 4 has been written, 0.2.4 cannot restore that checkpoint. Inventory host-side references before marking any Skill-state branch GC-eligible: core sees its Activation, Runs, operations, and child branches, but not application tables or backups. See Migrating to 0.2.5.

Canonical operational history is bounded by default. An Instance retains the 256 most recently updated terminal loops and up to 1,024 additional historical correlations, while always preserving live loops and their primary correlations. Tune the limits when starting the Instance:

operation_terminal_loop_retention: 256,
operation_correlation_retention: 1_024

Both options accept a non-negative integer or :unlimited. Use :unlimited only with an external compaction policy: every checkpoint serializes the full canonical value. The Subject Registry independently defaults to 4,096 live link intents and 1,024 terminal audit intents (link_intent_capacity and link_intent_retention), for a bounded maximum of 5,120 retained intents.

Nested custom structs in conversational state are restored from their module name and fields. That is a storage representation, not a struct migration protocol: keep the module loadable and backward compatible, or persist an application-owned versioned map and migrate it before constructing the struct.

Action idempotency

Every effect has an id and idempotency_key. Store the key at the real business boundary and make duplicate execution return the already committed outcome:

def perform(args, ctx) do
  key = Keyword.fetch!(ctx.opts, :idempotency_key)

  MyApp.Idempotency.once(key, fn ->
    MyApp.Payments.capture(args)
  end)
end

Do not retry an ambiguous effect merely because the terminal Spectre state was not observed. First inspect the application idempotency record.

Sessions and supervision

Start Spectre.Supervisor in the application tree when using live sessions:

children = [
  {Spectre.Supervisor, name: MyApp.SpectreSupervisor}
]

A session serializes calls for one conversation and protects against stale execution results. It is not durable storage. Configure an explicit state adapter when state must survive a process or node failure.

Use finite idle or shutdown values for high-cardinality conversation workloads. Supervisors may recreate a session from durable state on demand. Sessions use transient restart semantics: abnormal exits restart under a supervisor, while normal dismiss and idle shutdown stay stopped. A failed initial restore must not leave a registered process behind.

Provider deadlines

Every external callback should have a finite deadline. Spectre supplies conservative defaults and accepts provider-specific overrides:

config :spectre, :provider,
  llm_timeout: 30_000,
  classifier_timeout: 5_000,
  embedding_timeout: 10_000,
  semantic_cache_timeout: 5_000,
  action_timeout: 60_000,
  state_timeout: 5_000

The local worker is terminated after a timeout. Cancellation of work already sent to a remote service depends on the host adapter and its client library.

Journal and telemetry

Journaling is disabled unless configured. A safe production default is asynchronous delivery without conversation content:

journal MyApp.SpectreJournal,
  events: :all,
  mode: :async,
  on_error: :warn,
  include_input: false,
  include_reply: false,
  buffer_size: 1_000,
  overflow: :drop_newest,
  retention: %{class: :operational, days: 30}

Add a redactor even when content recording is disabled if metadata can contain tenant identifiers or regulated data. Journal writes, telemetry handlers, and monitor fallback callbacks must not perform unbounded blocking work.

Spectre.Telemetry emits privacy-safe events under the [:spectre, ...] prefix. A telemetry_handler: callback works without adding :telemetry as a runtime dependency; if the standard library is installed, events are sent to it as well.

Semantic cache

The built-in cache is owned by a supervised process and survives individual request-process exits, but its online rows are ETS data. Snapshot verified rows or use a custom durable adapter when learned examples must survive application restarts.

Online rows are capped at 1,000 per Agent by default and the least recently updated rows are evicted. Set semantic_cache_online_capacity per call or :online_capacity under config :spectre, :semantic_cache; :unlimited is available only for hosts that enforce their own retention. Rule-example embeddings use a separate 2,048-entry bounded cache and can be tuned with embedding_example_cache_capacity.

Only cache routes that are safe to replay as classifications. Keep destructive or highly contextual routes at cache: false, and require review before using unverified online examples in sensitive flows.

Deployment checklist

• Pin Spectre and read the changelog.
• Use durable state with optimistic revisions where concurrency is possible.
• Persist canonical Instance checkpoints before relying on Work or Vigil recovery, and reconcile every ambiguous compare-and-swap result.
• Quiesce old checkpoint writers during schema upgrades, and retain private Skill-state branches until both core and host-side references are retired.
• Make business actions idempotent by effect key.
• Give every operational side effect an accurate :idempotent, :reconcilable, or :non_idempotent declaration.
• Configure finite provider deadlines.
• Keep authorization in the action/provider boundary.
• Keep prompt, journal, and telemetry content privacy-safe.
• Configure session idle shutdown for unbounded conversation IDs.
• Snapshot or externalize learned semantic-cache rows and their embeddings; runtime snapshot loading must not regenerate stored vectors.
• Run route evaluation against version-controlled cases before deployment.
• Exercise policy rejection, timeout, stale state, and ambiguous persistence in application integration tests.
• Inject failures immediately before and after each host-owned durable commit, then assert callback cardinality, durable state, restart behavior, and idempotent replay.