Journal

Copy Markdown View Source

Spectre's journal is an opt-in stream of structured explanation records. It is separate from chat history and logs:

  • chat history is context that may influence a later turn;
  • journal records explain an individual runtime decision;
  • logs and telemetry describe operational health.

The current implementation records Run, routing/arbitration, policy, lifecycle, execution, and persistence boundaries. All phases carry stable correlation identities and exclude conversation content, effect arguments, action results, and provider errors by default.

For a Stack-bound Agent, every record also carries authoritative deployment identity: Stack id, owner and digest plus each installation's id, package version, and digest. This lets an audit correlate behavior with the exact installed implementation without copying package configuration, secrets, clients, PIDs, connections, or other runtime handles.

Configure A Store

Declare a store on an agent:

defmodule MyApp.SupportAgent do
  use Spectre.Agent

  journal MyApp.SpectreJournal,
    events: [:run, :routing, :policy, :lifecycle, :execution, :persistence],
    mode: :async,
    on_error: :warn,
    sample_rate: 1.0,
    include_input: false,
    buffer_size: 1_000,
    overflow: :drop_newest,
    store_opts: [index: "agent-decisions"]
end

:arbitration is accepted as an alias for :routing. Use events: [:all] to record every supported phase.

The store implements one append callback:

defmodule MyApp.SpectreJournal do
  @behaviour Spectre.Journal.Store

  @impl true
  def append(%Spectre.Journal.Record{} = record, opts) do
    MyApp.AgentJournal.insert(record, opts)
  end
end

Valid success results are :ok and {:ok, value}. Return {:error, reason} when the append fails.

An application-wide default can be configured without changing every agent:

config :spectre,
  journal:
    {MyApp.SpectreJournal,
     [events: [:routing], mode: :async, on_error: :warn]}

Use journal(false) on an agent to disable that application default. A host can also pass journal: {Store, opts} to one Spectre.ask/3 or Spectre.turn/3 call; per-call options win over agent configuration.

Privacy Defaults

Routing records contain labels, providers, scores, margins, configured thresholds, the selected route, and structured reason codes. They do not contain:

  • user input text or raw input payloads;
  • model output or visible reply text;
  • candidate match text and examples;
  • state data, effect arguments, or action results.

Set include_input: true only when the store's access, redaction, and retention policy can safely handle conversation content. Reply recording is reserved but requires include_reply: true.

Use redact: for a final application-controlled record transformation and retention: for metadata the store can enforce:

journal MyApp.AuditJournal,
  events: [:all],
  redact: {MyApp.JournalRedactor, :redact},
  retention: [days: 30]

The redactor receives a %Spectre.Journal.Record{} and returns a record, {:ok, record}, or {:error, reason}. Policy and execution phases default to unsampled delivery; sample_rates: %{policy: 0.5} can override that explicitly.

Delivery And Failures

The monitoring default is mode: :async, on_error: :warn. A supervised, bounded queue serializes appends and keeps journal latency outside the turn. buffer_size defaults to 1_000 including the active write.

Overflow policies are:

  • :drop_newest rejects the incoming record and logs a warning;
  • :drop_oldest replaces the oldest queued record and logs a warning.

Delivery is ordered through one worker. A store crash is isolated to that append and the buffer continues with the next queued record.

Set on_error: :ignore when the host intentionally wants silent best-effort delivery. Async mode cannot use on_error: :error, because an append finishes after the turn has continued.

For a required audit artifact, use:

journal MyApp.AuditJournal,
  events: [:routing],
  mode: :sync,
  on_error: :error

Before state persistence, a failed append returns {:error, {:journal_append_failed, reason}} from the turn.

The persistence record is necessarily appended after compare-and-set. If that strict append fails, the state is already committed and Spectre returns {:error, {:persistence_journal_failed, {:journal_append_failed, reason}, committed_result}}. A Session retains the committed result so its in-memory revision does not move backward. Request-scoped callers should retain that result or reload durable state before continuing.

If state and audit records must commit atomically, use an application outbox in the same storage transaction rather than relying on a remote append. A remote strict journal can report the post-commit failure, but it cannot roll the state write back.

Correlation And Idempotency

Every runtime turn receives a UUIDv7 turn_id and uses it as the default trace_id. A record ID is deterministically derived from the agent, turn, phase, and sequence. Pass a durable turn_id: when retrying the same logical turn so the store can enforce uniqueness on record.id:

Spectre.turn(MyApp.SupportAgent, input,
  conversation_id: conversation.id,
  turn_id: inbound_message.id
)

Without a host-provided ID, each call is a new logical turn and receives a new identifier.

Run lifecycle records keep the Run's shared trace_id, while their turn_id is derived deterministically from the Run id, revision, and event. This gives each lifecycle fact its own retry-stable journal identity. A successful Runtime.resume/3 records both :run_resumed and the resulting :run_awaiting, :run_boundary, or :run_completed event; their sequences preserve that order.

Sampling

sample_rate accepts a value from 0.0 through 1.0. Sampling is deterministic for a record ID, so a retry with the same turn_id makes the same sampling decision. Routing records can be sampled; future policy and execution records will default to unsampled because they are audit-sensitive.

Record Shape

The current :arbitration record includes:

%Spectre.Journal.Record{
  schema_version: 1,
  id: "journal:...",
  agent: MyApp.SupportAgent,
  conversation_id: "chat-123",
  turn_id: "...",
  trace_id: "...",
  sequence: 1,
  phase: :arbitration,
  decision: %{
    kind: :route_selected,
    label: :BILLING,
    strategy: :local_classifier,
    accepted?: true
  },
  reason: %{
    code: :candidate_selected,
    provider: :local_classifier,
    label: :BILLING
  },
  evidence: [
    %{
      provider: :local_classifier,
      label: :BILLING,
      score: 0.97,
      margin: 0.21,
      accepted?: true
    }
  ],
  input: nil,
  reply: nil
}

Stores should persist schema_version and tolerate newly added optional fields. They should use record.id as an idempotency key rather than generating a new identity during append.

Persisted schema-v1 maps can be checked with Spectre.Journal.Record.restore/1. Unknown additive fields are ignored and an unsupported schema version returns an explicit error.

For a per-agent timeline, query by agent, conversation_id, and turn_id, then order by sequence, occurred_at. For aggregate dashboards, group by phase, decision.kind, reason.code, and the arbitration strategy; content fields are unnecessary for those queries.