Routing Evaluation

Copy Markdown View Source

Spectre's evaluation tools measure two different properties together:

  • whether the router selected an expected route or outcome;
  • whether it used the LLM classifier only when the case allowed or required it.

This catches a regression that route-only accuracy misses: a route can be correct while still making an unnecessary model call.

Run A Corpus

Store one JSON object per line:

{"id":"deterministic-help","input":"help","expected_route":"HELP","expected_strategy":"regex","llm":"forbidden","tags":["deterministic","english"]}
{"id":"local-help","input":"show me the available commands","expected_route":"HELP","llm":"forbidden","tags":["local","english"]}
{"id":"ambiguous-billing","input":"something is wrong with the invoice for my project","allowed_routes":["BILLING","PROJECT_SUPPORT"],"llm":"required","tags":["ambiguous","english"]}

Then run:

mix spectre.eval MyApp.Agent test/fixtures/routing.jsonl

The equivalent --agent form is:

mix spectre.eval --agent MyApp.Agent test/fixtures/routing.jsonl

The summary includes case pass rate, route accuracy, LLM policy violations, outcomes, strategy usage, and p50/p95 routing duration.

Case Fields

  • id is a required stable case identifier.
  • input is a required string or Spectre input-shaped object.
  • expected_outcome is route, clarify, unknown, or error; it defaults to route.
  • expected_route is one accepted label.
  • allowed_routes adds other accepted labels for genuinely ambiguous cases.
  • expected_strategy optionally fixes the expected winning provider.
  • llm is forbidden, allowed, or required; it defaults to allowed.
  • state supplies an explicit state snapshot without loading the configured state adapter. Existing flow names may be written as strings.
  • tags groups cases for report analysis.
  • max_duration_us optionally sets a per-case duration ceiling. Avoid tight ceilings for network-backed providers.

Blank lines and lines beginning with # are ignored. Invalid JSON or invalid fields report the exact source line.

docs/examples/routing-eval.jsonl is a small starter corpus. Its labels and expectations are illustrative and should be adapted to the agent under evaluation.

CI Thresholds And JSON Artifacts

By default, every case must pass and no forbidden or required LLM policy may be violated:

mix spectre.eval MyApp.Agent test/fixtures/routing.jsonl \
  --json tmp/spectre-routing.json

Thresholds can be relaxed explicitly while a corpus is being calibrated:

mix spectre.eval MyApp.Agent test/fixtures/routing.jsonl \
  --min-pass-rate 0.98 \
  --max-unnecessary-llm 1 \
  --max-missing-llm 0 \
  --max-errors 0

The task exits unsuccessfully when thresholds are not met, so it can be used as a CI regression gate. The JSON artifact contains privacy-safe receipts and structured violations for every case.

Programmatic API

Evaluate a complete corpus:

{:ok, report} =
  Spectre.Eval.run(
    MyApp.Agent,
    "test/fixtures/routing.jsonl",
    router_opts: [tenant_id: "evaluation"]
  )

if Spectre.Eval.Report.acceptable?(report) do
  :ok
else
  {:error, report}
end

Evaluate one route:

{:ok, receipt} =
  Spectre.Router.evaluate(
    MyApp.Agent,
    "show me my invoices",
    state: %Spectre.State{current_flow: :billing}
  )

receipt.label
receipt.strategy
receipt.llm_called?
receipt.attempts

Spectre.Router.Receipt excludes input text, prompts, model output, candidate matches, and route handlers. It contains the outcome, winning label and strategy, sanitized provider attempts and candidates, total duration, and a provider_calls list with the normalized outcome and duration of every shared provider-boundary invocation. llm_called? becomes true only when an LLM adapter worker was actually invoked; selecting LLM arbitration before prompt construction is not counted as model use.

A provider-call entry has only operational metadata:

%{
  provider: :llm,
  purpose: :classifier,
  outcome: :ok,
  duration_us: 12_430,
  invoked?: true
}

Configuration rejected before a worker starts is recorded with invoked?: false. Raw provider errors and responses are never copied into this list.

Execution Boundary

Evaluation runs the configured input and router pipelines because testing a different path would give misleading results. It does not:

  • load or persist through the state adapter;
  • load or write memory;
  • run the selected route handler;
  • plan or execute actions;
  • deliver journal records;
  • write online semantic-learning examples.

Classifier, embedding, semantic-cache lookup, and LLM adapters are router providers and therefore may be called. A real LLM evaluation can incur latency and provider charges. CI should normally use deterministic adapter fixtures; run a separate opt-in corpus when measuring a live provider.

The receipt is intended to become the common fact source for evaluation, journaling, and minimal telemetry. It does not calculate monetary token cost or export metrics.