mix spectre.eval (Spectre v0.3.0)

Copy Markdown View Source

Evaluates Spectre routing accuracy and LLM usage from a JSONL corpus.

mix spectre.eval MyApp.Agent test/fixtures/routing.jsonl

mix spectre.eval --agent MyApp.Agent test/fixtures/routing.jsonl       --json tmp/routing-report.json       --min-pass-rate 0.98

The task exits unsuccessfully when configured thresholds are not met.

Options

  • --agent - agent module; may instead be the first positional argument.
  • --json - write the complete JSON report to this path.
  • --min-pass-rate - minimum accepted case pass ratio, default 1.0.
  • --max-unnecessary-llm - maximum forbidden LLM calls, default 0.
  • --max-missing-llm - maximum missing required LLM calls, default 0.
  • --max-errors - optional maximum number of router errors.