Spectre.Eval (Spectre v0.3.0)

Copy Markdown View Source

Route-only routing regression evaluation.

Spectre.Eval executes only input normalization and routing. Selected route handlers, actions, state adapters, memory adapters, journal stores, and online semantic learning are not executed.

Summary

Functions

load(path)

Loads and validates JSONL cases or an in-memory case list.

run(agent, source, opts \\ [])

Evaluates an agent against a JSONL path or a list of case structs/maps.

Functions

load(path)

@spec load(Path.t() | [Spectre.Eval.Case.t() | map()]) ::
  {:ok, [Spectre.Eval.Case.t()]} | {:error, term()}

Loads and validates JSONL cases or an in-memory case list.

run(agent, source, opts \\ [])

@spec run(module(), Path.t() | [Spectre.Eval.Case.t() | map()], keyword()) ::
  {:ok, Spectre.Eval.Report.t()} | {:error, term()}

Evaluates an agent against a JSONL path or a list of case structs/maps.

{:ok, report} = Spectre.Eval.run(MyAgent, "test/fixtures/routing.jsonl")