Spectre.Eval.Report (Spectre v0.3.0)

Aggregate routing quality and LLM-use metrics for an evaluation corpus.

Summary

Types

t()

Functions

acceptable?(report, opts \\ [])

Checks configurable CI thresholds.

format(report)

Formats the compact terminal summary used by mix spectre.eval.

new(results)

Aggregates case results into a report.

to_map(report)

Returns a JSON-safe report map.

Types

t()

@type t() :: %Spectre.Eval.Report{
  clarifications: non_neg_integer(),
  confusion_matrix: map(),
  correct_routes: non_neg_integer(),
  duration_us: map(),
  errors: non_neg_integer(),
  failed: non_neg_integer(),
  llm_calls: non_neg_integer(),
  missing_required_llm_calls: non_neg_integer(),
  outcomes: map(),
  pass_rate: float(),
  passed: non_neg_integer(),
  results: [Spectre.Eval.Result.t()],
  route_accuracy: float(),
  route_cases: non_neg_integer(),
  strategies: map(),
  tags: map(),
  total: non_neg_integer(),
  unnecessary_llm_calls: non_neg_integer()
}

Functions

acceptable?(report, opts \\ [])

@spec acceptable?(
  t(),
  keyword()
) :: boolean()

Checks configurable CI thresholds.

Defaults require every case to pass and allow no missing or unnecessary LLM calls.

format(report)

@spec format(t()) :: String.t()

Formats the compact terminal summary used by mix spectre.eval.

new(results)

@spec new([Spectre.Eval.Result.t()]) :: t()

Aggregates case results into a report.

to_map(report)

@spec to_map(t()) :: map()

Returns a JSON-safe report map.