adk_eval_set (erlang_adk v0.7.0)
View SourceVersioned, bounded multi-turn agent evaluation.
Schema/result version 2 adds full-case criteria, repeated samples, strict result accounting, deterministic sample identities, and per-case adapter lifecycle. Version 1 sets, adapters, per-turn metrics, and saved results remain accepted.
Summary
Functions
Compare a saved baseline with a current result. Options are max_pass_rate_drop and binary-keyed metric_tolerances.
Render a checked result or comparison as stable JSON or Markdown.
Run a versioned set.
Types
Functions
-spec compare(eval_result(), eval_result(), map()) -> {ok, map()} | {error, term()}.
Compare a saved baseline with a current result. Options are max_pass_rate_drop and binary-keyed metric_tolerances.
-spec decode_result(map()) -> {ok, eval_result()} | {error, term()}.
-spec encode_result(eval_result()) -> {ok, map()} | {error, term()}.
-spec report(eval_result() | map(), json | markdown) -> {ok, binary()} | {error, term()}.
Render a checked result or comparison as stable JSON or Markdown.
-spec result_schema_version() -> pos_integer().
Run a versioned set.
Existing module descriptors default to the legacy per-turn score/4 callback. A descriptor with scope set to case calls score_case/2 with the entire canonical case. Built-ins use criterion = exact_response, trajectory_exact, trajectory_in_order, trajectory_any_order, or trajectory_subset. Trajectory config accepts args = exact, subset, or ignored.
New options are sample_count (alias samples), sample_concurrency, sample_pass_rate_threshold, min_successful_samples, empty_criteria (error by default; pass must be explicit), and max_report_bytes.
-spec schema_version() -> pos_integer().