adk_eval (erlang_adk v0.7.0)

adk_eval - Lightweight evaluation helpers for Erlang ADK.

Allows defining datasets and running them against an agent to calculate metrics.

Summary

Functions

run(AgentRef, Dataset, MetricFn)

Run an evaluation.

run(AgentRef, Dataset, MetricFn, Opts)

Run an evaluation with options (e.g. concurrency).

Types

dataset/0

-type dataset() :: [#{input => term(), expected => term(), metadata => map()}].

metric_fn/0

-type metric_fn() :: fun((Expected :: term(), Actual :: term()) -> float()).

Functions

run(AgentRef, Dataset, MetricFn)

-spec run(AgentRef :: term(), Dataset :: dataset(), MetricFn :: metric_fn()) ->
             {ok, map()} | {error, term()}.

Run an evaluation.

run(AgentRef, Dataset, MetricFn, Opts)

-spec run(AgentRef :: term(), Dataset :: dataset(), MetricFn :: metric_fn(), Opts :: map()) ->
             {ok, map()} | {error, term()}.

Run an evaluation with options (e.g. concurrency).

concurrency controls the number of evaluator worker processes. Calls made to one stateful agent pid are still serialized by that agent's gen_server; use separate agent processes when model calls themselves must overlap.