Evaluate a program against examples and a metric.
Evaluation is the hinge between "the model answered" and "the program got
better." A metric turns each (example, prediction) pair into a score;
optimizers use those scores to compare candidate programs.
Example
iex> lm = Imp.LM.Static.new(handler: fn _messages, _opts -> %{answer: "Paris"} end)
iex> program = Imp.predict("question -> answer", lm: lm)
iex> devset = [
...> Imp.example(question: "Capital of France?", answer: "Paris")
...> |> Imp.with_inputs(:question)
...> ]
iex> metric = Imp.Metrics.exact_match(:answer)
iex> evaluator = Imp.Evaluate.new(devset, metric)
iex> report = Imp.Evaluate.run(evaluator, program)
iex> report.score
1.0Metrics may return booleans, numbers, maps with :score and :feedback, or
%Imp.Metrics.Result{}. An arity-3 metric receives nil as its trace here,
as in DSPy (see Imp.Metrics).
Program and metric failures are recorded as failed rows so optimizers can keep
searching and report diagnostics. A row whose inputs were never declared with
Imp.with_inputs/2 is not a failed row: run/2 raises ArgumentError before
calling the program, because every field, labels included, would otherwise
reach it. A plain map or field pair list cannot declare inputs, so a devset of
them raises the same way. DSPy's Evaluate records such a row's error and
goes on until max_errors; Imp refuses before any row runs, since no row of
such a devset can be scored honestly.
The per-row :timeout defaults to :infinity, matching DSPy's Evaluate
(which imposes no per-example deadline). When a finite :timeout kills a row
it is logged loudly and the row carries {:evaluation_task_exit, :timeout}
with a nil prediction, so a killed call stays distinguishable from a wrong
answer. A finite :timeout is enforced at every concurrency level,
including the default num_threads: 1.
When :max_errors is reached the evaluation halts LOUDLY by raising
Imp.EvaluationCancelledError, mirroring DSPy's ParallelExecutor
(halts once error_count >= max_errors and raises "Execution cancelled
due to errors or interruption."). A truncated run never returns a
normal-looking partial score. Deviation from upstream: :max_errors
defaults to :infinity here, while DSPy inherits dspy.settings.max_errors
(default 10); pass :max_errors explicitly for the upstream behavior.
Summary
Types
@type t() :: %Imp.Evaluate{ deadline: term(), devset: Enumerable.t(), display_progress: boolean(), failure_score: number(), max_errors: non_neg_integer() | :infinity, metric: function(), num_threads: pos_integer(), timeout: timeout() }