Evaluate a program against examples and a metric.
Evaluation is the hinge between "the model answered" and "the program got
better." A metric turns each (example, prediction) pair into a score;
optimizers use those scores to compare candidate programs.
Example
iex> lm = Imp.LM.Static.new(handler: fn _messages, _opts -> %{answer: "Paris"} end)
iex> program = Imp.predict("question -> answer", lm: lm)
iex> devset = [
...> Imp.example(question: "Capital of France?", answer: "Paris")
...> |> Imp.with_inputs(:question)
...> ]
iex> metric = Imp.Metrics.exact_match(:answer)
iex> evaluator = Imp.Evaluate.new(devset, metric)
iex> report = Imp.Evaluate.run(evaluator, program)
iex> report.score
1.0Metrics may return booleans, numbers, maps with :score and :feedback, or
%Imp.Metrics.Result{}. An arity-3 metric receives nil as its trace here,
as in DSPy (see Imp.Metrics).
Program and metric failures are recorded as failed rows so optimizers can keep
searching and report diagnostics.
The per-row :timeout defaults to :infinity, matching DSPy's Evaluate
(which imposes no per-example deadline). When a finite :timeout kills a row
it is logged loudly and the row carries {:evaluation_task_exit, :timeout}
with a nil prediction, so a killed call stays distinguishable from a wrong
answer. A finite :timeout is enforced at every concurrency level,
including the default num_threads: 1.
When :max_errors is reached the evaluation halts LOUDLY by raising
Imp.EvaluationCancelledError, mirroring DSPy's ParallelExecutor
(halts once error_count >= max_errors and raises "Execution cancelled
due to errors or interruption."). A truncated run never returns a
normal-looking partial score. Deviation from upstream: :max_errors
defaults to :infinity here, while DSPy inherits dspy.settings.max_errors
(default 10); pass :max_errors explicitly for the upstream behavior.
Summary
Types
@type t() :: %Imp.Evaluate{ deadline: term(), devset: Enumerable.t(), display_progress: boolean(), failure_score: number(), max_errors: non_neg_integer() | :infinity, metric: function(), num_threads: pos_integer(), timeout: timeout() }