Imp.Evaluate (Imp v0.5.0)

Copy Markdown View Source

Evaluate a program against examples and a metric.

Evaluation is the hinge between "the model answered" and "the program got better." A metric turns each (example, prediction) pair into a score; optimizers use those scores to compare candidate programs.

Example

iex> lm = Imp.LM.Static.new(handler: fn _messages, _opts -> %{answer: "Paris"} end)
iex> program = Imp.predict("question -> answer", lm: lm)
iex> devset = [
...>   Imp.example(question: "Capital of France?", answer: "Paris")
...>   |> Imp.with_inputs(:question)
...> ]
iex> metric = Imp.Metrics.exact_match(:answer)
iex> evaluator = Imp.Evaluate.new(devset, metric)
iex> report = Imp.Evaluate.run(evaluator, program)
iex> report.score
1.0

Metrics may return booleans, numbers, maps with :score and :feedback, or %Imp.Metrics.Result{}. An arity-3 metric receives nil as its trace here, as in DSPy (see Imp.Metrics). Program and metric failures are recorded as failed rows so optimizers can keep searching and report diagnostics.

The per-row :timeout defaults to :infinity, matching DSPy's Evaluate (which imposes no per-example deadline). When a finite :timeout kills a row it is logged loudly and the row carries {:evaluation_task_exit, :timeout} with a nil prediction, so a killed call stays distinguishable from a wrong answer. A finite :timeout is enforced at every concurrency level, including the default num_threads: 1.

When :max_errors is reached the evaluation halts LOUDLY by raising Imp.EvaluationCancelledError, mirroring DSPy's ParallelExecutor (halts once error_count >= max_errors and raises "Execution cancelled due to errors or interruption."). A truncated run never returns a normal-looking partial score. Deviation from upstream: :max_errors defaults to :infinity here, while DSPy inherits dspy.settings.max_errors (default 10); pass :max_errors explicitly for the upstream behavior.

Summary

Types

t()

@type t() :: %Imp.Evaluate{
  deadline: term(),
  devset: Enumerable.t(),
  display_progress: boolean(),
  failure_score: number(),
  max_errors: non_neg_integer() | :infinity,
  metric: function(),
  num_threads: pos_integer(),
  timeout: timeout()
}

Functions

new(devset, metric, opts \\ [])

run(evaluator, program)