Aludel.Evals.Sampling (aludel v0.7.0)

Copy Markdown View Source

Configuration and deterministic reduction for repeated test case execution.

Sampling is bounded to protect callers from accidental request amplification. Attempts remain ordered and are retained in the aggregate result. The latest attempt matching the reduced outcome supplies representative output and artifacts, while scores and usage are aggregated across every attempt.

Summary

Types

option()
reducer()
reducer_option()
t()

Functions

aggregate(attempts, sampling)

Aggregates ordered attempt results according to a validated configuration.

from_result(result)

Restores the sampling configuration stored in a test case result.

new(opts \\ [])

Validates sampling options.

Types

option()

@type option() :: {:samples, pos_integer()} | {:reducer, reducer_option()}

reducer()

@type reducer() :: :all | :any | :majority | :minimum_pass_rate

reducer_option()

@type reducer_option() :: :all | :any | :majority | {:minimum_pass_rate, number()}

t()

@type t() :: %Aludel.Evals.Sampling{
  minimum_pass_rate: float() | nil,
  reducer: reducer(),
  samples: pos_integer()
}

Functions

aggregate(attempts, sampling)

@spec aggregate([map()], t()) :: map()

Aggregates ordered attempt results according to a validated configuration.

The aggregate retains every attempt, reports pass-rate evidence, sums token, cost, and latency usage, and averages available attempt scores.

from_result(result)

@spec from_result(map()) :: {:ok, t()} | {:error, {:invalid_sampling, String.t()}}

Restores the sampling configuration stored in a test case result.

Results created before sampling default to one attempt with the :all reducer.

new(opts \\ [])

@spec new([option()]) :: {:ok, t()} | {:error, {:invalid_sampling, String.t()}}

Validates sampling options.

:samples defaults to 1 and accepts values up to 20. :reducer accepts :all, :any, :majority, or {:minimum_pass_rate, rate} where the rate is between 0.0 and 1.0.