Aludel.ExUnit (aludel v0.7.0)

Adds evaluation assertions to ExUnit tests.

Use assert_evaluation/2 for one generated response, or assert_evaluations/2 when the same response must satisfy several metrics. Persisted suite runs can be checked with assert_suite_run/1. assert_suite/3 or assert_suite/4 executes and persists a suite before applying the same gate.

Failure messages identify the metric, failed test case, or quality-policy rule without copying generated model output into test logs.

Usage

defmodule MyApp.AnswerTest do
  use ExUnit.Case
  use Aludel.ExUnit

  test "answers with the expected city" do
    output = MyApp.answer("What is the capital of France?")

    assert_evaluation(output, %{
      "type" => "exact_match",
      "value" => "Paris"
    })
  end
end

The helpers can also be called as qualified functions without use.

Summary

Types

evaluation_assertion()

An assertion accepted by Aludel.Evals.AssertionEvaluator.

Functions

assert_evaluation(input, assertion)

Evaluates one assertion and returns its normalized result when it passes.

assert_evaluations(input, assertions)

Evaluates a non-empty list of assertions and returns the ordered results.

assert_suite(suite, prompt_version, provider, options \\ [])

Executes, persists, and gates an evaluation suite from an ExUnit test.

assert_suite_run(suite_run)

Requires an existing persisted suite run to have an effective passing status.

Types

evaluation_assertion()

@type evaluation_assertion() :: map()

An assertion accepted by Aludel.Evals.AssertionEvaluator.

Functions

assert_evaluation(input, assertion)

@spec assert_evaluation(
  Aludel.Evals.Metric.Context.t() | String.t(),
  evaluation_assertion()
) :: map()

Evaluates one assertion and returns its normalized result when it passes.

The input can be generated output text or an Aludel.Evals.Metric.Context. A failed, invalid, unsupported, or unavailable evaluator raises ExUnit.AssertionError with bounded diagnostic evidence. Generated output and expected values are omitted from the failure message.

assert_evaluations(input, assertions)

@spec assert_evaluations(Aludel.Evals.Metric.Context.t() | String.t(), [
  evaluation_assertion()
]) :: [
  map()
]

Evaluates a non-empty list of assertions and returns the ordered results.

All assertions run so one failure reports every non-passing metric, up to a bounded diagnostic limit. Raises ExUnit.AssertionError when the list is empty or any assertion does not pass.

assert_suite(suite, prompt_version, provider, options \\ [])

@spec assert_suite(
  Aludel.Evals.Suite.t(),
  Aludel.Prompts.PromptVersion.t(),
  Aludel.Providers.Provider.t(),
  keyword()
) :: Aludel.Evals.SuiteRun.t()

Executes, persists, and gates an evaluation suite from an ExUnit test.

Sampling options are forwarded to Aludel.Evals.execute_suite/4. A completed non-passing run remains persisted before this helper raises. Configuration or execution errors raise with a stable error category.

assert_suite_run(suite_run)

@spec assert_suite_run(Aludel.Evals.SuiteRun.t()) :: Aludel.Evals.SuiteRun.t()

Requires an existing persisted suite run to have an effective passing status.

A stored quality-policy result determines the effective status when present. Otherwise, every test case must pass and an empty run fails. The passing run is returned unchanged so callers can make additional assertions about it.