Aludel.Evals.Metric behaviour (aludel v0.7.0)

Contract and shared helpers for evaluation metrics.

Metrics return a normalized result. Compatibility with historical assertion result maps is handled by Aludel.Evals.AssertionEvaluator.

Summary

Types

input()

json_value()

Callbacks

evaluate(input, map)

type()

Functions

boolean_result(type, passed, success_reason, failure_reason, metadata \\ %{})

Builds a normalized metric result from a boolean decision.

compare_json_values(actual, expected)

Compares JSON-compatible values while preserving scalar types.

decode_json(output)

Decodes JSON output after removing an optional Markdown code fence.

invalid_result(type)

Returns a failed result for an invalid metric configuration.

normalize_threshold(arg1)

Returns an assertion's numeric threshold or the default of 100.0.

output(output)

Returns generated output from either a contextual or legacy metric input.

Types

input()

@type input() :: Aludel.Evals.Metric.Context.t() | String.t()

json_value()

@type json_value() ::
  nil
  | boolean()
  | number()
  | String.t()
  | [json_value()]
  | %{optional(String.t()) => json_value()}

Callbacks

evaluate(input, map)

@callback evaluate(input(), map()) :: Aludel.Evals.Metric.Result.t()

type()

@callback type() :: String.t()

Functions

boolean_result(type, passed, success_reason, failure_reason, metadata \\ %{})

@spec boolean_result(String.t(), boolean(), String.t(), String.t(), map()) ::
  Aludel.Evals.Metric.Result.t()

Builds a normalized metric result from a boolean decision.

Passing results score 100.0; failing results score 0.0. The selected reason and optional metadata are retained in the result.

compare_json_values(actual, expected)

@spec compare_json_values(term(), term()) :: boolean()

Compares JSON-compatible values while preserving scalar types.

Maps and lists are compared through their encoded JSON representation while scalar values use strict Elixir equality.

decode_json(output)

@spec decode_json(String.t()) :: {:ok, json_value()} | {:error, Jason.DecodeError.t()}

Decodes JSON output after removing an optional Markdown code fence.

invalid_result(type)

@spec invalid_result(String.t()) :: Aludel.Evals.Metric.Result.t()

Returns a failed result for an invalid metric configuration.

normalize_threshold(arg1)

@spec normalize_threshold(map()) :: float()

Returns an assertion's numeric threshold or the default of 100.0.

output(output)

@spec output(input()) :: String.t()

Returns generated output from either a contextual or legacy metric input.