LangEx.Eval.Trajectory (LangEx v0.12.0)

Copy Markdown View Source

Extraction and matching of agent tool-call trajectories.

A trajectory is the ordered list of tool calls an agent made during a run. It can be extracted from a message history or a saved checkpoint, then compared against an expected trajectory to make agent regressions CI-testable (the agentevals pattern).

Summary

Types

expected_step()
match_result()
mode()
step()

Functions

from_checkpoint(checkpoint)

Extract the tool-call trajectory from a checkpoint's state messages.

from_messages(messages)

Extract the tool-call trajectory from a message history.

match(actual, expected, opts \\ [])

Match an actual trajectory against an expected one.

Types

expected_step()

@type expected_step() :: %{:name => String.t(), optional(:args) => map()}

match_result()

@type match_result() :: %{
  match?: boolean(),
  missing: [expected_step()],
  unexpected: [step()],
  detail: String.t()
}

mode()

@type mode() :: :strict | :unordered | :subset

step()

@type step() :: %{name: String.t(), args: map()}

Functions

from_checkpoint(checkpoint)

@spec from_checkpoint(LangEx.Checkpoint.t()) :: [step()]

Extract the tool-call trajectory from a checkpoint's state messages.

Returns [] when the checkpoint state carries no :messages key.

from_messages(messages)

@spec from_messages([LangEx.Message.t()]) :: [step()]

Extract the tool-call trajectory from a message history.

Walks the history in order and flattens every AI turn's tool calls into %{name: name, args: args} steps.

match(actual, expected, opts \\ [])

@spec match([step()], [expected_step()], keyword()) :: match_result()

Match an actual trajectory against an expected one.

An expected step matches an actual step when their names are equal and every key-value pair in the expected :args (when given) equals the actual args value — partial args matching. Each expected step consumes a distinct actual step, so duplicated expected steps require as many matching actual steps.

Options

  • :mode - :strict (same steps, same order, same length), :unordered (same steps in any order), or :subset (every expected step present; extra actual steps are reported in :unexpected but do not fail the match). Defaults to :subset.