LLM-as-judge scoring of agent tool-use trajectories.
Renders either a message history (compact transcript) or an extracted
trajectory (see LangEx.Eval.Trajectory) and asks a judge model to
score it between 0.0 and 1.0 against the supplied criteria, using
structured output for a machine-readable verdict.
Summary
Functions
Score a trajectory with an LLM judge.
Functions
@spec run( [LangEx.Message.t()] | [LangEx.Eval.Trajectory.step()], keyword() ) :: {:ok, %{score: float(), reasoning: String.t()}} | {:error, term()}
Score a trajectory with an LLM judge.
Accepts either a full message history (rendered as a compact
transcript of AI turns, tool calls, and tool replies) or a list of
trajectory steps (rendered as one name(args) line per call).
Options
:model(required) - judge model string:provider- LLM provider module (defaultLangEx.LLM.Anthropic):criteria- description of what a good trajectory looks like, spliced into the judge prompt:llm_opts- extra options forwarded to the provider call