Behaviour for LLM-as-judge assertions.
All judges (built-in and custom) implement this behaviour. This provides a consistent interface for evaluation criteria.
Example
defmodule MyApp.Judges.BrandVoice do
@behaviour Tribunal.Judge
@impl true
def name, do: :brand_voice
@impl true
def prompt(test_case, _opts) do
"""
Evaluate if the response matches our brand voice guidelines:
- Friendly but professional tone
- No jargon or technical terms
- Empathetic and helpful
Response to evaluate:
#{test_case.actual_output}
Query: #{Tribunal.TestCase.evaluation_input(test_case)}
"""
end
endConfiguration
Register your custom judges in config:
config :tribunal, :custom_judges, [
MyApp.Judges.BrandVoice,
MyApp.Judges.Compliance
]The value returned by name/0 becomes an assertion type for datasets and
Tribunal.Assertions.evaluate/3. Name the condition that makes the assertion
pass. For example, a detector for policy violations should expose
:no_policy_violation because a compliant response passes.
Summary
Callbacks
Optional: customize how the LLM result is interpreted.
Returns the atom name for this judge.
Optional: whether "no" verdict means pass (for negative metrics like toxicity).
Builds the evaluation prompt for the LLM judge.
Optional: validate that the test case has required fields.
Functions
Returns list of all judge names (built-in + custom).
Returns all judge modules (built-in + custom).
Checks if a name is a built-in judge.
Returns list of built-in judge names.
Returns all built-in judge modules.
Checks if a name is a registered custom judge.
Returns list of custom judge names.
Returns all configured custom judge modules.
Finds a judge module by name.
Callbacks
Optional: customize how the LLM result is interpreted.
By default, uses verdict and threshold logic. Override for custom pass/fail logic.
Should return {:pass, details} or {:fail, details}.
@callback name() :: atom()
Returns the atom name for this judge.
This name is used as the assertion type in datasets and
Tribunal.Assertions.evaluate/3. It should describe the passing condition.
@callback negative_metric?() :: boolean()
Optional: whether "no" verdict means pass (for negative metrics like toxicity).
When true, verdict "no" = pass and "yes" = fail. When false (default), verdict "yes" = pass and "no" = fail.
@callback prompt(test_case :: Tribunal.TestCase.t(), opts :: keyword()) :: String.t()
Builds the evaluation prompt for the LLM judge.
Receives the test case and any options passed to the assertion. Should return a prompt string that asks the LLM to evaluate the response and return a JSON verdict.
The prompt should instruct the LLM to return JSON with:
verdict: "yes", "no", or "partial"reason: explanation for the verdictscore: confidence score 0.0-1.0
@callback validate(test_case :: Tribunal.TestCase.t()) :: :ok | {:error, String.t()}
Optional: validate that the test case has required fields.
Return :ok if valid, or {:error, reason} if not.
Default implementation always returns :ok.
Functions
Returns list of all judge names (built-in + custom).
Returns all judge modules (built-in + custom).
Checks if a name is a built-in judge.
Returns list of built-in judge names.
Returns all built-in judge modules.
Checks if a name is a registered custom judge.
Returns list of custom judge names.
Returns all configured custom judge modules.
Finds a judge module by name.
Returns {:ok, module} or :error.