LLM evaluation framework for Elixir.

Tribunal provides tools for evaluating LLM outputs and measuring response quality.

Quick Start

In Tests (ExUnit)

defmodule MyApp.RAGEvalTest do
  use ExUnit.Case
  use Tribunal.ExUnit

  @moduletag :eval

  test "response is grounded in context" do
    response = MyApp.RAG.query("What's the return policy?")

    assert response =~ "30 days"
    assert_faithful response, context: @docs, threshold: 0.8
  end
end

Dataset-Driven Evals

# test/evals/datasets/questions.json
[
  {
    "input": "What's the return policy?",
    "context": "Returns within 30 days with receipt.",
    "expected": {
      "contains": "30 days",
      "faithful": {"threshold": 0.8}
    }
  }
]

Then run: mix tribunal.eval

Assertion Types

Deterministic (no LLM, instant)

• contains - Output includes one substring
• not_contains - Output excludes substring(s)
• contains_any - Output includes at least one
• contains_all - Output includes all
• regex - Output matches pattern
• is_json - Output is valid JSON
• latency_ms - Response within time limit

LLM-as-Judge (requires req_llm)

• faithful - Response grounded in context
• relevant - Response addresses query
• correctness - Response matches expected answer
• refusal - Output is a refusal
• no_toxicity - Response contains no abusive language or materially harmful content
• no_pii - Response contains no personal information
• no_policy_violation - Response follows a supplied policy, including fairness, allowed scope, confidentiality, action claims, and identity

Embedding (requires alike)

• similar - Semantic similarity to golden answer

Installation

def deps do
  [
    {:tribunal, "~> 2.0"},

    # Optional: LLM-as-judge metrics
    {:req_llm, ">= 1.2.0 and < 2.0.0"},

    # Optional: embedding similarity
    {:alike, ">= 0.4.0 and < 0.5.0"}
  ]
end