Retrieval evaluation for measuring search quality.
Generates synthetic test cases from your document chunks and evaluates retrieval performance with standard IR metrics.
Usage
# Generate test cases from chunks
{:ok, test_cases} = Arcana.Evaluation.generate_test_cases(
repo: MyApp.Repo,
llm: my_llm,
sample_size: 50
)
# Run evaluation
{:ok, run} = Arcana.Evaluation.run(repo: MyApp.Repo, mode: :vector)
# View metrics
run.metrics
# => %{recall_at_5: 0.84, precision_at_5: 0.68, mrr: 0.76, ...}Summary
Functions
Returns count of test cases.
Creates a manual test case.
Deletes an evaluation run.
Deletes a test case.
Generates synthetic test cases from existing chunks.
Gets a single evaluation run by ID.
Gets a single test case by ID.
Lists past evaluation runs.
Lists all test cases.
Runs evaluation against existing test cases.
Functions
Returns count of test cases.
Accepts the same :collections scoping option as list_test_cases/1.
Creates a manual test case.
Options
:repo- Ecto repo (required):question- The question text (required):relevant_chunk_ids- List of chunk IDs considered relevant (required):reference_answer- Optional ground-truth answer text, used by correctness scoring inrun/1when an:answereris configured.
Deletes an evaluation run.
Scoped the same way as delete_test_case/2.
Deletes a test case.
With :collections, the scope predicate rides inside the DELETE, so a
test case outside the allowed collections is rejected with
{:error, :not_found} and nothing can change between the check and the
delete. A malformed id is rejected the same way.
Generates synthetic test cases from existing chunks.
Samples chunks randomly and uses an LLM to generate questions that should retrieve those chunks.
Options
:repo- Ecto repo (required):llm- LLM implementing Arcana.LLM protocol (required):sample_size- Number of chunks to sample (default: 50):source_id- Limit to chunks from specific source:prompt- Custom prompt template
Gets a single evaluation run by ID.
Accepts the same :collections scoping option as list_runs/1.
Gets a single test case by ID.
Accepts the same :collections scoping option as list_test_cases/1;
a test case outside the scope reads as missing.
Lists past evaluation runs.
Options
:repo- Ecto repo (required):limit- Maximum runs to return (default: 20):collections- Only list runs recorded as having run under a non-empty subset of these collection names
Lists all test cases.
Options
:repo- Ecto repo (required):source_id- Filter by source (optional):collections- List of collection names to scope the listing to. See "Collection scoping" below.
Collection scoping
A test case has no collection of its own: it reaches one through its
chunks (the relevant chunks it is scored against, and the source chunk it
was generated from). When :collections is given, only test cases whose
every linked chunk resolves to one of those collections are returned, and
at least one link has to resolve. Anything else — a test case straddling
two collections, one whose chunks were deleted, one with no links at all
— stays hidden, since rendering it would expose the foreign half.
Runs evaluation against existing test cases.
Options
:repo- Ecto repo (required):mode- Search mode:vector | :keyword | :hybrid(default::vector).:semanticand:fulltextare deprecated aliases and log a warning.:source_id- Limit evaluation to specific source:evaluate_answers- When true, also evaluates answer quality (default: false):llm- LLM function (required when evaluate_answers is true):retriever- Custom retriever function(question, opts) -> {:ok, chunks}. Defaults toArcana.search/2. Use this to evaluate alternative retrieval strategies (e.g.,Arcana.Loop) against the same test set with the same metrics. The chunks returned must be maps with:idso the metrics can match them against the test case'srelevant_chunks.:run_ref- Opaque term echoed back in the per-test-case telemetry metadata.[:arcana, :evaluation, :test_case, :*]events carry the question being evaluated, and handlers are global, so a listener that wants only its own run's questions has to filter on something; this is it.:collections- List of collection names to confine the run to. It scopes which test cases run (seelist_test_cases/1), is forwarded to the retriever so retrieval can't reach outside those collections, and is recorded on the run so scoped listings can find it again. Retrieval then runs withstrict_collections: trueunless:strict_collectionssays otherwise, so a collection name with no row fails the search instead of widening it to the whole corpus.