Retrieval evaluation for measuring search quality.
Generates synthetic test cases from your document chunks and evaluates retrieval performance with standard IR metrics.
Usage
# Generate test cases from chunks
{:ok, test_cases} = Arcana.Evaluation.generate_test_cases(
repo: MyApp.Repo,
llm: my_llm,
sample_size: 50
)
# Run evaluation
{:ok, run} = Arcana.Evaluation.run(repo: MyApp.Repo, mode: :vector)
# View metrics
run.metrics
# => %{recall_at_5: 0.84, precision_at_5: 0.68, mrr: 0.76, ...}Summary
Functions
Returns count of test cases.
Creates a manual test case.
Deletes an evaluation run.
Deletes a test case.
Generates synthetic test cases from existing chunks.
Gets a single evaluation run by ID.
Gets a single test case by ID.
Lists past evaluation runs.
Lists all test cases.
Runs evaluation against existing test cases.
Functions
Returns count of test cases.
Creates a manual test case.
Options
:repo- Ecto repo (required):question- The question text (required):relevant_chunk_ids- List of chunk IDs considered relevant (required):reference_answer- Optional ground-truth answer text, used by correctness scoring inrun/1when an:answereris configured.
Deletes an evaluation run.
Deletes a test case.
Generates synthetic test cases from existing chunks.
Samples chunks randomly and uses an LLM to generate questions that should retrieve those chunks.
Options
:repo- Ecto repo (required):llm- LLM implementing Arcana.LLM protocol (required):sample_size- Number of chunks to sample (default: 50):source_id- Limit to chunks from specific source:prompt- Custom prompt template
Gets a single evaluation run by ID.
Gets a single test case by ID.
Lists past evaluation runs.
Options
:repo- Ecto repo (required):limit- Maximum runs to return (default: 20)
Lists all test cases.
Options
:repo- Ecto repo (required):source_id- Filter by source (optional)
Runs evaluation against existing test cases.
Options
:repo- Ecto repo (required):mode- Search mode:vector | :keyword | :hybrid(default::vector).:semanticand:fulltextare deprecated aliases and log a warning.:source_id- Limit evaluation to specific source:evaluate_answers- When true, also evaluates answer quality (default: false):llm- LLM function (required when evaluate_answers is true):retriever- Custom retriever function(question, opts) -> {:ok, chunks}. Defaults toArcana.search/2. Use this to evaluate alternative retrieval strategies (e.g.,Arcana.Loop) against the same test set with the same metrics. The chunks returned must be maps with:idso the metrics can match them against the test case'srelevant_chunks.