LLM-based re-ranker that scores chunk relevance in a single batched call.
Sends all chunks to the LLM in one prompt, gets back JSON scores (0-10), then filters by threshold and sorts by score descending.
Cost
One LLM call per search, on the critical path. Measured on a ~230 document
collection at limit: 8 with over_fetch: 3: 16-35 seconds per search against
0.2-0.8 seconds unreranked. That is fine for a batch or background path and
usually too slow for an interactive one - Arcana.Reranker.CrossEncoder and
Arcana.Reranker.ColBERT score locally if you want reranking without the round
trip.
Also note the threshold filters: it defaults to 7, so chunks the LLM scores
below that are dropped from the results rather than moved down. See
Arcana.Reranker for what that trade means.
Which LLM it uses
In order: :llm in these opts (whether passed beside :reranker or inside its
option list), then config :arcana, llm: .... Arcana.Pipeline supplies
ctx.llm through the first. It raises only when neither is set.
Usage
# With Arcana.Pipeline (uses ctx.llm automatically)
ctx
|> Pipeline.search()
|> Pipeline.rerank()
|> Pipeline.answer()
# Directly
{:ok, reranked} = Arcana.Reranker.LLM.rerank(
"What is Elixir?",
chunks,
llm: &my_llm/1,
threshold: 7
)Custom prompt
Pass a :prompt function with arity 2 receiving (question, passages) where
passages is a list of {id, chunk} tuples:
prompt_fn = fn question, passages ->
# Build your own prompt using the question and passages
end
LLM.rerank("question", chunks, llm: llm, prompt: prompt_fn)