Async cross-encoder reranking server with request multiplexing

A gen_server that manages communication with a Python cross-encoder server. Supports concurrent requests by tracking request IDs and routing responses back to the correct callers.

Cross-encoders score query-document pairs directly, providing more accurate relevance scores than bi-encoder similarity for reranking candidate results.

Requirements

   pip install transformers torch

Usage

   %% Start the reranker
   {ok, Server} = barrel_rerank:start_link(#{}).
  
   %% Make concurrent rerank requests
   spawn(fun() -> barrel_rerank:rerank(Server, Query1, Docs1) end).
   spawn(fun() -> barrel_rerank:rerank(Server, Query2, Docs2) end).
  
   %% Stop when done
   barrel_rerank:stop(Server).

Request Multiplexing

Multiple Erlang processes can call rerank/3,4 concurrently. Each request is assigned a unique ID and sent to Python. The Python server includes the ID in its response, allowing this gen_server to route responses to the correct waiting callers.

Configuration

   Config = #{
       python => "python3",                               %% Python executable (fallback)
       model => "cross-encoder/ms-marco-MiniLM-L-6-v2",   %% Model name
       timeout => 120000                                  %% Timeout in ms
   }.

Managed Venv

The rerank server uses the managed venv from barrel_rerank_venv. Dependencies (transformers, torch, uvloop) are auto-installed when the application starts.

Supported Models

- "cross-encoder/ms-marco-MiniLM-L-6-v2" - Default, fast, good quality - "cross-encoder/ms-marco-MiniLM-L-12-v2" - Better quality, slower - "BAAI/bge-reranker-base" - Good quality - "BAAI/bge-reranker-large" - Best quality, slowest

Integration with Search

Typical two-stage retrieval:

   %% Stage 1: Fast vector search (top 100)
   {ok, Candidates} = barrel_vectordb:search(Store, Query, #{k => 100}),
  
   %% Stage 2: Rerank top candidates
   Docs = [maps:get(text, C) || C <- Candidates],
   {ok, Ranked} = barrel_rerank:rerank(Server, Query, Docs),
  
   %% Get top 10 after reranking
   Top10Indices = [Idx || {Idx, _Score} <- lists:sublist(Ranked, 10)],
   Top10 = [lists:nth(Idx + 1, Candidates) || Idx <- Top10Indices].