BM25 sparse embeddings (pure Erlang, in-memory)

Implements BM25 (Best Matching 25) algorithm for sparse text retrieval. No external dependencies - runs entirely in Erlang.

Important: In-Memory Index

This module maintains an **in-memory** index. The index is not persisted and will be lost when the process terminates. This design is intentional:

- Simple and fast for small/medium datasets - Suitable for hybrid search (combine with dense vectors) - Can be rebuilt from documents stored in barrel_vectordb on startup - No additional storage overhead

For large datasets requiring persistence, consider rebuilding the index from your document store on application startup.

Use Cases

- Hybrid search: combine BM25 keyword scores with dense vector similarity - Re-ranking: use BM25 to re-rank dense search results - Keyword filtering: pre-filter candidates before expensive dense search - Small datasets: where the index comfortably fits in memory

Usage

   %% Create a new BM25 index
   Index = barrel_vectordb_bm25:new().
  
   %% Add documents
   Index1 = barrel_vectordb_bm25:add(Index, <<"doc1">>, <<"Hello world">>).
   Index2 = barrel_vectordb_bm25:add(Index1, <<"doc2">>, <<"Hello there">>).
  
   %% Search
   Results = barrel_vectordb_bm25:search(Index2, <<"hello">>, 10).
   %% => [{<<"doc1">>, 0.85}, {<<"doc2">>, 0.82}]
  
   %% Get sparse vector for a document
   {ok, SparseVec} = barrel_vectordb_bm25:get_vector(Index2, <<"doc1">>).
   %% => #{<<"hello">> => 0.5, <<"world">> => 0.8}
  
   %% Hybrid search example (combine with dense vectors)
   DenseResults = barrel_vectordb:search(Store, Query, #{k => 100}),
   BM25Results = barrel_vectordb_bm25:search(Index, Query, 100),
   HybridResults = combine_scores(DenseResults, BM25Results, 0.5).

Algorithm

BM25 score = sum(IDF(t) * TF(t,d) * (k1 + 1) / (TF(t,d) + k1 * (1 - b + b * |d| / avgdl)))

Where: - IDF(t) = log((N - n(t) + 0.5) / (n(t) + 0.5) + 1) - TF(t,d) = term frequency of t in document d - N = total number of documents - n(t) = number of documents containing term t - |d| = length of document d (in tokens) - avgdl = average document length - k1 = 1.2 (term frequency saturation parameter) - b = 0.75 (length normalization parameter)