barrel_vectordb_bm25_disk (barrel_vectordb v2.1.2)
View SourceDisk-Native BM25 Backend
Implements BM25 text search with disk-native storage using: - Block-Max MaxScore algorithm for early termination - Hot layer for fast writes, background compaction to disk - RocksDB for term/doc ID mapping - mmap for block-max index reads
Architecture: - Hot layer (RAM): Recent documents, sub-ms write latency - Disk layer (SSD): Compressed postings with block-max index - RocksDB: term string <-> int ID, doc string <-> int ID mapping - ETS: Doc lengths and stats (small, hot)
Summary
Functions
Add a document to the index
Build index from list of documents Docs format: [{DocId, Text}, ...]
Close the index
Compact hot layer to disk
Encode text into sparse vector without adding to index
Get sparse vector representation of a document
Get detailed index info
Create a new disk-native BM25 index Options: - base_path: Path for disk storage (required) - k1: BM25 k1 parameter (default: 1.2) - b: BM25 b parameter (default: 0.75) - min_term_length: Minimum token length (default: 1) - lowercase: Lowercase tokens (default: true) - hot_max_size: Max docs in hot layer (default: 50000) - hot_compaction_threshold: Trigger compaction at this % (default: 0.8) - block_size: Documents per posting block (default: 128)
Open an existing disk-native BM25 index
Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env is the EncryptedEnv for the bm25.ids RocksDB.
Remove a document from the index
Search the index
Search the index and return metrics for debugging/tuning Returns {Results, Metrics} where Metrics includes block skip statistics
Get index statistics
Sync to disk
Types
-type bm25_disk_index() :: #bm25_disk_index{config :: #bm25_disk_config{k1 :: float(), b :: float(), min_term_length :: pos_integer(), lowercase :: boolean(), block_size :: pos_integer()}, base_path :: binary(), total_docs :: non_neg_integer(), total_tokens :: non_neg_integer(), next_term_int_id :: non_neg_integer(), next_doc_int_id :: non_neg_integer(), id_db :: rocksdb:db_handle() | undefined, cf_terms_fwd :: rocksdb:cf_handle() | undefined, cf_terms_rev :: rocksdb:cf_handle() | undefined, cf_docs_fwd :: rocksdb:cf_handle() | undefined, cf_docs_rev :: rocksdb:cf_handle() | undefined, id_db_standalone :: boolean(), id_db_env :: term() | undefined, file_handle :: term() | undefined, doc_stats_table :: ets:tid() | undefined, term_stats_table :: ets:tid() | undefined, hot_enabled :: boolean(), hot_max_size :: non_neg_integer(), hot_compaction_threshold :: float(), hot_postings :: #{non_neg_integer() => [{non_neg_integer(), pos_integer()}]}, hot_docs :: #{non_neg_integer() => #{non_neg_integer() => pos_integer()}}, hot_doc_lengths :: #{non_neg_integer() => non_neg_integer()}, hot_size :: non_neg_integer(), hot_tokens :: non_neg_integer(), disk_doc_count :: non_neg_integer(), disk_term_count :: non_neg_integer(), disk_total_tokens :: non_neg_integer(), blockmax_index :: #{non_neg_integer() => [map()]}, compaction_in_progress :: boolean()}.
Functions
-spec add(bm25_disk_index(), binary(), binary()) -> {ok, bm25_disk_index()} | {error, term()}.
Add a document to the index
-spec build(bm25_disk_index(), [{binary(), binary()}]) -> {ok, bm25_disk_index()} | {error, term()}.
Build index from list of documents Docs format: [{DocId, Text}, ...]
-spec close(bm25_disk_index()) -> ok.
Close the index
-spec compact(bm25_disk_index()) -> {ok, bm25_disk_index()} | {error, term()}.
Compact hot layer to disk
-spec encode(bm25_disk_index(), binary()) -> sparse_vector().
Encode text into sparse vector without adding to index
-spec get_vector(bm25_disk_index(), binary()) -> {ok, sparse_vector()} | {error, not_found}.
Get sparse vector representation of a document
-spec info(bm25_disk_index()) -> map().
Get detailed index info
-spec new(map()) -> {ok, bm25_disk_index()} | {error, term()}.
Create a new disk-native BM25 index Options: - base_path: Path for disk storage (required) - k1: BM25 k1 parameter (default: 1.2) - b: BM25 b parameter (default: 0.75) - min_term_length: Minimum token length (default: 1) - lowercase: Lowercase tokens (default: true) - hot_max_size: Max docs in hot layer (default: 50000) - hot_compaction_threshold: Trigger compaction at this % (default: 0.8) - block_size: Documents per posting block (default: 128)
-spec open(binary() | string()) -> {ok, bm25_disk_index()} | {error, term()}.
Open an existing disk-native BM25 index
-spec open(binary() | string(), map()) -> {ok, bm25_disk_index()} | {error, term()}.
Open with options: crypto => none | #{key := <<_:256>>, env => rocksdb env}. The key encrypts the flat files; the env is the EncryptedEnv for the bm25.ids RocksDB.
-spec remove(bm25_disk_index(), binary()) -> {ok, bm25_disk_index()} | {error, not_found}.
Remove a document from the index
-spec search(bm25_disk_index(), binary(), pos_integer()) -> [{binary(), float()}].
Search the index
-spec search_with_metrics(bm25_disk_index(), binary(), pos_integer()) -> {[{binary(), float()}], map()}.
Search the index and return metrics for debugging/tuning Returns {Results, Metrics} where Metrics includes block skip statistics
-spec stats(bm25_disk_index()) -> map().
Get index statistics
-spec sync(bm25_disk_index()) -> ok.
Sync to disk