barrel_vectordb_bm25_disk_file (barrel_vectordb v2.1.2)
View SourceBM25 Disk File I/O
Handles persistent storage for disk-native BM25 index with: - Varint encoding for compact posting storage - 4KB sector-aligned writes for optimal SSD performance - Block-max index for early termination - mmap support for zero-copy reads
File layout: - bm25.meta: Header + global stats (4KB) - bm25.postings: Compressed posting blocks (4KB aligned) - bm25.blockmax: Block-max index (mmap'd) - bm25.terms/: RocksDB for term <-> int mapping - bm25.docs/: RocksDB for doc <-> int mapping
Summary
Functions
Close file handles
Create a new BM25 disk index
Decode a posting block Returns list of {DocIntId, TF}
Encode a block of postings using delta encoding + varint Input: [{DocIntId, TF}, ...] sorted by DocIntId ascending Output: Binary with format: [Count:varint][Delta1:varint][TF1:varint]...
Get the base path
Check if file has mmap handle
Lookup blocks for a specific term
Close mmap handle
Open a file with mmap for zero-copy reads
Read from mmap
Open an existing BM25 disk index
Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.
Pad binary to sector alignment
Read a block from the specified offset
Read block-max index from file Returns map of TermIntId => [BlockMaxEntry]
Read multiple blocks (batch read)
Read index header (returns cached header)
Read postings from specified offset and size
Fresh postings nonce for a full rewrite (compaction): posting offsets are reused across compactions, so the static-mode nonce must rotate before the rewrite. Persisted by the next header write.
Sync all file handles to disk
Update statistics in header
Decode a varint from the beginning of a binary Returns {Value, Rest} where Rest is the remaining binary
Decode multiple varints from a binary Returns list of values
Encode an unsigned integer as a varint (7-bit encoding) Each byte uses 7 bits for data and 1 bit as continuation flag
Encode a list of integers as varints
Write a block at the specified offset (sector-aligned)
Write block-max index to file Format: [TermCount:32][Entry1][Entry2]... Entry: [TermIntId:32][BlockCount:16][Block1]...[BlockN] Block: [MaxImpact:32/float][DocStart:32][DocEnd:32][Offset:64][Size:32]
Write index header
Write postings for a term at specified offset
Types
-type block_max_entry() :: #{max_impact := float(), doc_start := non_neg_integer(), doc_end := non_neg_integer(), offset := non_neg_integer(), size := non_neg_integer()}.
-type bm25_file() :: #bm25_file{path :: binary(), meta_fd :: file:fd() | undefined, postings_fd :: file:fd() | undefined, blockmax_fd :: file:fd() | undefined, blockmax_mmap :: term() | undefined, header :: map(), crypto :: undefined | #{key := binary(), postings_nonce := binary(), key_check := binary()}}.
-type posting() :: {DocIntId :: non_neg_integer(), TF :: pos_integer()}.
Functions
-spec close(bm25_file()) -> ok.
Close file handles
Create a new BM25 disk index
Decode a posting block Returns list of {DocIntId, TF}
Encode a block of postings using delta encoding + varint Input: [{DocIntId, TF}, ...] sorted by DocIntId ascending Output: Binary with format: [Count:varint][Delta1:varint][TF1:varint]...
Get the base path
Check if file has mmap handle
-spec lookup_blocks_for_term(bm25_file(), non_neg_integer()) -> {ok, [block_max_entry()]} | {error, not_found}.
Lookup blocks for a specific term
-spec mmap_close(term()) -> ok.
Close mmap handle
Open a file with mmap for zero-copy reads
-spec mmap_read(term(), non_neg_integer(), non_neg_integer()) -> {ok, binary()} | {error, term()}.
Read from mmap
Open an existing BM25 disk index
Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.
Pad binary to sector alignment
-spec read_block(bm25_file(), non_neg_integer()) -> {ok, binary()} | {error, term()}.
Read a block from the specified offset
-spec read_blockmax_index(bm25_file()) -> {ok, #{non_neg_integer() => [block_max_entry()]}} | {error, term()}.
Read block-max index from file Returns map of TermIntId => [BlockMaxEntry]
-spec read_blocks(bm25_file(), [{non_neg_integer(), non_neg_integer()}]) -> {ok, [binary()]} | {error, term()}.
Read multiple blocks (batch read)
Read index header (returns cached header)
-spec read_postings(bm25_file(), non_neg_integer(), non_neg_integer()) -> {ok, [posting()]} | {error, term()}.
Read postings from specified offset and size
Fresh postings nonce for a full rewrite (compaction): posting offsets are reused across compactions, so the static-mode nonce must rotate before the rewrite. Persisted by the next header write.
-spec sync(bm25_file()) -> ok.
Sync all file handles to disk
Update statistics in header
-spec varint_decode(binary()) -> {non_neg_integer(), binary()} | {error, incomplete}.
Decode a varint from the beginning of a binary Returns {Value, Rest} where Rest is the remaining binary
-spec varint_decode_list(binary(), non_neg_integer()) -> {[non_neg_integer()], binary()}.
Decode multiple varints from a binary Returns list of values
-spec varint_encode(non_neg_integer()) -> binary().
Encode an unsigned integer as a varint (7-bit encoding) Each byte uses 7 bits for data and 1 bit as continuation flag
-spec varint_encode_list([non_neg_integer()]) -> binary().
Encode a list of integers as varints
-spec write_block(bm25_file(), non_neg_integer(), binary()) -> ok | {error, term()}.
Write a block at the specified offset (sector-aligned)
-spec write_blockmax_index(bm25_file(), #{non_neg_integer() => [block_max_entry()]}) -> {ok, bm25_file()} | {error, term()}.
Write block-max index to file Format: [TermCount:32][Entry1][Entry2]... Entry: [TermIntId:32][BlockCount:16][Block1]...[BlockN] Block: [MaxImpact:32/float][DocStart:32][DocEnd:32][Offset:64][Size:32]
Write index header
-spec write_postings(bm25_file(), non_neg_integer(), [posting()]) -> {ok, non_neg_integer()} | {error, term()}.
Write postings for a term at specified offset