barrel_vectordb_diskann_file (barrel_vectordb v2.1.2)
View SourceDiskANN sector-aligned disk I/O
Handles persistent storage for DiskANN index with: - 4KB sector-aligned writes for optimal SSD performance - Separate files for graph, vectors, and metadata - Batch read operations for beam search
File layout: - diskann.meta: Erlang term metadata - diskann.graph: Vamana graph (4KB aligned nodes) - diskann.vectors: Full float32 vectors (4KB aligned) - diskann.pq: PQ codebooks + codes - diskann.idmap: ID to offset mapping
Summary
Functions
Close file handles
Create a new DiskANN index file set
The file set's crypto state, reduced to what companions (the pq_state term file) need: none or the key.
Get the base path
Check if mmap is available for zero-copy reads
Open an existing DiskANN index
Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.
Read index header (returns cached header)
Read header directly from graph file Supports both V1 (string entry point) and V2 (integer entry point) formats
Read a node from the graph file by offset
Read a node from the graph file by integer ID Returns {ok, {IntId, [NeighborIntIds]}} or {error, term()}
Batch read multiple nodes (for beam search)
Batch read multiple nodes by integer IDs Uses batch pread/2 for efficiency (single system call)
Read PQ codes for a range of vectors
Read a vector from the vector file
Read a vector using mmap (zero-copy) Falls back to regular pread if mmap is not available
Sync all file handles to disk
Write index header
Write a node to the graph file (sector-aligned) Returns {ok, Offset} where Offset is the byte offset
Write a node to the graph file using integer ID Node format: [IntId:64/little][NeighborCount:16/little][Neighbor1:64/little]...[NeighborR:64/little][padding] Each node is stored at offset: SECTOR_SIZE + (IntId * SECTOR_SIZE)
Write PQ codes for a batch of vectors. Slots are written once per index (immutable append): STATIC mode.
Write a vector to the vector file. A slot is written once per index (immutable append), which is what makes STATIC mode safe here.
Types
-type diskann_file() :: #diskann_file{path :: binary(), graph_fd :: file:fd() | undefined, vector_fd :: file:fd() | undefined, vector_mmap :: term() | undefined, pq_fd :: file:fd() | undefined, header :: map(), crypto :: undefined | #{key := binary(), vectors_nonce := binary(), pq_nonce := binary(), key_check := binary()}}.
Functions
-spec close(diskann_file()) -> ok.
Close file handles
-spec create(binary() | string(), map()) -> {ok, diskann_file()} | {error, term()}.
Create a new DiskANN index file set
-spec get_crypto(diskann_file()) -> none | #{key := binary()}.
The file set's crypto state, reduced to what companions (the pq_state term file) need: none or the key.
-spec get_path(diskann_file()) -> binary().
Get the base path
-spec has_mmap(diskann_file()) -> boolean().
Check if mmap is available for zero-copy reads
-spec open(binary() | string()) -> {ok, diskann_file()} | {error, term()}.
Open an existing DiskANN index
-spec open(binary() | string(), map()) -> {ok, diskann_file()} | {error, term()}.
Open with options: crypto => none | #{key := <<_:256>>}. The open fails closed on any encrypted/plaintext mismatch or wrong key.
-spec read_header(diskann_file()) -> map().
Read index header (returns cached header)
-spec read_header_from_file(diskann_file()) -> {ok, map()} | {error, term()}.
Read header directly from graph file Supports both V1 (string entry point) and V2 (integer entry point) formats
-spec read_node(diskann_file(), non_neg_integer()) -> {ok, {binary(), map()}} | {error, term()}.
Read a node from the graph file by offset
-spec read_node_by_int_id(diskann_file(), non_neg_integer(), pos_integer()) -> {ok, {non_neg_integer(), [non_neg_integer()]}} | {error, term()}.
Read a node from the graph file by integer ID Returns {ok, {IntId, [NeighborIntIds]}} or {error, term()}
-spec read_nodes_batch(diskann_file(), [non_neg_integer()]) -> [{ok, {binary(), map()}} | {error, term()}].
Batch read multiple nodes (for beam search)
-spec read_nodes_batch_int(diskann_file(), [non_neg_integer()], pos_integer()) -> [{ok, {non_neg_integer(), [non_neg_integer()]}} | {error, term()}].
Batch read multiple nodes by integer IDs Uses batch pread/2 for efficiency (single system call)
-spec read_pq_codes(diskann_file(), {non_neg_integer(), non_neg_integer()}) -> {ok, [binary()]} | {error, term()}.
Read PQ codes for a range of vectors
-spec read_vector(diskann_file(), non_neg_integer()) -> {ok, [float()]} | {error, term()}.
Read a vector from the vector file
-spec read_vector_mmap(diskann_file(), non_neg_integer()) -> {ok, [float()]} | {error, term()}.
Read a vector using mmap (zero-copy) Falls back to regular pread if mmap is not available
-spec sync(diskann_file()) -> ok.
Sync all file handles to disk
-spec write_header(diskann_file(), map()) -> {ok, diskann_file()}.
Write index header
-spec write_node(diskann_file(), binary(), map()) -> {ok, non_neg_integer(), diskann_file()} | {error, term()}.
Write a node to the graph file (sector-aligned) Returns {ok, Offset} where Offset is the byte offset
-spec write_node_int(diskann_file(), non_neg_integer(), [non_neg_integer()], pos_integer()) -> ok | {error, term()}.
Write a node to the graph file using integer ID Node format: [IntId:64/little][NeighborCount:16/little][Neighbor1:64/little]...[NeighborR:64/little][padding] Each node is stored at offset: SECTOR_SIZE + (IntId * SECTOR_SIZE)
-spec write_pq_codes(diskann_file(), non_neg_integer(), [binary()]) -> ok | {error, term()}.
Write PQ codes for a batch of vectors. Slots are written once per index (immutable append): STATIC mode.
-spec write_vector(diskann_file(), non_neg_integer(), [float()]) -> ok | {error, term()}.
Write a vector to the vector file. A slot is written once per index (immutable append), which is what makes STATIC mode safe here.