barrel_store_rocksdb (barrel_docdb v1.0.0)
View SourceRocksDB storage backend for barrel_docdb
Implements the barrel_store behaviour using RocksDB 2.4.0.
Summary
Functions
Add entity put to a batch Helper for building batch operations with entities Uses fixed binary format encoding for performance.
Delete a value from the body column family
Get a value from the body column family
Get a value from the body column family with a snapshot.
Get multiple values from the body column family (batch read) Uses short_range profile by default.
Get multiple values from body CF with explicit read profile. BlobDB has different I/O characteristics than LSM - use appropriate profile: - point: Small batches (<50 keys), cache friendly - short_range: Medium batches (50-200 keys), auto readahead - long_scan: Large batches (>200 keys), avoid cache pollution, explicit readahead
Get multiple values from body CF with snapshot for consistent reads. Uses short_range profile by default.
Get multiple values from body CF with snapshot and explicit read profile.
Put a value in the body column family
Put a value in the body column family with options
Hard-link snapshot of the whole database (all column families) into a new directory. RocksDB flushes the memtable itself; the target path must not exist. The checkpoint dir is a fully independent database.
Close the database
Trigger compaction on the default column family This runs the compaction filter which prunes old revisions. Flushes memtable first, then forces full compaction to ensure filter runs.
Decode entity from fixed binary format (v4).
Delete a key
Delete an entity
Encode entity columns to fixed binary format. Format v4 (version vectors; the rev-tree formats are gone, pre-1.0 format break): <<VerLen:16, Version/binary, Deleted:8, HlcLen:16, Hlc/binary, VVLen:32, VV/binary, CreatedAtLen:16, CreatedAt/binary, ExpiresAt:64, Tier:8, NConflicts:16, EmbExt/binary>> The embedding extension <<EmbLen:32, Emb/binary, EmbSrc:8>> is appended only when a vector is stored (src 0 = client, 1 = computed).
Fold over all keys with a given prefix
Fold over a key range with default read options
Fold over a key range with explicit read profile Profile controls RocksDB read options for different access patterns. Currently all profiles use defaults; infrastructure for future tuning.
Fold over a key range with auto-selected read profile based on limit. Automatically chooses appropriate RocksDB read options: - Limit =< 10: point profile (fill_cache=true) - Limit =< 100: short_range profile (fill_cache=true) - Limit > 100 or infinity: long_scan profile (fill_cache=false, readahead)
Fold over a key range optimized for long sequential scans Uses readahead_size for prefetching and fill_cache=false to avoid cache pollution. Best for scanning large datasets like full changes feed.
Fold over posting list entries in a range
Fold over posting list entries for compare/range queries with bloom filter. Uses total_order_seek=false to enable bloom filter optimization. Unlike prefix fold, does NOT use prefix_same_as_start since range queries may span multiple value prefixes (e.g., age > 50 spans 51, 52, 53...).
Fold over posting list entries for compare/range queries with snapshot. Same as fold_range_posting_compare but uses a snapshot for consistent reads.
Fold over posting list entries with prefix bloom optimization Uses prefix_same_as_start=true to stop at prefix boundary and total_order_seek=false to enable prefix bloom filter. Use this for prefix queries where all keys share a common prefix.
Fold over posting list entries with prefix bloom and snapshot
Fold over posting list entries in a range in reverse order
Fold over posting list entries with snapshot for consistent reads
Fold over a key range with prefix bloom optimization and snapshot Uses prefix_same_as_start=true to stop at prefix boundary and total_order_seek=false to enable prefix bloom filter. Use this for prefix queries where all keys share a common prefix (e.g., value_index scans).
Fold over a key range in reverse order with default options Useful for building sorted lists with prepend: [Item | Acc]
Fold over a key range in reverse order with explicit read profile
Fold over a key range with snapshot for consistent reads The fold function receives (Key, Value, Acc) and should return: {ok, NewAcc} - continue iteration {stop, FinalAcc} - stop iteration early
Get a value by key
Get estimated database size in bytes Uses RocksDB's estimate-live-data-size property as primary metric, falling back to total-sst-files-size if not available.
Get a wide-column entity from the default column family Returns {ok, [{Name, Value}]} or not_found Uses fast binary decoding instead of term_to_binary.
Get a wide-column entity with a snapshot for consistent reads.
Get a specific RocksDB property Common properties: - "rocksdb.estimate-live-data-size" - estimated size of live data - "rocksdb.total-sst-files-size" - total size of SST files - "rocksdb.estimate-num-keys" - estimated number of keys - "rocksdb.cur-size-all-mem-tables" - current size of all memtables
Get RocksDB statistics Returns all available statistics as a proplist.
Get with a snapshot for consistent reads
Check if a key exists. Uses rocksdb:get which leverages BloomFilter for fast negative lookups. Much faster than iterator seek for single key checks.
Delete a local document
Fold over local documents with a prefix Useful for listing all local docs for a database or all system docs.
Get a local document
Put a local document
Merge a value with the counter merge operator
Get multiple values by keys (batch read)
Get multiple wide-column entities (batch read) Uses fast binary decoding instead of term_to_binary.
Get multiple wide-column entities with a snapshot for consistent reads.
Multi-get with a snapshot for consistent reads
Check if multiple keys exist (batch existence check). Uses multi_get internally and converts to boolean results. Much faster than calling key_exists for each key individually.
Open a RocksDB database with column families Uses create_missing_column_families to auto-create any missing CFs on existing DBs.
Append a DocId to a posting list
Get a posting list from the posting column family Returns list of active DocIds (tombstoned entries are filtered)
Get raw posting list binary from the posting column family. Returns the binary without decoding, for use with postings_open/1.
Get multiple posting lists from the posting column family (batch read)
Remove a DocId from a posting list using tombstone marker
Put a key-value pair
Put a key-value pair with options
Put an entity in the default column family Columns is a list of {Name, Value} tuples where Name and Value are binaries. Uses fixed binary format encoding for performance.
Put an entity with options Uses fixed binary format encoding for performance.
Release a snapshot
Release a snapshot, swallowing any error. Use on cleanup paths where the handle may already be gone (process termination, cursor expiry) and crashing is undesirable.
Create a snapshot for consistent reads
Execute a batch of operations atomically (async by default)
Execute a batch of operations atomically with options Options: - sync: boolean() - if true, sync to disk before returning (default: false) Operations: - {put, Key, Value} - put to default CF - {delete, Key} - delete from default CF - {merge, Key, Delta} - merge counter in default CF - {posting_append, Key, DocId} - append DocId to posting list - {posting_remove, Key, DocId} - remove DocId from posting list - {body_put, Key, Value} - put to body CF (BlobDB) - {body_delete, Key} - delete from body CF - {entity_put, Key, Columns} - put wide-column entity to default CF - {entity_delete, Key} - delete entity from default CF - {local_put, Key, Value} - put to local CF (config/state) - {local_delete, Key} - delete from local CF
Types
-type db_ref() :: #{ref := rocksdb:db_handle(), path := string(), default_cf := rocksdb:cf_handle(), bitmap_cf := rocksdb:cf_handle(), posting_cf := rocksdb:cf_handle(), body_cf := rocksdb:cf_handle(), local_cf := rocksdb:cf_handle(), env => rocksdb:env_handle()}.
-type read_profile() :: point | short_range | long_scan.
-type snapshot() :: rocksdb:snapshot_handle().
Functions
-spec batch_put_entity(rocksdb:batch_handle(), binary(), [{binary(), binary()}]) -> ok.
Add entity put to a batch Helper for building batch operations with entities Uses fixed binary format encoding for performance.
Delete a value from the body column family
Get a value from the body column family
-spec body_get_with_snapshot(db_ref(), binary(), snapshot()) -> {ok, binary()} | not_found | {error, term()}.
Get a value from the body column family with a snapshot.
Get multiple values from the body column family (batch read) Uses short_range profile by default.
-spec body_multi_get(db_ref(), [binary()], read_profile()) -> [{ok, binary()} | not_found | {error, term()}].
Get multiple values from body CF with explicit read profile. BlobDB has different I/O characteristics than LSM - use appropriate profile: - point: Small batches (<50 keys), cache friendly - short_range: Medium batches (50-200 keys), auto readahead - long_scan: Large batches (>200 keys), avoid cache pollution, explicit readahead
-spec body_multi_get_with_snapshot(db_ref(), [binary()], snapshot()) -> [{ok, binary()} | not_found | {error, term()}].
Get multiple values from body CF with snapshot for consistent reads. Uses short_range profile by default.
-spec body_multi_get_with_snapshot(db_ref(), [binary()], snapshot(), read_profile()) -> [{ok, binary()} | not_found | {error, term()}].
Get multiple values from body CF with snapshot and explicit read profile.
Put a value in the body column family
Put a value in the body column family with options
Hard-link snapshot of the whole database (all column families) into a new directory. RocksDB flushes the memtable itself; the target path must not exist. The checkpoint dir is a fully independent database.
-spec close(db_ref()) -> ok.
Close the database
Trigger compaction on the default column family This runs the compaction filter which prunes old revisions. Flushes memtable first, then forces full compaction to ensure filter runs.
Decode entity from fixed binary format (v4).
Delete a key
Delete an entity
Encode entity columns to fixed binary format. Format v4 (version vectors; the rev-tree formats are gone, pre-1.0 format break): <<VerLen:16, Version/binary, Deleted:8, HlcLen:16, Hlc/binary, VVLen:32, VV/binary, CreatedAtLen:16, CreatedAt/binary, ExpiresAt:64, Tier:8, NConflicts:16, EmbExt/binary>> The embedding extension <<EmbLen:32, Emb/binary, EmbSrc:8>> is appended only when a vector is stored (src 0 = client, 1 = computed).
Fold over all keys with a given prefix
Fold over a key range with default read options
Fold over a key range with explicit read profile Profile controls RocksDB read options for different access patterns. Currently all profiles use defaults; infrastructure for future tuning.
-spec fold_range_limit(db_ref(), binary(), binary(), fun(), term(), non_neg_integer() | infinity) -> term().
Fold over a key range with auto-selected read profile based on limit. Automatically chooses appropriate RocksDB read options: - Limit =< 10: point profile (fill_cache=true) - Limit =< 100: short_range profile (fill_cache=true) - Limit > 100 or infinity: long_scan profile (fill_cache=false, readahead)
This is the recommended function for range queries with known limits.
Fold over a key range optimized for long sequential scans Uses readahead_size for prefetching and fill_cache=false to avoid cache pollution. Best for scanning large datasets like full changes feed.
Fold over posting list entries in a range
Fold over posting list entries for compare/range queries with bloom filter. Uses total_order_seek=false to enable bloom filter optimization. Unlike prefix fold, does NOT use prefix_same_as_start since range queries may span multiple value prefixes (e.g., age > 50 spans 51, 52, 53...).
-spec fold_range_posting_compare_with_snapshot(db_ref(), binary(), binary(), fun(), term(), snapshot()) -> term().
Fold over posting list entries for compare/range queries with snapshot. Same as fold_range_posting_compare but uses a snapshot for consistent reads.
Fold over posting list entries with prefix bloom optimization Uses prefix_same_as_start=true to stop at prefix boundary and total_order_seek=false to enable prefix bloom filter. Use this for prefix queries where all keys share a common prefix.
-spec fold_range_posting_prefix_with_snapshot(db_ref(), binary(), binary(), fun(), term(), snapshot()) -> term().
Fold over posting list entries with prefix bloom and snapshot
Fold over posting list entries in a range in reverse order
-spec fold_range_posting_with_snapshot(db_ref(), binary(), binary(), fun(), term(), snapshot()) -> term().
Fold over posting list entries with snapshot for consistent reads
-spec fold_range_prefix_with_snapshot(db_ref(), binary(), binary(), fun(), term(), snapshot()) -> term().
Fold over a key range with prefix bloom optimization and snapshot Uses prefix_same_as_start=true to stop at prefix boundary and total_order_seek=false to enable prefix bloom filter. Use this for prefix queries where all keys share a common prefix (e.g., value_index scans).
Fold over a key range in reverse order with default options Useful for building sorted lists with prepend: [Item | Acc]
Fold over a key range in reverse order with explicit read profile
Fold over a key range with snapshot for consistent reads The fold function receives (Key, Value, Acc) and should return: {ok, NewAcc} - continue iteration {stop, FinalAcc} - stop iteration early
Get a value by key
-spec get_db_size(db_ref()) -> {ok, non_neg_integer()} | {error, term()}.
Get estimated database size in bytes Uses RocksDB's estimate-live-data-size property as primary metric, falling back to total-sst-files-size if not available.
Get a wide-column entity from the default column family Returns {ok, [{Name, Value}]} or not_found Uses fast binary decoding instead of term_to_binary.
-spec get_entity_with_snapshot(db_ref(), binary(), snapshot()) -> {ok, [{binary(), binary()}]} | not_found | {error, term()}.
Get a wide-column entity with a snapshot for consistent reads.
Get a specific RocksDB property Common properties: - "rocksdb.estimate-live-data-size" - estimated size of live data - "rocksdb.total-sst-files-size" - total size of SST files - "rocksdb.estimate-num-keys" - estimated number of keys - "rocksdb.cur-size-all-mem-tables" - current size of all memtables
Get RocksDB statistics Returns all available statistics as a proplist.
-spec get_with_snapshot(db_ref(), binary(), snapshot()) -> {ok, binary()} | not_found | {error, term()}.
Get with a snapshot for consistent reads
Check if a key exists. Uses rocksdb:get which leverages BloomFilter for fast negative lookups. Much faster than iterator seek for single key checks.
Delete a local document
-spec local_fold(db_ref(), binary(), fun((binary(), binary(), term()) -> term()), term()) -> term().
Fold over local documents with a prefix Useful for listing all local docs for a database or all system docs.
Get a local document
Put a local document
Merge a value with the counter merge operator
Get multiple values by keys (batch read)
-spec multi_get_entity(db_ref(), [binary()]) -> [{ok, [{binary(), binary()}]} | not_found | {error, term()}].
Get multiple wide-column entities (batch read) Uses fast binary decoding instead of term_to_binary.
-spec multi_get_entity_with_snapshot(db_ref(), [binary()], snapshot()) -> [{ok, [{binary(), binary()}]} | not_found | {error, term()}].
Get multiple wide-column entities with a snapshot for consistent reads.
-spec multi_get_with_snapshot(db_ref(), [binary()], snapshot()) -> [{ok, binary()} | not_found | {error, term()}].
Multi-get with a snapshot for consistent reads
Check if multiple keys exist (batch existence check). Uses multi_get internally and converts to boolean results. Much faster than calling key_exists for each key individually.
Open a RocksDB database with column families Uses create_missing_column_families to auto-create any missing CFs on existing DBs.
Append a DocId to a posting list
Get a posting list from the posting column family Returns list of active DocIds (tombstoned entries are filtered)
Get raw posting list binary from the posting column family. Returns the binary without decoding, for use with postings_open/1.
Get multiple posting lists from the posting column family (batch read)
Remove a DocId from a posting list using tombstone marker
Put a key-value pair
Put a key-value pair with options
Put an entity in the default column family Columns is a list of {Name, Value} tuples where Name and Value are binaries. Uses fixed binary format encoding for performance.
Put an entity with options Uses fixed binary format encoding for performance.
-spec release_snapshot(snapshot()) -> ok.
Release a snapshot
-spec safe_release_snapshot(snapshot() | undefined) -> ok.
Release a snapshot, swallowing any error. Use on cleanup paths where the handle may already be gone (process termination, cursor expiry) and crashing is undesirable.
Create a snapshot for consistent reads
Execute a batch of operations atomically (async by default)
Execute a batch of operations atomically with options Options: - sync: boolean() - if true, sync to disk before returning (default: false) Operations: - {put, Key, Value} - put to default CF - {delete, Key} - delete from default CF - {merge, Key, Delta} - merge counter in default CF - {posting_append, Key, DocId} - append DocId to posting list - {posting_remove, Key, DocId} - remove DocId from posting list - {body_put, Key, Value} - put to body CF (BlobDB) - {body_delete, Key} - delete from body CF - {entity_put, Key, Columns} - put wide-column entity to default CF - {entity_delete, Key} - delete entity from default CF - {local_put, Key, Value} - put to local CF (config/state) - {local_delete, Key} - delete from local CF