barrel_store_keys (barrel_docdb v1.1.1)
View SourceKey encoding for barrel_docdb storage
Provides functions to encode and decode keys for RocksDB storage. Keys are prefixed to enable efficient range scans.
Summary
Functions
Attachment data key
Prefix for document attachments
Change bucket key for time-bucketed change hints. BucketTs is typically erlang:system_time(second) div 60 (minute granularity). Value stores {min_hlc, max_hlc, count} for quick "has changes?" checks.
End marker for change bucket range scan.
Prefix for all change buckets in a database.
End marker for a channel feed range scan.
Key for a channel feed row at a change-sequence HLC. One row per doc per channel: rewrites delete the previous HLC key.
Start key for scanning one channel's feed.
Deleted documents count key
Documents count key
Last HLC timestamp key
General database metadata key
Database UID key
Extract {Channel, Hlc} from a channel feed key.
The doc id inside a TTL index key.
Extract doc_id from a doc_info key
Extract DocId from a doc_info key
Extract the encoded version from a version-chain key.
Extract the change HLC from a history key.
Decode binary to HLC timestamp
Extract HLC from a doc_hlc key
Decode an outbox key to extract tag and HLC.
Decode a path from binary.
Decode path_hlc key to extract topic and HLC. Returns {Topic, Hlc} tuple.
Decode a view key
Document body key for current revision (no revision in key). This enables direct body fetch without knowing the revision.
Document body key for a specific (non-current) revision. Used to store old revision bodies when updating a document.
Document current state key (stores {rev, deleted, hlc})
End marker for doc_current range scan
Prefix for all doc_current keys in a database
Document entity key for wide-column storage. Stores all document metadata as named columns: - rev: current revision ID - deleted: "true" or "false" - hlc: 12-byte encoded HLC timestamp - revtree: term_to_binary encoded revision tree
End marker for doc_entity range scan
Prefix for all doc_entity keys in a database
Doc TTL index row: sorted by expiry instant so the sweeper folds exactly the expired range.
Prefix of all TTL index rows of a database.
Exclusive upper bound covering every row expiring at or before NowMs (the doc id tail sorts above the bare instant, so bound at NowMs + 1).
Document HLC key (for changes feed with HLC ordering) HLC timestamps are 12 bytes (8 wall_time + 4 logical)
End marker for HLC range scan
Prefix for all HLC keys
Document info key (stores doc_info record)
End marker for doc_info range scan
Prefix for all doc_info keys in a database
Reverse index key: doc_id -> list of indexed paths. Used to remove old paths when updating a document.
Prefix for doc_paths keys.
Document revision key (stores document body)
Prefix for all revisions of a document
Document revision tree key (stores revtree as term_to_binary)
Key for one non-current version of a document. Key format: prefix | db_name | doc_id | ':' | version (storage encoding, HLC-first, so the chain scans in causal order).
End marker for a version chain scan.
Start key for scanning a document's version chain.
Extract bucket (first 2 bytes) from DocId. Short DocIds are padded with zeros.
Encode HLC timestamp to binary (big-endian for sort order) Uses barrel_hlc:encode/1 which produces 12 bytes
Encode a path for lexicographic ordering. Path components are encoded with length prefix and type tags. This ensures correct sort order across different types.
Encode topic for null-terminated storage. Topics are MQTT-style paths like "users/123/name"
Encode a view key for sorted storage Uses term_to_binary with ordered encoding to preserve Erlang term ordering
End marker for a history range scan.
Key for one history entry. The change-sequence HLC is issued locally and strictly increasing per database, so keys never collide and scan in write order.
Start key for scanning a database's history.
Convert HLC to time bucket number.
Local document key (not replicated) - legacy format in default CF
Local document key for local_cf (per-database) Key format: DbName + NUL separator + DocId
Prefix for all local docs in a database Use with local_fold to enumerate all local docs for a database.
End marker for the whole-database outbox range.
Start key for scanning ALL outbox entries of a database (every tag), for maintenance scans (timeline rewind).
End marker for an outbox tag range scan.
Outbox key for a tagged entry. Key format: prefix | db_name | tag (null-terminated) | hlc The tag is an opaque binary chosen by the producer (e.g. <<"embed">>); the HLC gives time-ordered scans per tag and exact-key acks.
Start key for scanning all outbox entries under a tag.
Parse a key to determine its type and extract components. Used by compaction filter to identify doc_entity keys.
Path bitmap key for bitmap index. Stores a bitmap of document positions matching a specific path+value.
Path-HLC key for indexing changes by topic path. Key format: prefix | db_name | topic (null-terminated) | hlc Topic is an MQTT-style path like "users/123/name"
End marker for path_hlc range scan. Use with path_hlc_prefix or path_hlc for bounded range scans.
Prefix for scanning path_hlc entries for a specific topic. Returns all changes under this topic since the beginning of time.
End key for wildcard topic prefix matching (# patterns). Use with path_hlc_wildcard_start for bounded range scans matching all topics that start with the given prefix.
Start key for wildcard topic prefix matching (# patterns). Unlike path_hlc_prefix, this does NOT include the null terminator, allowing the range to capture all topics that START with the prefix. Example: prefix "users" matches topics "users", "users/123", "users/abc/def", etc.
End marker for path index range scan.
Path index key for a document path. Key format: prefix | db_name | encoded_path | docid Path includes the value at the end: [field1, field2, value]
Prefix for scanning path index entries. Can be used with partial paths for prefix scans.
End marker for posting list range scan.
Posting list key for path index. Key format: prefix | db_name | encoded_path (NO DocId - DocIds are in value) Path includes the value at the end: [field1, field2, value]
Prefix for scanning posting list entries. Can be used with partial paths for prefix scans.
Path stats key for cardinality counter. Stores the count of documents matching a specific path+value.
End key for prefix changes range scan. Creates an upper bound key that's lexicographically after all bucket keys for this prefix.
Create prefix changes key for a specific bucket. Key format: PREFIX_CHANGES | db_name | prefix | 0x00 | bucket (4 bytes BE)
Start key for prefix changes range scan. Used to scan from a specific bucket onwards.
System (global) document key for local_cf Key format: "_system" + NUL separator + DocId
Prefix for all system docs Use with local_fold to enumerate all system docs.
Truncate a value to max 128 bytes for value-first index. Only applies to binary values; other types are unchanged.
End marker for value-first index range scan.
Value-first index key for iterable equality queries. Format: [value_prefix, path, DocId] enables prefix scan with early termination. Unlike value_posting_key which stores DocIds in a posting list, this stores one key per DocId allowing iteration without full deserialization.
Prefix for scanning value-first index by value and path. Use this to find all DocIds matching a specific (path, value) pair.
End marker for bucketed posting list range scan.
Bucketed posting list key for sorted iteration. Format: [value_prefix, path, bucket] where bucket is first 2 bytes of DocId. This splits posting lists by DocId prefix, enabling: - Sorted iteration (buckets are in lexicographic order) - Smaller chunks (faster decode per bucket) - Early termination in intersection
Prefix for scanning bucketed posting lists for a path+value. Use this to iterate all buckets in sorted order.
End marker for value-first posting list range scan.
Value-first posting list key for fast equality queries. Key format: prefix | db_name | value_prefix | encoded_path Value is truncated to 128 bytes max for efficient prefix scans. Path is the field path WITHOUT the value (e.g., [<<"type">>] for type=user)
Prefix for scanning value-first posting lists by value. Use this to find all paths with a specific value.
View by docid key (tracks which index entries belong to each doc)
End marker for view by docid range scan
Prefix for view by docid entries
View index entry key
End marker for view index range scan
Prefix for view index entries
View metadata key
View indexed sequence key
Types
-type att_info() :: #{name := binary(), content_type := binary(), length := non_neg_integer(), digest := binary(), chunked => boolean(), chunk_size => pos_integer(), chunk_count => pos_integer()}.
-type change() :: map().
-type db_name() :: binary().
-type docid() :: binary().
-type revid() :: binary().
-type seq() :: barrel_hlc:timestamp().
-type seq_string() :: binary().
-type view_name() :: binary().
Functions
Attachment data key
Prefix for document attachments
-spec change_bucket(db_name(), non_neg_integer()) -> binary().
Change bucket key for time-bucketed change hints. BucketTs is typically erlang:system_time(second) div 60 (minute granularity). Value stores {min_hlc, max_hlc, count} for quick "has changes?" checks.
End marker for change bucket range scan.
Prefix for all change buckets in a database.
End marker for a channel feed range scan.
-spec channel_key(db_name(), binary(), barrel_hlc:timestamp()) -> binary().
Key for a channel feed row at a change-sequence HLC. One row per doc per channel: rewrites delete the previous HLC key.
Start key for scanning one channel's feed.
Deleted documents count key
Documents count key
Last HLC timestamp key
General database metadata key
Database UID key
-spec decode_channel_key(db_name(), binary()) -> {binary(), barrel_hlc:timestamp()}.
Extract {Channel, Hlc} from a channel feed key.
-spec decode_doc_expiry_key(db_name(), binary()) -> {non_neg_integer(), binary()}.
The doc id inside a TTL index key.
Extract doc_id from a doc_info key
Extract DocId from a doc_info key
Extract the encoded version from a version-chain key.
-spec decode_history_key(db_name(), binary()) -> barrel_hlc:timestamp().
Extract the change HLC from a history key.
-spec decode_hlc(binary()) -> barrel_hlc:timestamp().
Decode binary to HLC timestamp
-spec decode_hlc_key(db_name(), binary()) -> barrel_hlc:timestamp().
Extract HLC from a doc_hlc key
-spec decode_outbox_key(db_name(), binary()) -> {binary(), barrel_hlc:timestamp()}.
Decode an outbox key to extract tag and HLC.
Decode a path from binary.
-spec decode_path_hlc_key(db_name(), binary()) -> {binary(), barrel_hlc:timestamp()}.
Decode path_hlc key to extract topic and HLC. Returns {Topic, Hlc} tuple.
Decode a view key
Document body key for current revision (no revision in key). This enables direct body fetch without knowing the revision.
Document body key for a specific (non-current) revision. Used to store old revision bodies when updating a document.
Document current state key (stores {rev, deleted, hlc})
End marker for doc_current range scan
Prefix for all doc_current keys in a database
Document entity key for wide-column storage. Stores all document metadata as named columns: - rev: current revision ID - deleted: "true" or "false" - hlc: 12-byte encoded HLC timestamp - revtree: term_to_binary encoded revision tree
End marker for doc_entity range scan
Prefix for all doc_entity keys in a database
-spec doc_expiry(db_name(), non_neg_integer(), binary()) -> binary().
Doc TTL index row: sorted by expiry instant so the sweeper folds exactly the expired range.
Prefix of all TTL index rows of a database.
-spec doc_expiry_upto(db_name(), non_neg_integer()) -> binary().
Exclusive upper bound covering every row expiring at or before NowMs (the doc id tail sorts above the bare instant, so bound at NowMs + 1).
-spec doc_hlc(db_name(), barrel_hlc:timestamp()) -> binary().
Document HLC key (for changes feed with HLC ordering) HLC timestamps are 12 bytes (8 wall_time + 4 logical)
End marker for HLC range scan
Prefix for all HLC keys
Document info key (stores doc_info record)
End marker for doc_info range scan
Prefix for all doc_info keys in a database
Reverse index key: doc_id -> list of indexed paths. Used to remove old paths when updating a document.
Prefix for doc_paths keys.
Document revision key (stores document body)
Prefix for all revisions of a document
Document revision tree key (stores revtree as term_to_binary)
Key for one non-current version of a document. Key format: prefix | db_name | doc_id | ':' | version (storage encoding, HLC-first, so the chain scans in causal order).
End marker for a version chain scan.
Start key for scanning a document's version chain.
Extract bucket (first 2 bytes) from DocId. Short DocIds are padded with zeros.
-spec encode_hlc(barrel_hlc:timestamp()) -> binary().
Encode HLC timestamp to binary (big-endian for sort order) Uses barrel_hlc:encode/1 which produces 12 bytes
Encode a path for lexicographic ordering. Path components are encoded with length prefix and type tags. This ensures correct sort order across different types.
Encode topic for null-terminated storage. Topics are MQTT-style paths like "users/123/name"
Encode a view key for sorted storage Uses term_to_binary with ordered encoding to preserve Erlang term ordering
End marker for a history range scan.
-spec history_key(db_name(), barrel_hlc:timestamp()) -> binary().
Key for one history entry. The change-sequence HLC is issued locally and strictly increasing per database, so keys never collide and scan in write order.
Start key for scanning a database's history.
-spec hlc_to_bucket(barrel_hlc:timestamp()) -> non_neg_integer().
Convert HLC to time bucket number.
Local document key (not replicated) - legacy format in default CF
Local document key for local_cf (per-database) Key format: DbName + NUL separator + DocId
Prefix for all local docs in a database Use with local_fold to enumerate all local docs for a database.
End marker for the whole-database outbox range.
Start key for scanning ALL outbox entries of a database (every tag), for maintenance scans (timeline rewind).
End marker for an outbox tag range scan.
-spec outbox_key(db_name(), binary(), barrel_hlc:timestamp()) -> binary().
Outbox key for a tagged entry. Key format: prefix | db_name | tag (null-terminated) | hlc The tag is an opaque binary chosen by the producer (e.g. <<"embed">>); the HLC gives time-ordered scans per tag and exact-key acks.
Start key for scanning all outbox entries under a tag.
-spec parse_key(binary()) -> {doc_entity, DbName :: binary(), DocId :: binary()} | {doc_body_rev, DbName :: binary(), DocId :: binary(), Rev :: binary()} | other.
Parse a key to determine its type and extract components. Used by compaction filter to identify doc_entity keys.
Path bitmap key for bitmap index. Stores a bitmap of document positions matching a specific path+value.
-spec path_hlc(db_name(), binary(), barrel_hlc:timestamp()) -> binary().
Path-HLC key for indexing changes by topic path. Key format: prefix | db_name | topic (null-terminated) | hlc Topic is an MQTT-style path like "users/123/name"
End marker for path_hlc range scan. Use with path_hlc_prefix or path_hlc for bounded range scans.
Prefix for scanning path_hlc entries for a specific topic. Returns all changes under this topic since the beginning of time.
End key for wildcard topic prefix matching (# patterns). Use with path_hlc_wildcard_start for bounded range scans matching all topics that start with the given prefix.
Start key for wildcard topic prefix matching (# patterns). Unlike path_hlc_prefix, this does NOT include the null terminator, allowing the range to capture all topics that START with the prefix. Example: prefix "users" matches topics "users", "users/123", "users/abc/def", etc.
End marker for path index range scan.
Path index key for a document path. Key format: prefix | db_name | encoded_path | docid Path includes the value at the end: [field1, field2, value]
Prefix for scanning path index entries. Can be used with partial paths for prefix scans.
End marker for posting list range scan.
Posting list key for path index. Key format: prefix | db_name | encoded_path (NO DocId - DocIds are in value) Path includes the value at the end: [field1, field2, value]
Prefix for scanning posting list entries. Can be used with partial paths for prefix scans.
Path stats key for cardinality counter. Stores the count of documents matching a specific path+value.
-spec prefix_changes_end(db_name(), binary(), non_neg_integer()) -> binary().
End key for prefix changes range scan. Creates an upper bound key that's lexicographically after all bucket keys for this prefix.
-spec prefix_changes_key(db_name(), binary(), non_neg_integer()) -> binary().
Create prefix changes key for a specific bucket. Key format: PREFIX_CHANGES | db_name | prefix | 0x00 | bucket (4 bytes BE)
-spec prefix_changes_start(db_name(), binary(), non_neg_integer()) -> binary().
Start key for prefix changes range scan. Used to scan from a specific bucket onwards.
System (global) document key for local_cf Key format: "_system" + NUL separator + DocId
-spec system_doc_prefix() -> binary().
Prefix for all system docs Use with local_fold to enumerate all system docs.
Truncate a value to max 128 bytes for value-first index. Only applies to binary values; other types are unchanged.
End marker for value-first index range scan.
Value-first index key for iterable equality queries. Format: [value_prefix, path, DocId] enables prefix scan with early termination. Unlike value_posting_key which stores DocIds in a posting list, this stores one key per DocId allowing iteration without full deserialization.
Prefix for scanning value-first index by value and path. Use this to find all DocIds matching a specific (path, value) pair.
End marker for bucketed posting list range scan.
Bucketed posting list key for sorted iteration. Format: [value_prefix, path, bucket] where bucket is first 2 bytes of DocId. This splits posting lists by DocId prefix, enabling: - Sorted iteration (buckets are in lexicographic order) - Smaller chunks (faster decode per bucket) - Early termination in intersection
Prefix for scanning bucketed posting lists for a path+value. Use this to iterate all buckets in sorted order.
End marker for value-first posting list range scan.
Value-first posting list key for fast equality queries. Key format: prefix | db_name | value_prefix | encoded_path Value is truncated to 128 bytes max for efficient prefix scans. Path is the field path WITHOUT the value (e.g., [<<"type">>] for type=user)
Prefix for scanning value-first posting lists by value. Use this to find all paths with a specific value.
View by docid key (tracks which index entries belong to each doc)
End marker for view by docid range scan
Prefix for view by docid entries
View index entry key
End marker for view index range scan
Prefix for view index entries
View metadata key
View indexed sequence key