barrel_ars_index (barrel_docdb v1.1.1)
View SourcePath index storage for automatic document indexing
Provides storage operations for the path index: - Index all paths extracted from a document - Update paths when a document changes - Remove all paths when a document is deleted - Query paths by prefix
Uses barrel_ars for path extraction and barrel_store_keys for encoding.
Summary
Functions
Get the value at a specific path for a DocId. Uses the reverse index (doc_paths) which stores all paths for a document. Returns {ok, Value} if found, not_found if path doesn't exist.
Check if a docid has a specific value at a path via point lookup. Uses the value-first index for O(1) verification instead of collecting the entire posting list. Uses key_exists which doesn't load the value.
Check if a DocId's value at a path satisfies a comparison. Op is one of: '>' | '<' | '>=' | '=<' | '=/=' | '==' Returns true if the condition is satisfied, false otherwise.
Filter a list of DocIds to only those that have a specific value at a path. Uses batch multi_get for efficiency instead of individual key lookups. Much faster than calling docid_has_value for each DocId individually.
Fold over DocIds matching a comparison operator (no path decoding). Optimized for pure range queries where only DocIds are needed. Callback receives (DocId, Acc) instead of ({Path, DocId}, Acc).
Fold over DocIds matching a comparison operator with snapshot. Same as fold_compare_docids but uses a snapshot for consistent reads.
Fold over path index entries matching a path prefix. The callback receives {Path, DocId} for each match. Uses posting lists internally - each posting list key contains multiple DocIds.
Fold over path index entries with explicit read profile. Uses posting lists: iterates posting keys and expands each to {Path, DocId} tuples.
Fold over path index entries in chunks for efficient batch processing. Collects DocIds into chunks of ChunkSize, calling Fun with each chunk. The callback can return: - {ok, Acc} to continue with next chunk - {ok, Acc, NewChunkSize} to adjust chunk size adaptively - {stop, Acc} to terminate early Returns the final accumulated value after processing all chunks. Uses posting lists internally.
Fold over path index entries in chunks with explicit read profile. Uses posting lists: iterates posting keys and chunks the DocIds.
Fold over path index entries in a key range. Lower-level function for range queries. Note: StartKey and EndKey should be posting key format.
Fold over path index entries in a key range with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
Fold over path index entries in reverse order. Iterates from last to first; useful for building sorted lists with prepend. Uses posting lists internally.
Fold over path index entries in reverse order with explicit read profile. Uses posting lists: iterates in reverse and expands to {Path, DocId} tuples.
Fold over all values for a path in ascending order. Useful for ORDER BY path ASC with early termination. The callback receives {FullPath, DocId} where FullPath includes the value. Uses posting lists internally.
Fold over all values for a path in ascending order with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
Fold over path values matching a comparison operator. Supports: '>' | '<' | '>=' | '=<' for range queries. Uses posting lists with proper range bounds. Example: fold_path_values_compare(S, Db, [<<"age">>],>', 50, Fun, Acc)' finds all docs where age > 50
Fold over all values for a path in descending order. Useful for ORDER BY path DESC with early termination. Iterates from highest value to lowest. Uses posting lists internally.
Fold over all values for a path in descending order with explicit read profile. Uses posting lists: iterates in reverse and expands to {Path, DocId} tuples.
Fold over posting lists matching a path prefix. The callback receives a list of DocIds from each posting list entry. Much more efficient than fold_path for exists queries since each key contains multiple DocIds instead of one. The callback can return: - {ok, Acc} to continue - {stop, Acc} to terminate early
Fold over posting lists with explicit read profile.
Fold over posting lists with snapshot for consistent reads. Same as fold_posting but uses a snapshot for read consistency.
Fold over path index entries matching a value prefix. Uses interval scan: [path, prefix] to [path, prefix ++ 0xFF] Much faster than collecting all values and filtering. Example: fold_prefix(S, Db, [<<"name">>], <<"John">>, Fun, Acc) matches: John, Johnny, Johnson, etc. Uses posting lists internally.
Fold over path index entries matching a value prefix with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
Fold over posting lists matching a value prefix. Like fold_prefix but returns raw DocId lists instead of expanding to tuples. Much more efficient for pure prefix queries with early termination. Uses prefix bloom optimization for faster key lookup. The callback receives (Key, DocIds, Acc) for each posting list. Example: fold_prefix_posting(S, Db, [<<"name">>], <<"John">>, Fun, Acc) matches: John, Johnny, Johnson, etc.
Fold over prefix posting lists with snapshot for consistent reads. Same as fold_prefix_posting but uses a snapshot for read consistency.
Fold over value-first index with early termination support. Iterates over individual keys (one per DocId) allowing bounded queries to stop after collecting enough results.
Fold over value-first index with snapshot for consistent reads.
Get DocIds from bucketed posting lists in sorted order. Iterates buckets (each ~20-100 DocIds) in sorted bucket order. Within each bucket, DocIds are merged and sorted. FullPath format: [field1, field2, value] where value is the last element.
Get stored paths for a document from the reverse index. Returns {ok, Paths} or not_found.
Get the cardinality (document count) for a path+value. Returns 0 if the path has never been indexed. Uses counter stored in path_stats_key (O(1) lookup).
Get exact cardinality from posting list using postings_count. This reads the posting list binary and counts keys directly. More accurate than counter (no drift), but requires reading posting list.
Get posting list for an exact path+value. This is O(1) lookup - no iteration needed. Returns list of DocIds or empty list if path not found.
Get posting list binary for an exact path+value. Returns the raw binary from storage without decoding. Use with barrel_postings:open/1 for repeated lookups.
Get posting list for an exact path+value, returning SORTED DocIds. V2 posting lists store keys in lexicographic order, so no sorting needed. Uses postings_keys/1 which returns already-sorted keys from V2 format. FullPath format: [field1, field2, value] where value is the last element.
Get posting list as a postings resource for fast repeated lookups. Parse once, lookup many times with O(1) or O(log n) performance. Returns an opaque postings resource for use with barrel_postings functions.
Index all paths from a document. Extracts paths using barrel_ars:analyze/1 and stores them. Also stores the reverse index (doc_id -> paths) for later updates. Updates posting lists for O(1) equality query lookups.
Return batch operations to index all paths from a document. Use this to combine with other operations in a single write_batch.
Intersect multiple posting list binaries using native roaring bitmap. Keys in V2 posting lists are pre-sorted in lexicographic order. Returns {ok, Postings} with the intersection result.
Check if a postings resource contains a DocId using bitmap lookup. O(1) hash-based lookup, may have rare false positives. Use posting_contains/2 for exact checks when false positives are unacceptable.
Check if a postings resource contains a DocId using exact lookup. O(log n) binary search on sorted keys.
Remove all paths for a document. Reads the reverse index to find all paths and deletes them.
Return batch operations to remove all paths for a document. Takes the stored paths (from reverse index) as parameter. Use this to combine with other operations in a single write_batch.
Update paths when a document changes. Computes the diff between old and new paths and applies changes.
Return batch operations to update paths when a document changes. Use this to combine with other operations in a single write_batch. Note: Does not include posting list updates - use update_doc for full updates.
Types
-type att_info() :: #{name := binary(), content_type := binary(), length := non_neg_integer(), digest := binary(), chunked => boolean(), chunk_size => pos_integer(), chunk_count => pos_integer()}.
-type change() :: map().
-type db_name() :: binary().
-type docid() :: binary().
-type read_profile() :: barrel_store_rocksdb:read_profile().
-type revid() :: binary().
-type seq() :: barrel_hlc:timestamp().
-type seq_string() :: binary().
-type store_ref() :: barrel_store_rocksdb:db_ref().
-type view_name() :: binary().
Functions
-spec docid_get_value(store_ref(), db_name(), docid(), [term()]) -> {ok, term()} | not_found | {error, term()}.
Get the value at a specific path for a DocId. Uses the reverse index (doc_paths) which stores all paths for a document. Returns {ok, Value} if found, not_found if path doesn't exist.
Check if a docid has a specific value at a path via point lookup. Uses the value-first index for O(1) verification instead of collecting the entire posting list. Uses key_exists which doesn't load the value.
-spec docid_satisfies_compare(store_ref(), db_name(), docid(), [term()], atom(), term()) -> boolean().
Check if a DocId's value at a path satisfies a comparison. Op is one of: '>' | '<' | '>=' | '=<' | '=/=' | '==' Returns true if the condition is satisfied, false otherwise.
Filter a list of DocIds to only those that have a specific value at a path. Uses batch multi_get for efficiency instead of individual key lookups. Much faster than calling docid_has_value for each DocId individually.
-spec fold_compare_docids(store_ref(), db_name(), [term()], '>' | '<' | '>=' | '=<', term(), fun(), term()) -> term().
Fold over DocIds matching a comparison operator (no path decoding). Optimized for pure range queries where only DocIds are needed. Callback receives (DocId, Acc) instead of ({Path, DocId}, Acc).
-spec fold_compare_docids_with_snapshot(store_ref(), db_name(), [term()], '>' | '<' | '>=' | '=<', term(), fun(), term(), barrel_store_rocksdb:snapshot()) -> term().
Fold over DocIds matching a comparison operator with snapshot. Same as fold_compare_docids but uses a snapshot for consistent reads.
Fold over path index entries matching a path prefix. The callback receives {Path, DocId} for each match. Uses posting lists internally - each posting list key contains multiple DocIds.
Fold over path index entries with explicit read profile. Uses posting lists: iterates posting keys and expands each to {Path, DocId} tuples.
-spec fold_path_chunked(store_ref(), db_name(), [term()], pos_integer(), fun(([docid()], Acc) -> {ok, Acc} | {ok, Acc, pos_integer()} | {stop, Acc}), Acc) -> Acc when Acc :: term().
Fold over path index entries in chunks for efficient batch processing. Collects DocIds into chunks of ChunkSize, calling Fun with each chunk. The callback can return: - {ok, Acc} to continue with next chunk - {ok, Acc, NewChunkSize} to adjust chunk size adaptively - {stop, Acc} to terminate early Returns the final accumulated value after processing all chunks. Uses posting lists internally.
-spec fold_path_chunked(store_ref(), db_name(), [term()], pos_integer(), fun(([docid()], Acc) -> {ok, Acc} | {ok, Acc, pos_integer()} | {stop, Acc}), Acc, read_profile()) -> Acc when Acc :: term().
Fold over path index entries in chunks with explicit read profile. Uses posting lists: iterates posting keys and chunks the DocIds.
Fold over path index entries in a key range. Lower-level function for range queries. Note: StartKey and EndKey should be posting key format.
-spec fold_path_range(store_ref(), db_name(), binary(), binary(), fun(), term(), read_profile()) -> term().
Fold over path index entries in a key range with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
Fold over path index entries in reverse order. Iterates from last to first; useful for building sorted lists with prepend. Uses posting lists internally.
Fold over path index entries in reverse order with explicit read profile. Uses posting lists: iterates in reverse and expands to {Path, DocId} tuples.
Fold over all values for a path in ascending order. Useful for ORDER BY path ASC with early termination. The callback receives {FullPath, DocId} where FullPath includes the value. Uses posting lists internally.
Fold over all values for a path in ascending order with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
-spec fold_path_values_compare(store_ref(), db_name(), [term()], '>' | '<' | '>=' | '=<', term(), fun(), term()) -> term().
Fold over path values matching a comparison operator. Supports: '>' | '<' | '>=' | '=<' for range queries. Uses posting lists with proper range bounds. Example: fold_path_values_compare(S, Db, [<<"age">>],>', 50, Fun, Acc)' finds all docs where age > 50
Fold over all values for a path in descending order. Useful for ORDER BY path DESC with early termination. Iterates from highest value to lowest. Uses posting lists internally.
-spec fold_path_values_reverse(store_ref(), db_name(), [term()], fun(), term(), read_profile()) -> term().
Fold over all values for a path in descending order with explicit read profile. Uses posting lists: iterates in reverse and expands to {Path, DocId} tuples.
Fold over posting lists matching a path prefix. The callback receives a list of DocIds from each posting list entry. Much more efficient than fold_path for exists queries since each key contains multiple DocIds instead of one. The callback can return: - {ok, Acc} to continue - {stop, Acc} to terminate early
Fold over posting lists with explicit read profile.
-spec fold_posting_with_snapshot(store_ref(), db_name(), [term()], fun(), term(), barrel_store_rocksdb:snapshot()) -> term().
Fold over posting lists with snapshot for consistent reads. Same as fold_posting but uses a snapshot for read consistency.
Fold over path index entries matching a value prefix. Uses interval scan: [path, prefix] to [path, prefix ++ 0xFF] Much faster than collecting all values and filtering. Example: fold_prefix(S, Db, [<<"name">>], <<"John">>, Fun, Acc) matches: John, Johnny, Johnson, etc. Uses posting lists internally.
-spec fold_prefix(store_ref(), db_name(), [term()], binary(), fun(), term(), read_profile()) -> term().
Fold over path index entries matching a value prefix with explicit read profile. Uses posting lists: iterates posting keys and expands to {Path, DocId} tuples.
Fold over posting lists matching a value prefix. Like fold_prefix but returns raw DocId lists instead of expanding to tuples. Much more efficient for pure prefix queries with early termination. Uses prefix bloom optimization for faster key lookup. The callback receives (Key, DocIds, Acc) for each posting list. Example: fold_prefix_posting(S, Db, [<<"name">>], <<"John">>, Fun, Acc) matches: John, Johnny, Johnson, etc.
-spec fold_prefix_posting_with_snapshot(store_ref(), db_name(), [term()], binary(), fun(), term(), barrel_store_rocksdb:snapshot()) -> term().
Fold over prefix posting lists with snapshot for consistent reads. Same as fold_prefix_posting but uses a snapshot for read consistency.
Fold over value-first index with early termination support. Iterates over individual keys (one per DocId) allowing bounded queries to stop after collecting enough results.
Fun receives (DocId, Acc) and should return: {ok, NewAcc} - continue iteration {stop, FinalAcc} - stop iteration early
Example: fold_value_index(S, Db, <<"user">>, [<<"type">>], Fun, Acc) finds all docs where type = "user"
-spec fold_value_index(store_ref(), db_name(), term(), [term()], fun(), term(), barrel_store_rocksdb:snapshot()) -> term().
Fold over value-first index with snapshot for consistent reads.
Get DocIds from bucketed posting lists in sorted order. Iterates buckets (each ~20-100 DocIds) in sorted bucket order. Within each bucket, DocIds are merged and sorted. FullPath format: [field1, field2, value] where value is the last element.
-spec get_doc_paths(store_ref(), db_name(), docid()) -> {ok, [{[term()], term()}]} | not_found | {error, term()}.
Get stored paths for a document from the reverse index. Returns {ok, Paths} or not_found.
-spec get_path_cardinality(store_ref(), db_name(), [term()]) -> {ok, non_neg_integer()} | {error, term()}.
Get the cardinality (document count) for a path+value. Returns 0 if the path has never been indexed. Uses counter stored in path_stats_key (O(1) lookup).
-spec get_posting_cardinality(store_ref(), db_name(), [term()]) -> {ok, non_neg_integer()} | not_found | {error, term()}.
Get exact cardinality from posting list using postings_count. This reads the posting list binary and counts keys directly. More accurate than counter (no drift), but requires reading posting list.
Get posting list for an exact path+value. This is O(1) lookup - no iteration needed. Returns list of DocIds or empty list if path not found.
-spec get_posting_list_binary(store_ref(), db_name(), [term()]) -> {ok, binary()} | not_found | {error, term()}.
Get posting list binary for an exact path+value. Returns the raw binary from storage without decoding. Use with barrel_postings:open/1 for repeated lookups.
Get posting list for an exact path+value, returning SORTED DocIds. V2 posting lists store keys in lexicographic order, so no sorting needed. Uses postings_keys/1 which returns already-sorted keys from V2 format. FullPath format: [field1, field2, value] where value is the last element.
-spec get_postings_resource(store_ref(), db_name(), [term()]) -> {ok, barrel_postings:postings()} | not_found | {error, term()}.
Get posting list as a postings resource for fast repeated lookups. Parse once, lookup many times with O(1) or O(log n) performance. Returns an opaque postings resource for use with barrel_postings functions.
Index all paths from a document. Extracts paths using barrel_ars:analyze/1 and stores them. Also stores the reverse index (doc_id -> paths) for later updates. Updates posting lists for O(1) equality query lookups.
Return batch operations to index all paths from a document. Use this to combine with other operations in a single write_batch.
-spec intersect_posting_lists([binary()]) -> {ok, barrel_postings:postings()} | {error, term()}.
Intersect multiple posting list binaries using native roaring bitmap. Keys in V2 posting lists are pre-sorted in lexicographic order. Returns {ok, Postings} with the intersection result.
-spec posting_bitmap_contains(barrel_postings:postings(), binary()) -> boolean().
Check if a postings resource contains a DocId using bitmap lookup. O(1) hash-based lookup, may have rare false positives. Use posting_contains/2 for exact checks when false positives are unacceptable.
-spec posting_contains(barrel_postings:postings(), binary()) -> boolean().
Check if a postings resource contains a DocId using exact lookup. O(log n) binary search on sorted keys.
Remove all paths for a document. Reads the reverse index to find all paths and deletes them.
-spec remove_doc_ops(db_name(), docid(), [{[term()], term()}]) -> [{posting_remove, binary(), binary()} | {delete, binary()} | {merge, binary(), integer()}].
Return batch operations to remove all paths for a document. Takes the stored paths (from reverse index) as parameter. Use this to combine with other operations in a single write_batch.
Update paths when a document changes. Computes the diff between old and new paths and applies changes.
-spec update_doc_ops(db_name(), docid(), doc(), doc()) -> [{put | delete, binary()} | {put, binary(), binary()}].
Return batch operations to update paths when a document changes. Use this to combine with other operations in a single write_batch. Note: Does not include posting list updates - use update_doc for full updates.