barrel_ngram_segment (barrel_ngram v0.7.1)
View SourceImmutable trigram segment file: write and read.
One segment is a self-contained, immutable index over a set of documents. Layout (little-endian, header sector-aligned):
[0, 4096) header (magic, offsets, doc_count, watermark)
[4096, +offset_table_len) offset table: u32 per gram, direct-addressed
[postings_off, +postings_len) postings region (byte 0 is a sentinel)
[sidecar_off, +sidecar_len) ordinal -> {key, hlc, deleted} sidecarThe offset table is direct-addressed: table[Gram] is the byte offset of that gram's posting block within the postings region, or 0 when the gram is absent. Byte 0 of the postings region is a reserved sentinel so a 0 entry is unambiguously "empty". The table spans only up to the highest gram present; a query for any higher gram reads past the table and is treated as empty. The table also serves as the gram directory for all_postings/1.
Each posting block is stored length-prefixed: [Len:32][block] where the block is a delta+varint list of ordinals (see barrel_ngram_postings).
The sidecar maps a local ordinal to its document key, the change HLC that produced it (the recency sequence number used when merging), and a deleted flag (a tombstone: a deleted key carries no grams). The index is [KeyOff:32][KeyLen:32][Deleted:8][Hlc:12] per ordinal, followed by the concatenated key bytes. It is read once at open; key bytes are read on demand with file:pread.
Reads use file:pread and leave caching to the OS/ZFS ARC. Handles carry a raw read fd owned by the opening process, so a query opens its own handle rather than sharing the shard's.
Summary
Functions
Every present {Gram, [Ordinal]} in the segment. Reads the offset table (the gram directory) sequentially, then each posting block. Used by the merger to rebuild per-ordinal grams.
Close a segment handle.
The posting-block codec of this segment.
Document count in the segment (includes tombstones).
Every ordinal as {Ordinal, Key, Hlc, Deleted}. Used by the merger.
Resolve ordinals to {Ordinal, Key} pairs (one batched pread). Out-of-range ordinals are dropped.
The raw (undecoded) posting block for a gram, or empty. Lets the query combine blocks natively (roaring) without decoding each to a list.
Posting list (ascending ordinals) for a gram, or empty.
Open a segment for reading. The returned handle owns a raw read fd; close it with close/1.
The high-watermark HLC (12-byte encoded) this segment covers.
Write an immutable segment to Path. Writes to a temp file and renames into place so a reader never sees a partial segment.
Types
-type codec() :: varint | roaring.
-opaque handle()
-type spec() :: #{doc_count := non_neg_integer(), watermark := binary(), postings := [{barrel_ngram_selector:gram(), [barrel_ngram_postings:ordinal()]}], entries := [entry()], codec => codec()}.
Functions
-spec all_postings(handle()) -> [{barrel_ngram_selector:gram(), [barrel_ngram_postings:ordinal()]}].
Every present {Gram, [Ordinal]} in the segment. Reads the offset table (the gram directory) sequentially, then each posting block. Used by the merger to rebuild per-ordinal grams.
-spec close(handle()) -> ok.
Close a segment handle.
The posting-block codec of this segment.
-spec doc_count(handle()) -> non_neg_integer().
Document count in the segment (includes tombstones).
-spec entries(handle()) -> [{barrel_ngram_postings:ordinal(), binary(), binary(), boolean()}].
Every ordinal as {Ordinal, Key, Hlc, Deleted}. Used by the merger.
-spec keys(handle(), [barrel_ngram_postings:ordinal()]) -> [{barrel_ngram_postings:ordinal(), binary()}].
Resolve ordinals to {Ordinal, Key} pairs (one batched pread). Out-of-range ordinals are dropped.
-spec lookup_block(handle(), barrel_ngram_selector:gram()) -> {ok, binary()} | empty | {error, term()}.
The raw (undecoded) posting block for a gram, or empty. Lets the query combine blocks natively (roaring) without decoding each to a list.
-spec lookup_postings(handle(), barrel_ngram_selector:gram()) -> {ok, [barrel_ngram_postings:ordinal()]} | empty | {error, term()}.
Posting list (ascending ordinals) for a gram, or empty.
-spec open(file:name_all()) -> {ok, handle()} | {error, term()}.
Open a segment for reading. The returned handle owns a raw read fd; close it with close/1.
The high-watermark HLC (12-byte encoded) this segment covers.
-spec write(file:name_all(), spec()) -> ok | {error, term()}.
Write an immutable segment to Path. Writes to a temp file and renames into place so a reader never sees a partial segment.