barrel_ngram_segment (barrel_ngram v0.7.1)

View Source

Immutable trigram segment file: write and read.

One segment is a self-contained, immutable index over a set of documents. Layout (little-endian, header sector-aligned):

   [0, 4096)                      header (magic, offsets, doc_count, watermark)
   [4096, +offset_table_len)      offset table: u32 per gram, direct-addressed
   [postings_off, +postings_len)  postings region (byte 0 is a sentinel)
   [sidecar_off, +sidecar_len)    ordinal -> {key, hlc, deleted} sidecar

The offset table is direct-addressed: table[Gram] is the byte offset of that gram's posting block within the postings region, or 0 when the gram is absent. Byte 0 of the postings region is a reserved sentinel so a 0 entry is unambiguously "empty". The table spans only up to the highest gram present; a query for any higher gram reads past the table and is treated as empty. The table also serves as the gram directory for all_postings/1.

Each posting block is stored length-prefixed: [Len:32][block] where the block is a delta+varint list of ordinals (see barrel_ngram_postings).

The sidecar maps a local ordinal to its document key, the change HLC that produced it (the recency sequence number used when merging), and a deleted flag (a tombstone: a deleted key carries no grams). The index is [KeyOff:32][KeyLen:32][Deleted:8][Hlc:12] per ordinal, followed by the concatenated key bytes. It is read once at open; key bytes are read on demand with file:pread.

Reads use file:pread and leave caching to the OS/ZFS ARC. Handles carry a raw read fd owned by the opening process, so a query opens its own handle rather than sharing the shard's.

Summary

Functions

Every present {Gram, [Ordinal]} in the segment. Reads the offset table (the gram directory) sequentially, then each posting block. Used by the merger to rebuild per-ordinal grams.

Close a segment handle.

The posting-block codec of this segment.

Document count in the segment (includes tombstones).

Every ordinal as {Ordinal, Key, Hlc, Deleted}. Used by the merger.

Resolve ordinals to {Ordinal, Key} pairs (one batched pread). Out-of-range ordinals are dropped.

The raw (undecoded) posting block for a gram, or empty. Lets the query combine blocks natively (roaring) without decoding each to a list.

Posting list (ascending ordinals) for a gram, or empty.

Open a segment for reading. The returned handle owns a raw read fd; close it with close/1.

The high-watermark HLC (12-byte encoded) this segment covers.

Write an immutable segment to Path. Writes to a temp file and renames into place so a reader never sees a partial segment.

Types

codec/0

-type codec() :: varint | roaring.

entry/0

-type entry() :: #{key := binary(), hlc := binary(), deleted := boolean()}.

handle/0

-opaque handle()

spec/0

-type spec() ::
          #{doc_count := non_neg_integer(),
            watermark := binary(),
            postings := [{barrel_ngram_selector:gram(), [barrel_ngram_postings:ordinal()]}],
            entries := [entry()],
            codec => codec()}.

Functions

all_postings(Segment)

-spec all_postings(handle()) -> [{barrel_ngram_selector:gram(), [barrel_ngram_postings:ordinal()]}].

Every present {Gram, [Ordinal]} in the segment. Reads the offset table (the gram directory) sequentially, then each posting block. Used by the merger to rebuild per-ordinal grams.

close(Segment)

-spec close(handle()) -> ok.

Close a segment handle.

codec(Segment)

-spec codec(handle()) -> codec().

The posting-block codec of this segment.

doc_count(Segment)

-spec doc_count(handle()) -> non_neg_integer().

Document count in the segment (includes tombstones).

entries(Segment)

-spec entries(handle()) -> [{barrel_ngram_postings:ordinal(), binary(), binary(), boolean()}].

Every ordinal as {Ordinal, Key, Hlc, Deleted}. Used by the merger.

keys(Handle, Ordinals)

Resolve ordinals to {Ordinal, Key} pairs (one batched pread). Out-of-range ordinals are dropped.

lookup_block(Segment, Gram)

-spec lookup_block(handle(), barrel_ngram_selector:gram()) -> {ok, binary()} | empty | {error, term()}.

The raw (undecoded) posting block for a gram, or empty. Lets the query combine blocks natively (roaring) without decoding each to a list.

lookup_postings(Segment, Gram)

-spec lookup_postings(handle(), barrel_ngram_selector:gram()) ->
                         {ok, [barrel_ngram_postings:ordinal()]} | empty | {error, term()}.

Posting list (ascending ordinals) for a gram, or empty.

open(Path)

-spec open(file:name_all()) -> {ok, handle()} | {error, term()}.

Open a segment for reading. The returned handle owns a raw read fd; close it with close/1.

watermark(Segment)

-spec watermark(handle()) -> binary().

The high-watermark HLC (12-byte encoded) this segment covers.

write(Path, Spec)

-spec write(file:name_all(), spec()) -> ok | {error, term()}.

Write an immutable segment to Path. Writes to a temp file and renames into place so a reader never sees a partial segment.