barrel_ngram_manifest (barrel_ngram v0.7.1)
View SourcePer-corpus segment manifest.
The manifest is the single source of truth for which segments are live and how far the index has consumed the changes feed. It lists the live segment-<gen>.ngseg files, the next generation number, and the applied HLC watermark (12-byte encoded, or the first sentinel).
It is written atomically (temp file + file:rename), so the rename is the commit point: a crash between writing a new segment and committing the manifest leaves an orphan segment that the reader never sees and that cleanup_orphans/2 removes at startup. The tail since the committed watermark is replayed from the feed.
Summary
Functions
Append a segment and advance the generation counter.
Delete segment-*.ngseg files (and stray *.tmp) in Dir that the manifest does not list. Call at startup to clear orphans left by a crash before the manifest commit.
An empty manifest for a fresh corpus.
The live segments, ascending by generation.
Load the manifest from a corpus directory. A missing manifest is an empty corpus.
The next generation number to assign.
Drop the segments whose file name is in Files (merge inputs).
Write the manifest atomically (temp + rename).
Set the applied watermark.
The applied HLC watermark (12-byte encoded, or first).
Types
-type manifest() :: #{version := pos_integer(), watermark := binary() | first, next_gen := non_neg_integer(), segments := [segment()]}.
-type segment() :: #{gen := non_neg_integer(), file := binary(), doc_count := non_neg_integer()}.
Functions
Append a segment and advance the generation counter.
-spec cleanup_orphans(file:name_all(), manifest()) -> ok.
Delete segment-*.ngseg files (and stray *.tmp) in Dir that the manifest does not list. Call at startup to clear orphans left by a crash before the manifest commit.
-spec empty() -> manifest().
An empty manifest for a fresh corpus.
The live segments, ascending by generation.
-spec load(file:name_all()) -> {ok, manifest()} | {error, term()}.
Load the manifest from a corpus directory. A missing manifest is an empty corpus.
-spec next_gen(manifest()) -> non_neg_integer().
The next generation number to assign.
Drop the segments whose file name is in Files (merge inputs).
-spec save(file:name_all(), manifest()) -> ok | {error, term()}.
Write the manifest atomically (temp + rename).
Set the applied watermark.
The applied HLC watermark (12-byte encoded, or first).