barrel_ngram_manifest (barrel_ngram v0.9.0)
View SourcePer-corpus segment manifest.
The manifest is the single source of truth for which segments are live and how far the index has consumed the changes feed. It lists the live segment-<gen>.ngseg files, the next generation number, and the applied HLC watermark (12-byte encoded, or the first sentinel). It also carries the corpus's index-critical config (phase2_selector_opts, fields): reopening with a different value would silently desync the query planner from what was actually indexed, so reconcile_config/2 rejects the mismatch instead.
It is written atomically (temp file + file:rename), so the rename is the commit point: a crash between writing a new segment and committing the manifest leaves an orphan segment that the reader never sees and that cleanup_orphans/2 removes at startup. The tail since the committed watermark is replayed from the feed.
Summary
Functions
Append a segment and advance the generation counter.
Delete segment-*.ngseg files (and stray *.tmp) in Dir that the manifest does not list. Call at startup to clear orphans left by a crash before the manifest commit.
The persisted corpus config, or undefined for a corpus never reconciled (a fresh manifest before its first reconcile_config/2).
An empty manifest for a fresh corpus. config is undefined until reconcile_config/2 persists the first requested config.
The live segments, ascending by generation.
Load the manifest from a corpus directory. A missing manifest is an empty corpus. A manifest written by a different (older or newer) version is rejected -- there is no migration path, a version bump means reindex.
The next generation number to assign.
Reconcile Requested (the caller's, already-defaulted, open/2 config) against the manifest's persisted config. A never-yet-persisted manifest (config =:= undefined, a fresh corpus) adopts Requested as what gets persisted at the next save/2. A manifest that already has a persisted config must match Requested exactly in every index-critical field, or the corpus was indexed under different assumptions than this open is making -- rejected rather than silently reindexed or silently queried under the wrong assumption.
Drop the segments whose file name is in Files (merge inputs).
Write the manifest atomically (temp + rename).
Set the applied watermark.
The applied HLC watermark (12-byte encoded, or first).
Types
-type manifest() :: #{version := pos_integer(), watermark := binary() | first, next_gen := non_neg_integer(), segments := [segment()], config := config() | undefined}.
-type segment() :: #{gen := non_neg_integer(), file := binary(), doc_count := non_neg_integer()}.
Functions
Append a segment and advance the generation counter.
-spec cleanup_orphans(file:name_all(), manifest()) -> ok.
Delete segment-*.ngseg files (and stray *.tmp) in Dir that the manifest does not list. Call at startup to clear orphans left by a crash before the manifest commit.
The persisted corpus config, or undefined for a corpus never reconciled (a fresh manifest before its first reconcile_config/2).
-spec empty() -> manifest().
An empty manifest for a fresh corpus. config is undefined until reconcile_config/2 persists the first requested config.
The live segments, ascending by generation.
-spec load(file:name_all()) -> {ok, manifest()} | {error, term()}.
Load the manifest from a corpus directory. A missing manifest is an empty corpus. A manifest written by a different (older or newer) version is rejected -- there is no migration path, a version bump means reindex.
-spec next_gen(manifest()) -> non_neg_integer().
The next generation number to assign.
-spec reconcile_config(manifest(), config()) -> {ok, manifest()} | {error, {config_mismatch, atom(), term(), term()}}.
Reconcile Requested (the caller's, already-defaulted, open/2 config) against the manifest's persisted config. A never-yet-persisted manifest (config =:= undefined, a fresh corpus) adopts Requested as what gets persisted at the next save/2. A manifest that already has a persisted config must match Requested exactly in every index-critical field, or the corpus was indexed under different assumptions than this open is making -- rejected rather than silently reindexed or silently queried under the wrong assumption.
Drop the segments whose file name is in Files (merge inputs).
-spec save(file:name_all(), manifest()) -> ok | {error, term()}.
Write the manifest atomically (temp + rename).
Set the applied watermark.
The applied HLC watermark (12-byte encoded, or first).