barrel_ngram_merge (barrel_ngram v0.7.1)

View Source

Segment compaction: merge several immutable segments into one.

Collapses each document key to its newest occurrence by change HLC (the recency sequence number carried per ordinal), dropping superseded versions. A key whose newest occurrence is a tombstone is either retained as a tombstone (so an older, un-merged segment does not resurrect it) or dropped entirely when DropTombstones says no older segment can hold it (a full compaction).

Recency is by HLC, not by file/generation order: a merge gives its output a fresh generation while its content may be older than a segment that was not part of the merge, so generation order cannot be trusted. Correctness never depends on getting eviction exactly right, the query confirm pass drops any resurfaced stale or deleted candidate; this only governs how compactly things are stored.

Runs in a worker process; the caller renames the temp segment into place and swaps the manifest.

Summary

Functions

Merge InputPaths into one temp segment. Returns its path, the document count, and the max input watermark.

Functions

merge(InputPaths, DropTombstones)

-spec merge([file:name_all()], boolean()) ->
               {ok, binary(), non_neg_integer(), binary()} | {error, term()}.

Merge InputPaths into one temp segment. Returns its path, the document count, and the max input watermark.