barrel_ngram_shard (barrel_ngram v0.9.0)
View SourcePer-corpus shard: changes-feed subscriber, segment writer, and compaction coordinator.
The shard keeps a corpus in sync with its barrel_docdb database. It subscribes to the changes feed in push mode and applies each batch to an in-memory buffer keyed by document id (an update replaces, a delete becomes a tombstone). The buffer holds each live key's corpus TEXT, not pre-computed grams: gram selection (both phase-1 dense and phase-2 positional) happens once, at freeze time, from that text, rather than once per change -- a document updated several times before a freeze only ever has its final version's grams computed once, and the buffer is not searched directly (every buffered key is always a candidate), so nothing needs its grams before freeze. When the buffer passes a threshold it freezes to a new immutable segment and commits the manifest, advancing the persisted watermark.
Segments only ever accumulate on their own, so when the live count crosses a threshold the shard compacts: an offloaded worker merges the segments, collapsing each key to its newest version by HLC and dropping superseded and deleted ordinals, and the shard swaps the manifest to the merged segment. compact/1 does this synchronously.
Recovery is the watermark: on start the shard loads the manifest and resubscribes from its watermark, so only the feed tail is replayed (idempotently). Correctness of updates/deletes never depends on compaction, the query confirm pass re-fetches the current document and drops a stale or deleted candidate.
Summary
Functions
Ids currently buffered (not yet frozen into a segment).
Synchronously compact every live segment into one, evicting superseded and deleted ordinals. Returns {error, busy} if a background compaction is in flight.
The corpus config held by the shard.
The live segments as {Gen, Path}, ascending by generation.
Synchronously drain the feed up to now and freeze the buffer. The deterministic catch-up point for tests and ops.
The live segments and an immutable copy of the buffer in one atomic read, so a query never straddles a freeze (which could move a doc out of the buffer into a segment the query did not see). The buffer snapshot carries each key's change HLC and whether it is live or a tombstone -- not just the key -- because the query layer's buffer/segment precedence rule needs to tell a live buffered update from a buffered delete (see barrel_ngram_query's confirm pass).
Functions
Ids currently buffered (not yet frozen into a segment).
Synchronously compact every live segment into one, evicting superseded and deleted ordinals. Returns {error, busy} if a background compaction is in flight.
The corpus config held by the shard.
-spec get_manifest(term()) -> {ok, [{non_neg_integer(), binary()}]}.
The live segments as {Gen, Path}, ascending by generation.
Synchronously drain the feed up to now and freeze the buffer. The deterministic catch-up point for tests and ops.
-spec snapshot(term()) -> {ok, [{non_neg_integer(), binary()}], #{binary() => {binary(), live | deleted}}}.
The live segments and an immutable copy of the buffer in one atomic read, so a query never straddles a freeze (which could move a doc out of the buffer into a segment the query did not see). The buffer snapshot carries each key's change HLC and whether it is live or a tombstone -- not just the key -- because the query layer's buffer/segment precedence rule needs to tell a live buffered update from a buffered delete (see barrel_ngram_query's confirm pass).
-spec start_link(barrel_ngram_shards:ref(), map()) -> {ok, pid()} | {error, term()}.