barrel_ngram_shard (barrel_ngram v0.7.1)
View SourcePer-corpus shard: changes-feed subscriber, segment writer, and compaction coordinator.
The shard keeps a corpus in sync with its barrel_docdb database. It subscribes to the changes feed in push mode and applies each batch to an in-memory buffer keyed by document id (an update replaces, a delete becomes a tombstone). When the buffer passes a threshold it freezes to a new immutable segment and commits the manifest, advancing the persisted watermark.
Segments only ever accumulate on their own, so when the live count crosses a threshold the shard compacts: an offloaded worker merges the segments, collapsing each key to its newest version by HLC and dropping superseded and deleted ordinals, and the shard swaps the manifest to the merged segment. compact/1 does this synchronously.
Recovery is the watermark: on start the shard loads the manifest and resubscribes from its watermark, so only the feed tail is replayed (idempotently). Correctness of updates/deletes never depends on compaction, the query confirm pass re-fetches the current document and drops a stale or deleted candidate.
Summary
Functions
Ids currently buffered (not yet frozen into a segment).
Synchronously compact every live segment into one, evicting superseded and deleted ordinals. Returns {error, busy} if a background compaction is in flight.
The corpus config held by the shard.
The live segments as {Gen, Path}, ascending by generation.
Synchronously drain the feed up to now and freeze the buffer. The deterministic catch-up point for tests and ops.
The live segments and the buffered ids in one atomic read, so a query never straddles a freeze (which could move a doc out of the buffer into a segment the query did not see).
Functions
Ids currently buffered (not yet frozen into a segment).
Synchronously compact every live segment into one, evicting superseded and deleted ordinals. Returns {error, busy} if a background compaction is in flight.
The corpus config held by the shard.
-spec get_manifest(term()) -> {ok, [{non_neg_integer(), binary()}]}.
The live segments as {Gen, Path}, ascending by generation.
Synchronously drain the feed up to now and freeze the buffer. The deterministic catch-up point for tests and ops.
-spec snapshot(term()) -> {ok, [{non_neg_integer(), binary()}], [binary()]}.
The live segments and the buffered ids in one atomic read, so a query never straddles a freeze (which could move a doc out of the buffer into a segment the query did not see).
-spec start_link(barrel_ngram_shards:ref(), map()) -> {ok, pid()} | {error, term()}.