barrel_ngram_shards (barrel_ngram v0.9.0)
View SourceRendezvous (HRW) sharding for a corpus.
Maps a document key to exactly one of N shards, so each shard indexes only its slice and a document's entry never moves (the key is stable). Also holds the per-corpus metadata (shard count + config) the query planner needs before it can address the shards.
A shard is identified by a Ref: the corpus name when N =:= 1 (unchanged single-shard path) and {Corpus, I} for N > 1.
Summary
Functions
Every corpus with a currently-active (query-trusted) meta entry. A single atomic point-in-time scan of the whole VM's persistent terms, filtered by this module's own key shape. Used by barrel_ngram_app to clear stale entries on application:stop/1 (persistent_term survives that, shard processes do not).
Every corpus with a currently-pending meta entry. See all_corpora/0 -- the pending-cache counterpart, swept alongside it by barrel_ngram_app:stop/1 so an interrupted open's entry does not leak across an application:stop/start cycle.
The shard refs for a corpus of N shards, index order.
The shard index owning Key out of N shards: the index with the highest phash2({I, Key}), tie-broken by the lowest index.
Types
-type ref() :: term().
Functions
-spec all_corpora() -> [term()].
Every corpus with a currently-active (query-trusted) meta entry. A single atomic point-in-time scan of the whole VM's persistent terms, filtered by this module's own key shape. Used by barrel_ngram_app to clear stale entries on application:stop/1 (persistent_term survives that, shard processes do not).
-spec all_pending_corpora() -> [term()].
Every corpus with a currently-pending meta entry. See all_corpora/0 -- the pending-cache counterpart, swept alongside it by barrel_ngram_app:stop/1 so an interrupted open's entry does not leak across an application:stop/start cycle.
-spec erase_meta(term()) -> ok.
-spec erase_pending_meta(term()) -> ok.
-spec refs(term(), pos_integer()) -> [ref()].
The shard refs for a corpus of N shards, index order.
-spec shard_for(binary(), pos_integer()) -> non_neg_integer().
The shard index owning Key out of N shards: the index with the highest phash2({I, Key}), tie-broken by the lowest index.