barrel_ngram_shards (barrel_ngram v0.7.1)

View Source

Rendezvous (HRW) sharding for a corpus.

Maps a document key to exactly one of N shards, so each shard indexes only its slice and a document's entry never moves (the key is stable). Also holds the per-corpus metadata (shard count + config) the query planner needs before it can address the shards.

A shard is identified by a Ref: the corpus name when N =:= 1 (unchanged single-shard path) and {Corpus, I} for N > 1.

Summary

Types

ref/0

Functions

erase_meta(Corpus)
get_meta(Corpus)
put_meta(Corpus, Meta)
refs(Corpus, N)

The shard refs for a corpus of N shards, index order.

shard_for(Key, N)

The shard index owning Key out of N shards: the index with the highest phash2({I, Key}), tie-broken by the lowest index.

Types

ref/0

-type ref() :: term().

Functions

erase_meta(Corpus)

-spec erase_meta(term()) -> ok.

get_meta(Corpus)

-spec get_meta(term()) -> {ok, map()} | undefined.

put_meta(Corpus, Meta)

-spec put_meta(term(), map()) -> ok.

refs(Corpus, N)

-spec refs(term(), pos_integer()) -> [ref()].

The shard refs for a corpus of N shards, index order.

shard_for(Key, N)

-spec shard_for(binary(), pos_integer()) -> non_neg_integer().

The shard index owning Key out of N shards: the index with the highest phash2({I, Key}), tie-broken by the lowest index.