barrel_ngram_selector_sparse (barrel_ngram v0.7.1)
View SourceSparse (content-defined) gram selector.
Selects only a content-determined subset of trigrams, shrinking the index and the candidate sets. A gram at byte position i (the trigram over Bytes[i..i+2]) is kept iff a hash of its local window passes a sampling test: phash2(Window, SampleRate) =:= 0. The window has radius r, spanning Bytes[i-r .. i+2+r] (width 3 + 2r), so the choice depends only on local bytes and identical substrings select identical grams.
The boundary rule
When a literal L occurs in a document D at position p, a gram at literal-position i whose window lies entirely inside L (i >= r and i+2+r =< len(L)-1) has, in D, the identical window (that range sits inside the matched region), so it is selected in D iff selected in L. Grams whose window spills past L's edge see different bytes in D and are NOT reliable.
So select_grams/2 (index side) selects over every position, padding document-edge windows with a sentinel, while reliable_grams/2 (query side) selects only over INTERIOR positions whose window is fully inside the literal, and falls back to brute_force when there is no such gram. Relying on a boundary gram would be a silent false negative, so the query never does.
Summary
Functions
Sparse samples grams, so an arbitrary trigram may be absent; the regex planner must not rely on the index being complete.
The literal's grams the planner may intersect over: only interior positions whose window is fully inside the literal. brute_force when there is no such sampled gram.
Grams selected over the full byte string, with sentinel-padded windows at the edges. Used by the indexer.
Functions
Sparse samples grams, so an arbitrary trigram may be absent; the regex planner must not rely on the index being complete.
-spec reliable_grams(binary(), map()) -> barrel_ngram_selector:reliable().
The literal's grams the planner may intersect over: only interior positions whose window is fully inside the literal. brute_force when there is no such sampled gram.
-spec select_grams(binary(), map()) -> [barrel_ngram_selector:gram()].
Grams selected over the full byte string, with sentinel-padded windows at the edges. Used by the indexer.