barrel_ngram_selector behaviour (barrel_ngram v0.9.0)

View Source

Gram-selection behaviour: the shared seam between the indexer and the query planner.

A selector maps a byte string to the set of trigrams it contributes to the index. The SAME selector is applied by the indexer (over full document bytes) and by the query planner (over the query literal), so that the grams a literal produces are always a subset of the grams its containing documents produced. That subset relationship is what makes the trigram intersection a correct necessary-condition filter.

Two callbacks, each taking the bytes and a selector-options map:

  • select_grams/2 - the grams a byte string contributes to the index. Used at index time.
  • reliable_grams/2 - the grams of a query literal the planner may safely intersect over, or brute_force when it may not (too short, or every gram sits on an unreliable boundary). Used at query time.

For the dense selector every gram is reliable, so reliable_grams/2 returns all of them (or brute_force below the trigram length). The boundary/interior distinction bites the content-defined (sparse) selector, which lives behind this same behaviour without changing anything above it. The options map carries per-selector tuning (e.g. the sparse selector's window radius and sample rate).

Two further callbacks, both optional, add byte-offset positions for a selector backing a positional (phase-2) index (today, only the sparse selector implements them; dense stays non-positional): select_grams_positional/2 (index time), reliable_grams_positional/2 (query time).

Summary

Functions

Whether the selector indexes every trigram (dispatch).

Dispatch reliable_grams/2 to a selector module.

Dispatch reliable_grams_positional/2 to a selector module.

Dispatch select_grams/2 to a selector module.

Dispatch select_grams_positional/2 to a selector module.

Types

gram/0

-type gram() :: 0..16777215.

offset/0

reliable/0

-type reliable() :: {reliable, [gram()]} | brute_force.

reliable_positional/0

-type reliable_positional() :: {reliable, [{gram(), offset()}]} | brute_force.

Callbacks

covers_all_grams/1

-callback covers_all_grams(map()) -> boolean().

reliable_grams/2

-callback reliable_grams(binary(), map()) -> reliable().
Whether the selector indexes EVERY trigram of a document. The regex planner needs this: an arbitrary mandatory trigram is only guaranteed present when the selector covers all grams (dense). A sampling selector (sparse) does not, so regex there must brute-force.

reliable_grams_positional/2

(optional)
-callback reliable_grams_positional(binary(), map()) -> reliable_positional().

select_grams/2

-callback select_grams(binary(), map()) -> [gram()].

select_grams_positional/2

(optional)
-callback select_grams_positional(binary(), map()) -> [{gram(), offset()}].
The query-literal analog of reliable_grams/2: the offset carried alongside each gram is the position within the QUERY literal itself (not a document), for the planner's distance-check math.

Functions

covers_all_grams(Mod, Opts)

-spec covers_all_grams(module(), map()) -> boolean().

Whether the selector indexes every trigram (dispatch).

reliable_grams(Mod, Opts, Query)

-spec reliable_grams(module(), map(), binary()) -> reliable().

Dispatch reliable_grams/2 to a selector module.

reliable_grams_positional(Mod, Opts, Query)

-spec reliable_grams_positional(module(), map(), binary()) -> reliable_positional().

Dispatch reliable_grams_positional/2 to a selector module.

select_grams(Mod, Opts, Bytes)

-spec select_grams(module(), map(), binary()) -> [gram()].

Dispatch select_grams/2 to a selector module.

select_grams_positional(Mod, Opts, Bytes)

-spec select_grams_positional(module(), map(), binary()) -> [{gram(), offset()}].

Dispatch select_grams_positional/2 to a selector module.