barrel_ngram_selector behaviour (barrel_ngram v0.7.1)

View Source

Gram-selection behaviour: the shared seam between the indexer and the query planner.

A selector maps a byte string to the set of trigrams it contributes to the index. The SAME selector is applied by the indexer (over full document bytes) and by the query planner (over the query literal), so that the grams a literal produces are always a subset of the grams its containing documents produced. That subset relationship is what makes the trigram intersection a correct necessary-condition filter.

Two callbacks, each taking the bytes and a selector-options map:

  • select_grams/2 - the grams a byte string contributes to the index. Used at index time.
  • reliable_grams/2 - the grams of a query literal the planner may safely intersect over, or brute_force when it may not (too short, or every gram sits on an unreliable boundary). Used at query time.

For the dense selector every gram is reliable, so reliable_grams/2 returns all of them (or brute_force below the trigram length). The boundary/interior distinction bites the content-defined (sparse) selector, which lives behind this same behaviour without changing anything above it. The options map carries per-selector tuning (e.g. the sparse selector's window radius and sample rate).

Summary

Functions

Whether the selector indexes every trigram (dispatch).

Dispatch reliable_grams/2 to a selector module.

Dispatch select_grams/2 to a selector module.

Types

gram/0

-type gram() :: 0..16777215.

reliable/0

-type reliable() :: {reliable, [gram()]} | brute_force.

Callbacks

covers_all_grams/1

-callback covers_all_grams(map()) -> boolean().

reliable_grams/2

-callback reliable_grams(binary(), map()) -> reliable().
Whether the selector indexes EVERY trigram of a document. The regex planner needs this: an arbitrary mandatory trigram is only guaranteed present when the selector covers all grams (dense). A sampling selector (sparse) does not, so regex there must brute-force.

select_grams/2

-callback select_grams(binary(), map()) -> [gram()].

Functions

covers_all_grams(Mod, Opts)

-spec covers_all_grams(module(), map()) -> boolean().

Whether the selector indexes every trigram (dispatch).

reliable_grams(Mod, Opts, Query)

-spec reliable_grams(module(), map(), binary()) -> reliable().

Dispatch reliable_grams/2 to a selector module.

select_grams(Mod, Opts, Bytes)

-spec select_grams(module(), map(), binary()) -> [gram()].

Dispatch select_grams/2 to a selector module.