Decant.Tokenizer (Decant v0.1.0-beta.2)

Splits a search string into tokens.

Defaults: split on runs of whitespace, trim each token, drop empties. Options:

:pattern — Regex.t() or binary delimiter passed to String.split/3. Defaults to ~r/\s+/.
:trim — trim each token. Defaults to true.
:drop_empty — discard empty tokens. Defaults to true.
:downcase — downcase each token. Defaults to false (redundant with case-insensitive ILIKE, exposed for LIKE callers).
:max_tokens — cap the token count (a backstop against pathologically long input producing huge WHERE clauses). Defaults to nil (no cap).

A nil term yields [].

Summary

@spec tokenize(
  String.t() | nil,
  keyword()
) :: [String.t()]