Token inspection using explicit analyzers or a table index.
Summary
Functions
Inspects tokens using an analyzer keyword list, with the same options as
FTS index creation. Returns maps with string keys "text" and "position";
positions start at zero and can have gaps after stop-word removal.
Empty text is accepted. Invalid UTF-8 or options raise ArgumentError.
Inspects tokens with a table index's saved analyzer, including on a snapshot.
Supply exactly one selector, :index or :column, as a string or atom.
Column selection requires exactly one FTS index. Missing, ambiguous, or
non-FTS indexes return an error. The operation changes neither table nor index.
Types
@type token() :: %{required(String.t()) => String.t() | non_neg_integer()}
Functions
@spec tokenize(String.t(), keyword()) :: LanceDB.result([token()])
Inspects tokens using an analyzer keyword list, with the same options as
FTS index creation. Returns maps with string keys "text" and "position";
positions start at zero and can have gaps after stop-word removal.
Empty text is accepted. Invalid UTF-8 or options raise ArgumentError.
@spec tokenize(LanceDB.Table.t(), String.t(), keyword()) :: LanceDB.result([token()])
Inspects tokens with a table index's saved analyzer, including on a snapshot.
Supply exactly one selector, :index or :column, as a string or atom.
Column selection requires exactly one FTS index. Missing, ambiguous, or
non-FTS indexes return an error. The operation changes neither table nor index.