SearchCore.Pipeline (search_core v0.1.0)

Copy Markdown View Source

Turn a raw string into a list of normalized, stemmed search tokens.

This is a cleaned-up version of the classic livebook pipeline:

  1. downcase
  2. tokenize on non-letter/non-digit boundaries (Unicode aware, keeps accents)
  3. drop tokens shorter than :min_length
  4. drop stopwords for the language (surface forms, accented)
  5. stem each token via the Stemmers NIF (accents preserved → best Snowball quality)
  6. optionally fold accents on the final stems (so "idee" matches "idée")

Accent folding is done last, after stemming, so Snowball still sees accented input while the stored/queried tokens stay accent-insensitive. Because indexing and querying both go through this same function, the two sides stay in lock-step — the usual "search returns nothing because the query wasn't stemmed the same way" bug cannot happen as long as you use process/3 on both.

iex> SearchCore.Pipeline.process("Les idées mangent les chevaux", :french)
["ide", "mangent", "cheval"]

Summary

Types

opt()

Functions

fold_accents(string)

Fold diacritics on a single string: NFD-normalize then drop combining marks.

process(text, lang, opts \\ [])

Process text in lang, returning the list of search tokens.

Types

opt()

@type opt() ::
  {:min_length, non_neg_integer()}
  | {:stopwords, boolean()}
  | {:fold_accents, boolean()}

Functions

fold_accents(string)

@spec fold_accents(String.t()) :: String.t()

Fold diacritics on a single string: NFD-normalize then drop combining marks.

iex> SearchCore.Pipeline.fold_accents("Élévation")
"Elevation"

process(text, lang, opts \\ [])

@spec process(String.t(), Stemmers.language(), [opt()]) :: [String.t()]

Process text in lang, returning the list of search tokens.

Options:

  • :min_length (default 2) — drop tokens shorter than this (measured before stemming)
  • :stopwords (default true) — remove language stopwords
  • :fold_accents (default true) — strip diacritics from the final tokens