SearchCore.Pipeline (search_core v0.3.0)

Turn a raw string into a list of normalized, stemmed search tokens.

This is a cleaned-up version of the classic livebook pipeline:

downcase
tokenize on non-letter/non-digit boundaries (Unicode aware, keeps accents)
drop tokens shorter than :min_length
drop stopwords for the language (surface forms, accented)
stem each token via Text.Stemmer (accents preserved → best Snowball quality)
optionally fold accents on the final stems (so "idee" matches "idée")

Accent folding is done last, after stemming, so Snowball still sees accented input while the stored/queried tokens stay accent-insensitive. Because indexing and querying both go through this same function, the two sides stay in lock-step — the usual "search returns nothing because the query wasn't stemmed the same way" bug cannot happen as long as you use process/3 on both.

iex> SearchCore.Pipeline.process("Les idées mangent les chevaux", :fr)
["ide", "mangent", "cheval"]

Summary

Types

opt()

Functions

fold_accents(string)

Fold diacritics on a single string: NFD-normalize then drop combining marks.

normalize(string)

The case- and accent-insensitive normal form of a string: trim, downcase, fold accents. No tokenizing, no stemming — this answers "do these strings read the same", not "do they share stems".

process(text, lang, opts \\ [])

Process text in lang, returning the list of search tokens.

Types

opt()

@type opt() ::
  {:min_length, non_neg_integer()}
  | {:max_bytes, pos_integer()}
  | {:stopwords, boolean()}
  | {:fold_accents, boolean()}

Functions

fold_accents(string)

@spec fold_accents(String.t()) :: String.t()

Fold diacritics on a single string: NFD-normalize then drop combining marks.

iex> SearchCore.Pipeline.fold_accents("Élévation")
"Elevation"

normalize(string)

@spec normalize(String.t()) :: String.t()

The case- and accent-insensitive normal form of a string: trim, downcase, fold accents. No tokenizing, no stemming — this answers "do these strings read the same", not "do they share stems".

Use it on both sides of any literal comparison (what you store and what you compare against), the same way process/3 backs both indexing and querying — the comparison only works if the two sides go through the same function.

iex> SearchCore.Pipeline.normalize("  Maraîcher Bio ")
"maraicher bio"

process(text, lang, opts \\ [])

@spec process(String.t(), SearchCore.Language.t(), [opt()]) :: [String.t()]

Process text in lang, returning the list of search tokens.

lang is an ISO 639-1 code (:fr) or an algorithm variant (:en_porter); it raises ArgumentError if the installed stemmer does not support it.

Options:

:min_length (default 2) — drop tokens shorter than this (measured before stemming)
:max_bytes (default 2046) — drop tokens longer than this, measured in bytes on the final token. This is Postgres' hard limit for a single lexeme; a base64 blob or a long hash in an indexed field would otherwise be unstorable.
:stopwords (default true) — remove language stopwords
:fold_accents (default true) — strip diacritics from the final tokens
iex> SearchCore.Pipeline.process("chat " <> String.duplicate("a", 3000), :fr) ["chat"]