SearchCore.Tsvector (search_core v0.4.0)

Build the two strings you feed to Postgres full-text search, both produced by the same SearchCore.Pipeline so the index side and the query side always agree.

Because tokens are already stemmed and accent-folded in Elixir, you use the 'simple' Postgres configuration (no further stemming/stopwords in the database).

There are two ways to fill the indexed column, and they need different SQL — pick one and use it in the index, the filter and the rank alike, or the index is silently skipped:

-- weighted/3: the column holds a tsvector literal, so SQL casts it.
-- This is the one to reach for; it is what carries per-field weights.
search_text::tsvector

-- searchable_text/3: the column holds plain stemmed tokens, so SQL parses them.
to_tsvector('simple', search_text)

-- querying, either way
to_tsquery('simple', $1)   -- $1 = tsquery/3

Tokens are restricted to letters/digits and capped in length (see SearchCore.Pipeline), so the produced tsquery string contains no operators to escape, and the literal weighted/3 builds needs no escaping and cannot exceed Postgres' per-lexeme limit.

Summary

Types

weight()

A Postgres tsvector weight class; :d is the default and lowest.

Functions

searchable_text(text, lang, opts \\ [])

Space-joined stemmed tokens to store in a search_text column for indexing.

tsquery(query, lang, opts \\ [])

Build a tsquery string from a user query.

weighted(segments, lang, opts \\ [])

Build a weighted tsvector literal from {text, weight} segments, to store in the search column instead of plain searchable_text/3 output.

Types

weight()

@type weight() :: :a | :b | :c | :d

A Postgres tsvector weight class; :d is the default and lowest.

Functions

searchable_text(text, lang, opts \\ [])

@spec searchable_text(String.t(), SearchCore.Language.t(), [SearchCore.Pipeline.opt()]) ::
  String.t()

Space-joined stemmed tokens to store in a search_text column for indexing.

iex> SearchCore.Tsvector.searchable_text("Les chevaux mangent", :fr)
"cheval mangent"

tsquery(query, lang, opts \\ [])

@spec tsquery(String.t(), SearchCore.Language.t(), keyword()) :: String.t()

Build a tsquery string from a user query.

Options (in addition to SearchCore.Pipeline options):

:combinator — :and (default) requires all terms, :or requires any
:prefix — when true, append :* to each term for prefix matching
:synonyms — a %{key => [phrase, ...]} map for query-time expansion (see below)

Returns "" when the query has no usable tokens; to_tsquery('simple', '') simply matches nothing.

iex> SearchCore.Tsvector.tsquery("chevaux mangent", :fr)
"cheval & mangent"

iex> SearchCore.Tsvector.tsquery("chev", :fr, prefix: true, combinator: :or)
"chev:*"

Synonym expansion

When :synonyms is given, a query token that matches a key is expanded into an OR-group alongside its alternatives — a multi-word alternative becoming an AND-group, which is why the query grows parentheses:

iex> SearchCore.Tsvector.tsquery("bl dupont", :fr, synonyms: %{"bl" => ["bon de livraison"]})
"(bl | (bon & livraison)) & dupont"

Keys and values pass through the same pipeline as the query, so they match what is indexed ("bon de livraison" → bon & livraison; de is a stopword). A single-token alternative needs no inner parentheses:

iex> SearchCore.Tsvector.tsquery("cde", :fr, synonyms: %{"cde" => ["commande"]})
"(cde | command)"

Deliberate limits: expansion is query-time (an edited map takes effect on the next search, no reindex); keys must reduce to a single token (multi-token keys are ignored, not raised); the mapping is one-way (add both entries for symmetry); and an alternative that reduces to nothing after the pipeline (all stopwords) is dropped, so the group is never (bl | ()). With no :synonyms, the output is byte-identical to the unexpanded query.

weighted(segments, lang, opts \\ [])

@spec weighted([{String.t(), weight()}], SearchCore.Language.t(), [
  SearchCore.Pipeline.opt()
]) ::
  String.t()

Build a weighted tsvector literal from {text, weight} segments, to store in the search column instead of plain searchable_text/3 output.

Weights are Postgres' :a | :b | :c | :d classes, :d being the default and lowest. ts_rank scores an :a match well above a :d one, which is how a hit in a reference or a title outranks the same hit in a body.

iex> SearchCore.Tsvector.weighted([{"Chevaux", :a}, {"mangent du foin", :d}], :fr)
"'cheval':1A 'mangent':2 'foin':3"

Positions are numbered across all segments in order, so term frequency still feeds the rank. The column holds a tsvector literal, so the SQL side casts rather than calls to_tsvector:

-- indexing
CREATE INDEX … USING GIN ((search_text::tsvector))

-- querying
search_text::tsvector @@ to_tsquery('simple', $1)

Tokens are letters and digits only (see SearchCore.Pipeline), so the generated literal needs no escaping. An empty result is "", which casts to an empty tsvector and matches nothing.