SearchCore.Tsvector (search_core v0.3.0)

Build the two strings you feed to Postgres full-text search, both produced by the same SearchCore.Pipeline so the index side and the query side always agree.

Because tokens are already stemmed and accent-folded in Elixir, you use the 'simple' Postgres configuration (no further stemming/stopwords in the database).

There are two ways to fill the indexed column, and they need different SQL — pick one and use it in the index, the filter and the rank alike, or the index is silently skipped:

-- weighted/3: the column holds a tsvector literal, so SQL casts it.
-- This is the one to reach for; it is what carries per-field weights.
search_text::tsvector

-- searchable_text/3: the column holds plain stemmed tokens, so SQL parses them.
to_tsvector('simple', search_text)

-- querying, either way
to_tsquery('simple', $1)   -- $1 = tsquery/3

Tokens are restricted to letters/digits and capped in length (see SearchCore.Pipeline), so the produced tsquery string contains no operators to escape, and the literal weighted/3 builds needs no escaping and cannot exceed Postgres' per-lexeme limit.

Summary

Types

weight()

A Postgres tsvector weight class; :d is the default and lowest.

Functions

searchable_text(text, lang, opts \\ [])

Space-joined stemmed tokens to store in a search_text column for indexing.

tsquery(query, lang, opts \\ [])

Build a tsquery string from a user query.

weighted(segments, lang, opts \\ [])

Build a weighted tsvector literal from {text, weight} segments, to store in the search column instead of plain searchable_text/3 output.

Types

weight()

@type weight() :: :a | :b | :c | :d

A Postgres tsvector weight class; :d is the default and lowest.

Functions

searchable_text(text, lang, opts \\ [])

@spec searchable_text(String.t(), SearchCore.Language.t(), [SearchCore.Pipeline.opt()]) ::
  String.t()

Space-joined stemmed tokens to store in a search_text column for indexing.

iex> SearchCore.Tsvector.searchable_text("Les chevaux mangent", :fr)
"cheval mangent"

tsquery(query, lang, opts \\ [])

@spec tsquery(String.t(), SearchCore.Language.t(), keyword()) :: String.t()

Build a tsquery string from a user query.

Options (in addition to SearchCore.Pipeline options):

:combinator — :and (default) requires all terms, :or requires any
:prefix — when true, append :* to each term for prefix matching

Returns "" when the query has no usable tokens; to_tsquery('simple', '') simply matches nothing.

iex> SearchCore.Tsvector.tsquery("chevaux mangent", :fr)
"cheval & mangent"

iex> SearchCore.Tsvector.tsquery("chev", :fr, prefix: true, combinator: :or)
"chev:*"

weighted(segments, lang, opts \\ [])

@spec weighted([{String.t(), weight()}], SearchCore.Language.t(), [
  SearchCore.Pipeline.opt()
]) ::
  String.t()

Build a weighted tsvector literal from {text, weight} segments, to store in the search column instead of plain searchable_text/3 output.

Weights are Postgres' :a | :b | :c | :d classes, :d being the default and lowest. ts_rank scores an :a match well above a :d one, which is how a hit in a reference or a title outranks the same hit in a body.

iex> SearchCore.Tsvector.weighted([{"Chevaux", :a}, {"mangent du foin", :d}], :fr)
"'cheval':1A 'mangent':2 'foin':3"

Positions are numbered across all segments in order, so term frequency still feeds the rank. The column holds a tsvector literal, so the SQL side casts rather than calls to_tsvector:

-- indexing
CREATE INDEX … USING GIN ((search_text::tsvector))

-- querying
search_text::tsvector @@ to_tsquery('simple', $1)

Tokens are letters and digits only (see SearchCore.Pipeline), so the generated literal needs no escaping. An empty result is "", which casts to an empty tsvector and matches nothing.