AshScylla.Search.Analyzer (AshScylla v1.9.0)

Text analysis pipeline coordinator.

Orchestrates the full text analysis pipeline:

Document text
  → Tokenizer (split into words)
  → Normalizer (lowercase, strip punctuation, NFC normalize)
  → Stop Words filter (remove common words)
  → Stemmer (reduce to root form)
  → Unique terms with counts

Usage

iex> Analyzer.analyze("Learning Elixir Phoenix Framework")
[{"phoenix", 1}, {"framework", 1}, {"learn", 1}, {"elixir", 1}]

The result is a keyword list of {term, frequency} pairs ready for indexing or query processing.

Summary

Functions

analyze(text, opts \\ [])

Analyzes text and returns a list of {term, term_frequency} tuples.

analyze_query(query, opts \\ [])

Analyzes a query string for search.

Functions

analyze(text, opts \\ [])

@spec analyze(
  String.t(),
  keyword()
) :: [{String.t(), pos_integer()}]

Analyzes text and returns a list of {term, term_frequency} tuples.

The terms are:

Tokenized from the input text
Normalized (lowercase, punctuation removal, NFC)
Filtered to remove stop words
Stemmed to their root form
Deduplicated with frequency counts

Options

:stem — whether to apply stemming (default: true)
:remove_stop_words — whether to remove stop words (default: true)
:min_length — minimum token length (default: 1)

Examples

iex> Analyzer.analyze("The Phoenix Framework is running fast")
[{"phoenix", 1}, {"framework", 1}, {"run", 1}, {"fast", 1}]

analyze_query(query, opts \\ [])

@spec analyze_query(
  String.t(),
  keyword()
) :: [String.t()]

Analyzes a query string for search.

Applies the same pipeline as analyze/2 — tokenize, normalize, stop-word filtering, stemming — so query terms are consistent with indexed terms regardless of casing or punctuation.

Examples

iex> Analyzer.analyze_query("learning phoenix framework")
["learn", "phoenix", "framework"]