SearchCore.Stopwords (search_core v0.1.0)

Copy Markdown View Source

Language stopword lists used by SearchCore.Pipeline to drop high-frequency, low-signal words before stemming.

Lists are stored as MapSets of lowercased, accented surface forms (matching is done before accent folding). Languages without a bundled list return an empty set, so the pipeline simply keeps every token for them.

Summary

Functions

Languages that ship a bundled stopword list.

Return the stopword MapSet for lang, or an empty set if none is bundled.

Functions

available_languages()

@spec available_languages() :: [atom()]

Languages that ship a bundled stopword list.

get(lang)

@spec get(atom()) :: MapSet.t(String.t())

Return the stopword MapSet for lang, or an empty set if none is bundled.