defmodule Text.Stemmer do @moduledoc """ Pre-compiled Snowball stemmers for Elixir. This package ships 36 pre-compiled stemming algorithms covering a wide range of natural languages. The stemmers themselves were generated from the canonical Snowball algorithm sources using the [`:snowball`](https://hex.pm/packages/snowball) compiler. Languages are identified by their [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) two-letter code. Algorithms that target a specific variant or stem from a named algorithm rather than a language use a `_` atom (e.g. `:nl_porter`) or a bare algorithm name (`:en_lovins`, `:en_porter`). | Atom | Language / Algorithm | |---------------|---------------------------| | `:ar` | Arabic | | `:ca` | Catalan | | `:cs` | Czech | | `:da` | Danish | | `:de` | German | | `:el` | Greek | | `:en` | English (Porter2) | | `:en_lovins` | English — Lovins (1968) | | `:en_porter` | English — Porter (1980) | | `:eo` | Esperanto | | `:es` | Spanish | | `:et` | Estonian | | `:eu` | Basque | | `:fa` | Persian | | `:fi` | Finnish | | `:fr` | French | | `:ga` | Irish | | `:hi` | Hindi | | `:hu` | Hungarian | | `:hy` | Armenian | | `:id` | Indonesian | | `:it` | Italian | | `:lt` | Lithuanian | | `:ne` | Nepali | | `:nl` | Dutch | | `:nl_porter` | Dutch — Porter variant | | `:no` | Norwegian | | `:pl` | Polish | | `:pt` | Portuguese | | `:ro` | Romanian | | `:ru` | Russian | | `:sr` | Serbian | | `:sv` | Swedish | | `:ta` | Tamil | | `:tr` | Turkish | | `:yi` | Yiddish | ## Examples iex> Text.Stemmer.stem("generalizations", :en) "general" iex> Text.Stemmer.stem("running", :en) "run" """ @typedoc """ An ISO 639-1 two-letter language code, or an algorithm-qualified atom for algorithm variants. See `supported_languages/0` for the full list. """ @type language :: atom() @doc """ Stem a single word using the named algorithm. ### Arguments * `word` is the input word as a UTF-8 binary. * `language` is an ISO 639-1 atom (e.g. `:en`, `:fr`) or an algorithm-qualified atom (e.g. `:en_porter`, `:nl_porter`). See `supported_languages/0` for the full list. ### Returns * The stemmed word as a UTF-8 binary. ### Examples iex> Text.Stemmer.stem("running", :en) "run" iex> Text.Stemmer.stem("gouvernements", :fr) "gouvern" """ @spec stem(binary(), language()) :: binary() def stem(word, language) when is_binary(word) and is_atom(language) do module = stemmer_module!(language) module.stem(word) end @doc """ Stem a list of words using the named algorithm. ### Arguments * `words` is a list of UTF-8 binaries. * `language` is an ISO 639-1 atom or algorithm-qualified atom naming the stemmer to use. ### Returns * A list of stemmed UTF-8 binaries, one per input word. ### Examples iex> Text.Stemmer.stem_list(["running", "ran", "runs"], :en) ["run", "ran", "run"] """ @spec stem_list([binary()], language()) :: [binary()] def stem_list(words, language) when is_list(words) and is_atom(language) do module = stemmer_module!(language) Enum.map(words, &module.stem/1) end @doc """ Return the sorted list of supported language atoms. Language atoms follow the [ISO 639-1](https://en.wikipedia.org/wiki/List_of_ISO_639-1_codes) two-letter convention. Algorithm-specific variants use a `_` form (`:en_porter`, `:en_lovins`, `:nl_porter`). ### Returns * A sorted list of atoms, one per supported algorithm. ### Examples iex> :en in Text.Stemmer.supported_languages() true iex> length(Text.Stemmer.supported_languages()) 36 """ @spec supported_languages() :: [language()] def supported_languages do [ :ar, :ca, :cs, :da, :de, :el, :en, :en_lovins, :en_porter, :eo, :es, :et, :eu, :fa, :fi, :fr, :ga, :hi, :hu, :hy, :id, :it, :lt, :ne, :nl, :nl_porter, :no, :pl, :pt, :ro, :ru, :sr, :sv, :ta, :tr, :yi ] end defp stemmer_module!(:ar), do: Text.Stemmer.Stemmers.Arabic defp stemmer_module!(:ca), do: Text.Stemmer.Stemmers.Catalan defp stemmer_module!(:cs), do: Text.Stemmer.Stemmers.Czech defp stemmer_module!(:da), do: Text.Stemmer.Stemmers.Danish defp stemmer_module!(:de), do: Text.Stemmer.Stemmers.German defp stemmer_module!(:el), do: Text.Stemmer.Stemmers.Greek defp stemmer_module!(:en), do: Text.Stemmer.Stemmers.English defp stemmer_module!(:en_lovins), do: Text.Stemmer.Stemmers.Lovins defp stemmer_module!(:en_porter), do: Text.Stemmer.Stemmers.Porter defp stemmer_module!(:eo), do: Text.Stemmer.Stemmers.Esperanto defp stemmer_module!(:es), do: Text.Stemmer.Stemmers.Spanish defp stemmer_module!(:et), do: Text.Stemmer.Stemmers.Estonian defp stemmer_module!(:eu), do: Text.Stemmer.Stemmers.Basque defp stemmer_module!(:fa), do: Text.Stemmer.Stemmers.Persian defp stemmer_module!(:fi), do: Text.Stemmer.Stemmers.Finnish defp stemmer_module!(:fr), do: Text.Stemmer.Stemmers.French defp stemmer_module!(:ga), do: Text.Stemmer.Stemmers.Irish defp stemmer_module!(:hi), do: Text.Stemmer.Stemmers.Hindi defp stemmer_module!(:hu), do: Text.Stemmer.Stemmers.Hungarian defp stemmer_module!(:hy), do: Text.Stemmer.Stemmers.Armenian defp stemmer_module!(:id), do: Text.Stemmer.Stemmers.Indonesian defp stemmer_module!(:it), do: Text.Stemmer.Stemmers.Italian defp stemmer_module!(:lt), do: Text.Stemmer.Stemmers.Lithuanian defp stemmer_module!(:ne), do: Text.Stemmer.Stemmers.Nepali defp stemmer_module!(:nl), do: Text.Stemmer.Stemmers.Dutch defp stemmer_module!(:nl_porter), do: Text.Stemmer.Stemmers.DutchPorter defp stemmer_module!(:no), do: Text.Stemmer.Stemmers.Norwegian defp stemmer_module!(:pl), do: Text.Stemmer.Stemmers.Polish defp stemmer_module!(:pt), do: Text.Stemmer.Stemmers.Portuguese defp stemmer_module!(:ro), do: Text.Stemmer.Stemmers.Romanian defp stemmer_module!(:ru), do: Text.Stemmer.Stemmers.Russian defp stemmer_module!(:sr), do: Text.Stemmer.Stemmers.Serbian defp stemmer_module!(:sv), do: Text.Stemmer.Stemmers.Swedish defp stemmer_module!(:ta), do: Text.Stemmer.Stemmers.Tamil defp stemmer_module!(:tr), do: Text.Stemmer.Stemmers.Turkish defp stemmer_module!(:yi), do: Text.Stemmer.Stemmers.Yiddish defp stemmer_module!(language) do raise ArgumentError, "unsupported language #{inspect(language)}. " <> "Supported: #{inspect(supported_languages())}" end end