Intl.Segmenter (Intl v1.0.0-rc.0)

Text segmentation, modelled on Intl.Segmenter.

Splits text into segments by grapheme cluster, word, or sentence boundaries.

:grapheme segmentation uses Elixir's built-in String.graphemes/1 and is always available.
:word and :sentence segmentation requires the optional unicode_string dependency. When that library is not installed, these granularities return an error.

segment/2 returns a flat list of segment strings for simplicity. segment_with_metadata/2 mirrors the JS segment objects, returning maps with :segment, :index, and :word_like? keys.

Summary

Functions

segment(string, options \\ [])

Segments a string into a list of substrings.

segment!(string, options \\ [])

Segments a string, raising on error.

segment_with_metadata(string, options \\ [])

Segments a string into a list of segment maps with metadata.

segment_with_metadata!(string, options \\ [])

Segments a string into segment maps, raising on error.

Functions

segment(string, options \\ [])

@spec segment(String.t(), Keyword.t()) :: {:ok, [String.t()]} | {:error, term()}

Segments a string into a list of substrings.

Arguments

string is the text to segment.
options is a keyword list of options.

Options

:granularity is :grapheme, :word, or :sentence. The default is :grapheme.
:locale is a locale identifier string. Only used for :word and :sentence granularity. The default is "root".
:trim is a boolean. When true, whitespace-only segments are removed. Only applies to :word and :sentence granularity. The default is false.

Returns

{:ok, segments} where segments is a list of strings.
{:error, reason} if the granularity is not supported or the unicode_string dependency is missing.

Examples

iex> Intl.Segmenter.segment("héllo", granularity: :grapheme)
{:ok, ["h", "é", "l", "l", "o"]}

segment!(string, options \\ [])

@spec segment!(String.t(), Keyword.t()) :: [String.t()] | no_return()

Segments a string, raising on error.

Same as segment/2 but returns the list directly or raises.

Arguments

string is the text to segment.
options is a keyword list of options.

Returns

A list of segment strings.

Examples

iex> Intl.Segmenter.segment!("héllo", granularity: :grapheme)
["h", "é", "l", "l", "o"]

segment_with_metadata(string, options \\ [])

@spec segment_with_metadata(String.t(), Keyword.t()) ::
  {:ok,
   [
     %{
       segment: String.t(),
       index: non_neg_integer(),
       word_like?: boolean() | nil
     }
   ]}
  | {:error, term()}

Segments a string into a list of segment maps with metadata.

Modelled on the JS Intl.Segmenter segment objects. Each map has a :segment (the text), an :index (the byte offset of the segment in the input, where JS uses UTF-16 code-unit indexes), and a :word_like? key. :word_like? is a boolean for :word granularity — true when the segment contains alphabetic or numeric content — and nil for other granularities, matching the JS isWordLike being undefined outside word segmentation.

Arguments

string is the text to segment.
options is a keyword list of options. Accepts the same options as segment/2. With trim: true, whitespace-only segments are removed after indexing, so the remaining :index values still refer to the original string.

Returns

{:ok, segments} where segments is a list of %{segment: String.t(), index: non_neg_integer, word_like?: boolean | nil} maps.
{:error, reason} if the granularity is not supported or the unicode_string dependency is missing.

Examples

iex> Intl.Segmenter.segment_with_metadata("Hi there!", granularity: :word)
{:ok, [
  %{segment: "Hi", index: 0, word_like?: true},
  %{segment: " ", index: 2, word_like?: false},
  %{segment: "there", index: 3, word_like?: true},
  %{segment: "!", index: 8, word_like?: false}
]}

segment_with_metadata!(string, options \\ [])

@spec segment_with_metadata!(String.t(), Keyword.t()) ::
  [
    %{
      segment: String.t(),
      index: non_neg_integer(),
      word_like?: boolean() | nil
    }
  ]
  | no_return()

Segments a string into segment maps, raising on error.

Same as segment_with_metadata/2 but returns the list directly or raises.

Arguments

string is the text to segment.
options is a keyword list of options.

Returns

A list of %{segment: String.t(), index: non_neg_integer, word_like?: boolean | nil} maps.

Examples

iex> Intl.Segmenter.segment_with_metadata!("Hi", granularity: :word)
[%{segment: "Hi", index: 0, word_like?: true}]