Unicode.String.Nif (Unicode String v2.4.0)

Copy Markdown View Source

Optional ICU4C backend for segmentation.

ICU's break iterator is one to two orders of magnitude faster than the native Elixir implementation on rule-driven segmentation. This module makes it available for workloads where that matters, without making ICU a requirement for everyone else.

Enabling

The NIF is opt-in and needs:

  1. ICU system libraries. On macOS brew install icu4c, on Debian or Ubuntu apt install libicu-dev.

  2. The :elixir_make dependency, which is optional and not fetched by default.

  3. The build enabled by either:

    • the environment variable UNICODE_STRING_NIF=true mix compile, or
    • config :unicode_string, :nif, true in config.exs.

The config key must be in config.exs rather than runtime.exs, because it is read at compile time to decide whether to add the :elixir_make compiler.

Using it

Pass backend: :nif to Unicode.String.split/2. When the NIF is unavailable the native implementation is used instead, so the option is always safe:

Unicode.String.split("Hello there", break: :word, backend: :nif)

available?/0 reports whether the shared library loaded.

Differences from the native implementation

The two are not always identical. ICU applies its own locale tailorings, most visibly the CJK loose/normal/strict line break modes that the native implementation does not have, and it uses its own dictionaries for Chinese, Japanese, Thai, Lao, Khmer and Burmese rather than the ones this library downloads. Where the two disagree, conformance.md describes why.

Summary

Functions

available?()

Returns whether the ICU backend is available.

split(string, break, locale \\ "root")

Splits string using the ICU break iterator.

Functions

available?()

@spec available?() :: boolean()

Returns whether the ICU backend is available.

Returns

  • true if the NIF shared library loaded.

  • false if it was not built, or if the ICU libraries are missing.

Examples

iex> is_boolean(Unicode.String.Nif.available?())
true

split(string, break, locale \\ "root")

@spec split(String.t(), atom(), String.t()) :: {:ok, [String.t()]} | {:error, atom()}

Splits string using the ICU break iterator.

Arguments

  • string is any String.t/0.

  • break is one of :grapheme, :word, :line or :sentence.

  • locale is an ICU locale identifier such as "en" or "ja". The default is "root".

Returns

  • {:ok, segments} where segments is a list of String.t/0, or

  • {:error, reason}, including {:error, :unavailable} when the NIF was not built.

Examples

iex> case Unicode.String.Nif.split("Hello there", :word) do
...>   {:ok, segments} -> Enum.member?(segments, "Hello")
...>   {:error, :unavailable} -> true
...> end
true