Introduction

unicode_string implements the Unicode text segmentation and casing algorithms for Elixir: where a string can be divided into user-perceived characters, words, sentences and line break opportunities, and how it changes case in a way that respects language.

Elixir's own String module covers the common cases well. This library exists for the cases it doesn't: segmentation that follows UAX #29 and UAX #14 exactly, locale tailoring from CLDR, and dictionary-based segmentation for scripts that do not put spaces between words.

Installation

def deps do
  [
    {:unicode_string, "~> 2.4"}
  ]
end

Thai, Lao, Khmer, Burmese, Chinese and Japanese word segmentation needs dictionaries, which are downloaded separately because of their size:

mix unicode.string.download.dictionaries

The four break types

Every segmentation function takes a :break option naming one of four algorithms. They answer different questions and it is worth being clear which one you want.

The default is :word.

iex> Unicode.String.split("Hello there. Goodbye.", break: :sentence)
["Hello there. ", "Goodbye."]

iex> Unicode.String.split("Hello there.", break: :word, trim: true)
["Hello", "there", "."]

The four functions

Each break type is available through the same four entry points.

split/2 returns all the segments:

iex> Unicode.String.split("one two", break: :word, trim: true)
["one", "two"]

next/2 returns the first segment and the rest, so you can drive segmentation yourself:

iex> Unicode.String.next("one two", break: :word)
{"one", " two"}

stream/2 returns a lazy enumerable, which avoids materialising every segment of a large string at once:

iex> "one two three" |> Unicode.String.stream(break: :word, trim: true) |> Enum.take(2)
["one", "two"]

break?/2 answers whether a boundary falls between two strings:

iex> Unicode.String.break?({"one", " two"}, break: :word)
true

iex> Unicode.String.break?({"th", "ere"}, break: :word)
false

Locales

Most of the algorithms are locale-independent, but some are not. Sentence breaking uses per-locale abbreviation lists so that "Mr." does not end a sentence; line breaking treats Japanese small kana differently from the default; Greek adds its own sentence terminator.

iex> Unicode.String.split("No, I don't have a Ph.D. but I don't think it matters.", break: :sentence, locale: :en)
["No, I don't have a Ph.D. but I don't think it matters."]

A locale with no tailoring falls back to :root, which carries the untailored rules. The locales that have segmentation data are returned by Unicode.String.Segment.known_segmentation_locales/0.

Casing

Case conversion is locale-sensitive in ways that surprise people — Turkish has a dotted capital I, Greek drops accents when upper casing, Dutch title cases "ij" as a unit. See the Casing guide.

iex> Unicode.String.upcase("the quick brown fox")
"THE QUICK BROWN FOX"

Conformance

The four break types are generated from the state machine tables published in PRI #555 rather than transcribed from the prose of the annexes, and all four pass the full Unicode conformance corpora. See conformance.md for the detail, including where this library differs from ICU.

An optional ICU backend

An opt-in NIF binding to ICU4C is available for workloads where segmentation is the bottleneck. It is off by default, needs ICU system libraries, and falls back to the native implementation whenever it is unavailable, so backend: :nif is always safe to pass. conformance.md has the measurements and the cases where it is worth enabling.