Word segmentation divides text into words and the things between them. It is the default break type, and it is what you want for word counts, search indexing, and "select word" behaviour.
iex> Unicode.String.split("This is a sentence.", break: :word)
["This", " ", "is", " ", "a", " ", "sentence", "."]The segments cover the whole input, including spaces and punctuation. trim: true drops the segments that are entirely whitespace:
iex> Unicode.String.split("This is a sentence.", break: :word, trim: true)
["This", "is", "a", "sentence", "."]Punctuation is still there. To keep only the word-like segments, filter with Unicode.String.word_like?/1:
iex> "This is a sentence." |> Unicode.String.split(break: :word, trim: true) |> Enum.filter(&Unicode.String.word_like?/1)
["This", "is", "a", "sentence"]What counts as one word
UAX #29 keeps together several things that a naive split on spaces would separate. Contractions stay whole because an apostrophe between letters does not break:
iex> Unicode.String.split("don't", break: :word)
["don't"]Numbers keep their internal separators:
iex> Unicode.String.split("1,234.56", break: :word)
["1,234.56"]And a full stop inside an abbreviation does not split it from the letters around it in the way a sentence-ending full stop would.
CLDR deviations
CLDR tailors the Unicode rules slightly, and this library follows CLDR. The most visible change is that a colon is not treated as a mid-letter character, so a:b breaks where the plain Unicode rules would keep it together. This is why a handful of lines in the Unicode conformance file are excluded from the test suite — they test the untailored behaviour that CLDR deliberately changes.
Scripts without spaces
Thai, Lao, Khmer, Burmese, Chinese and Japanese do not put spaces between words, so no rule over character classes can find the boundaries. These need a dictionary, and the dictionaries are downloaded separately:
mix unicode.string.download.dictionaries
With a dictionary present, pass the locale:
Unicode.String.split("ทดสอบภาษาไทย", break: :word, locale: :th)Without one, the standard rules are used instead, which is a much better answer than shattering the text into single characters. The locales with dictionaries are returned by Unicode.String.Dictionary.known_dictionary_locales/0.
Mixed-script text is handled by partitioning first: runs of the dictionary's own script go to the dictionary and everything else goes to the rules. Handing a whole mixed string to a dictionary that knows only one script would break the rest into single characters.
Streaming
For large inputs, stream/2 produces segments lazily:
iex> "one two three" |> Unicode.String.stream(break: :word, trim: true) |> Enum.to_list()
["one", "two", "three"]