Unicode.String.Break.Tailoring (Unicode String v2.4.0)

CLDR's locale tailoring of the segmentation rules.

UAX #14 and UAX #29 define one set of rules for all languages. CLDR carries per-locale departures from them as locale data, and this module applies the ones this library supports. Two kinds arise:

Break class tailoring, where a locale gives a character a different break class from the one the UCD assigns it. Greek treats U+003B as a sentence terminator; Japanese and Chinese treat conditional Japanese starters as ideographs rather than non-starters, the tailoring usually called CJK loose line breaking.
Abbreviation suppressions, where a locale lists abbreviations such as "Mr." that end in a full stop without ending a sentence.

Neither is expressible in the rules themselves, so both are applied around a break engine rather than inside it, and this module is the single home for that behaviour.

Summary

Functions

classify(arg1, codepoint)

Returns the sentence break class of a codepoint under a locale's tailoring.

suppressed?(segment, locale, suppressions)

Returns true when a segment ends in an abbreviation the locale suppresses breaking after.

tailor(string, locale, break_type)

Rewrites a locale's tailored characters to standard characters carrying the break class the locale gives them.

Functions

classify(arg1, codepoint)

Returns the sentence break class of a codepoint under a locale's tailoring.

Arguments

locale is a locale atom such as :en or :el.
codepoint is an integer codepoint.

Returns

The sentence break class as an atom, such as :sterm or :lower.

Examples

iex> Unicode.String.Break.Tailoring.classify(:en, ?;)
:scontinue

iex> Unicode.String.Break.Tailoring.classify(:el, ?;)
:sterm

suppressed?(segment, locale, suppressions)

Returns true when a segment ends in an abbreviation the locale suppresses breaking after.

CLDR lists abbreviations such as "Mr." and "Dr." that end in a full stop without ending a sentence. The rules break after them regardless, so the break is cancelled afterwards by matching the segment's trailing word against the locale's suppression set.

Only an ATerm-led break can be suppressed. A segment ending in an STerm is a sentence end whatever word precedes it, and the check rejects it because the character before the trailing Close* Sp* ParaSep? run is not an ATerm.

Arguments

segment is the candidate sentence, ending at the break being tested.
locale is a locale atom such as :en or :de.
suppressions is a MapSet of downcased abbreviations, as returned by Unicode.String.Segment.suppressions!/2.

Returns

true when the break should be cancelled and the segment extended.
false otherwise.

Examples

iex> suppressions = MapSet.new(["mr"])
iex> Unicode.String.Break.Tailoring.suppressed?("Hello Mr.", :en, suppressions)
true

iex> suppressions = MapSet.new(["mr"])
iex> Unicode.String.Break.Tailoring.suppressed?("Hello Ms.", :en, suppressions)
false

tailor(string, locale, break_type)

Rewrites a locale's tailored characters to standard characters carrying the break class the locale gives them.

A table-driven engine resolves a character to a symbol with a table fixed at compile time, so it has nowhere to put a per-locale exception. Rewriting the tailored characters to standard characters of the class the locale wants has the same effect on every rule.

Each substitute encodes to the same number of UTF-8 bytes as the character it replaces, so every byte offset computed over the returned string indexes the original string and segments are sliced from the original rather than from the rewritten copy. The rewrite never reaches the caller.

Arguments

string is the text about to be segmented.
locale is a locale atom such as :en, :el or :ja.
break_type is :sentence, :line, :word or :grapheme.

Returns

string unchanged when the locale has no tailoring for this break type, which is the overwhelming majority of cases.
A string of the same byte length with the tailored characters substituted.

Examples

iex> Unicode.String.Break.Tailoring.tailor("γδ; Ε", :en, :sentence)
"γδ; Ε"

iex> Unicode.String.Break.Tailoring.tailor("γδ; Ε", :el, :sentence)
"γδ! Ε"

iex> Unicode.String.Break.Tailoring.tailor("ぁあ", :en, :line)
"ぁあ"

iex> Unicode.String.Break.Tailoring.tailor("ぁあ", :ja, :line)
"一あ"