Table-driven engine implementing UAX #29 sentence breaking, with CLDR's locale tailoring and abbreviation suppressions.
Generated from the SentenceBreak state machine tables published in PRI #555.
See Unicode.String.Dfa for the engine and Unicode.String.Break.Tailoring
for the tailoring and suppressions.
The arity-1 functions implement UAX #29 as published. The arity-3 and arity-4 functions add the locale-dependent behaviour on top.
Summary
Functions
Returns true when a segment boundary falls between string_before and
string_after.
Returns true when a sentence boundary falls between the two strings.
Returns {segment, rest}, or nil when string is empty.
Returns {sentence, rest}, or nil when string is empty.
Splits string using the rules alone, without the dictionary pass.
Splits string into segments.
Splits string into sentences under locale.
Lazily splits string into sentences under locale.
Functions
Returns true when a segment boundary falls between string_before and
string_after.
The automaton always restarts at a boundary, so the question "is there a break at this join" is answered by segmenting from the start of the combined text and asking whether any boundary lands exactly on the join.
Returns true when a sentence boundary falls between the two strings.
Returns {segment, rest}, or nil when string is empty.
Returns {sentence, rest}, or nil when string is empty.
Splits string using the rules alone, without the dictionary pass.
Splits string into segments.
Where the data defines dictionary symbols, the rule-based segments are passed through the dictionary breaker afterwards, which is how the standard describes complex context-dependent breaking being triggered.
Splits string into sentences under locale.
Lazily splits string into sentences under locale.
The stream carries the tailored text alongside the original and the offset reached in it, so the tailoring is applied once at construction and every sentence is still sliced from the original string.