The matcher: one left-to-right walk producing slug text plus the spans it came from.
The algorithm, in one place
Everything positional is counted in grapheme clusters.
Pre-pass (once):
- locale-aware lowercase (
:turkic/:greek/ default) - normalise to NFC — every later step reads NFC text, nothing re-normalises
- compute word boundaries by Unicode property, before any mapping
Main pass — at each position, gather every rule from L2, L1 and L0 whose key matches here, then:
longest key wins
tie on length -> higher layer wins (L2 > L1 > L0)
tie within a layer -> first declared wins
nothing matched -> retry against the grapheme's canonical base
still nothing -> strip itWhy longest-match and layer precedence cannot fight
They look like competing rules. They cannot disagree, by construction:
- L0 keys are always one grapheme — enforced by the generator.
- L1 and L2 never both match — L2 is Latin-only, L1 keys come from non-Latin script blocks.
So a key longer than one grapheme only ever comes from one layer, and cross-layer
competition is always a length tie, settled by precedence. German ö→oe (L2)
cannot be beaten by a longer L0 rule, because no such rule can exist.
Base-retry, and why it must come after the precomposed lookup
A grapheme with no rule is retried against its canonical base (NFD, marks dropped)
before being stripped. That is what lets the Greek table omit ά έ ή ί ό ύ ώ — they
decompose to base vowels that are keyed.
The ordering is load-bearing. Decompose first and German ships o instead of oe,
because NFD turns ö into o + combining diaeresis and the de table keys the
precomposed U+00F6. Retry is the fallback, never the first move.
Spans
Each emitted chunk records the rule output that produced it, so truncation can refuse
to sever щ→shch into shc. Spans are carried through separator collapsing and
trimming, because those steps move every offset.
Summary
Functions
The token a rule emits for a word break; rendered as the caller's separator.
Walks text and returns {output, spans}.
Functions
@spec separator_token() :: String.t()
The token a rule emits for a word break; rendered as the caller's separator.
@spec walk(String.t(), [LocaleSlug.Tables.t() | nil], MapSet.t(), MapSet.t()) :: {String.t(), [{non_neg_integer(), pos_integer()}]}
Walks text and returns {output, spans}.
spans is a list of {start_index, length} over the returned string, one per rule
output. Truncation must not cut inside one.