LocaleSlug.Engine (LocaleSlug v0.2.0)

Copy Markdown View Source

The matcher: one left-to-right walk producing slug text plus the spans it came from.

The algorithm, in one place

Everything positional is counted in grapheme clusters.

Pre-pass (once):

  1. locale-aware lowercase (:turkic / :greek / default)
  2. normalise to NFC — every later step reads NFC text, nothing re-normalises
  3. compute word boundaries by Unicode property, before any mapping

Main pass — at each position, gather every rule from L2, L1 and L0 whose key matches here, then:

longest key wins
tie on length      -> higher layer wins (L2 > L1 > L0)
tie within a layer -> first declared wins
nothing matched    -> retry against the grapheme's canonical base
still nothing      -> strip it

Why longest-match and layer precedence cannot fight

They look like competing rules. They cannot disagree, by construction:

  • L0 keys are always one grapheme — enforced by the generator.
  • L1 and L2 never both match — L2 is Latin-only, L1 keys come from non-Latin script blocks.

So a key longer than one grapheme only ever comes from one layer, and cross-layer competition is always a length tie, settled by precedence. German ö→oe (L2) cannot be beaten by a longer L0 rule, because no such rule can exist.

Base-retry, and why it must come after the precomposed lookup

A grapheme with no rule is retried against its canonical base (NFD, marks dropped) before being stripped. That is what lets the Greek table omit ά έ ή ί ό ύ ώ — they decompose to base vowels that are keyed.

The ordering is load-bearing. Decompose first and German ships o instead of oe, because NFD turns ö into o + combining diaeresis and the de table keys the precomposed U+00F6. Retry is the fallback, never the first move.

Spans

Each emitted chunk records the rule output that produced it, so truncation can refuse to sever щ→shch into shc. Spans are carried through separator collapsing and trimming, because those steps move every offset.

Summary

Functions

The token a rule emits for a word break; rendered as the caller's separator.

Walks text and returns {output, spans}.

Functions

separator_token()

@spec separator_token() :: String.t()

The token a rule emits for a word break; rendered as the caller's separator.

walk(text, layers, word_starts, word_ends)

@spec walk(String.t(), [LocaleSlug.Tables.t() | nil], MapSet.t(), MapSet.t()) ::
  {String.t(), [{non_neg_integer(), pos_integer()}]}

Walks text and returns {output, spans}.

spans is a list of {start_index, length} over the returned string, one per rule output. Truncation must not cut inside one.