Grapheme Cluster Break

Copy Markdown View Source

A grapheme cluster is what a reader thinks of as a single character. It is often one codepoint, but frequently is not: a base letter plus a combining accent, an emoji built from several codepoints joined by zero-width joiners, a Hangul syllable, a flag made of two regional indicators.

This is the segmentation you want whenever you are counting characters, truncating text, moving a cursor, or reversing a string — anywhere splitting mid-cluster would corrupt what the user sees.

iex> Unicode.String.split("héllo", break: :grapheme) |> length()
5

Why not String.graphemes/1

Elixir's String.graphemes/1 implements the same algorithm and agrees with this library in the overwhelming majority of cases. It is faster, and for most text it is the right choice. Like String.length/1 and the other grapheme-based String functions it is Erlang's unicode_util underneath, so the differences below depend on your OTP release rather than your Elixir version.

The largest difference is in Indic conjuncts. Rule GB9c of UAX #29 keeps a consonant-virama-consonant sequence together as one cluster, but only in the scripts the Indic_Conjunct_Break property covers. OTP has implemented GB9c since OTP 28, but it derives the sets the rule works on from Indic_Syllabic_Category instead, which is not restricted by script. It therefore joins conjuncts in around twenty scripts where UAX #29 requires a break.

Kannada is one of them:

iex> Unicode.String.split("ಕ್ಯಾ", break: :grapheme)
["ಕ್", "ಯಾ"]

# On OTP 28 and later, String.graphemes("ಕ್ಯಾ") returns the single
# cluster ["ಕ್ಯಾ"] instead. OTP 27 and earlier agree with this library.

Devanagari, where the property does apply, agrees on OTP 28 and later:

iex> Unicode.String.split("क्ष", break: :grapheme)
["क्ष"]

# String.graphemes("क्ष") also returns ["क्ष"] on OTP 28 and later.
# OTP 27 and earlier predate GB9c and return ["क्", "ष"].

The scripts affected are those whose virama is not InCB=Linker, including Tamil, Kannada, Gurmukhi and Sinhala. If you are counting characters or extracting the first letter of a word in one of those scripts, the difference is visible to your users.

OTP also departs from UAX #29 in three narrower cases, which can arise in any script:

  • A zero-width non-joiner after a virama asks for the virama to be shown instead of a conjunct, so UAX #29 breaks after it. On OTP 28 and later String.graphemes/1 keeps the conjunct together across it.

  • A combining mark or skin tone modifier after an emoji and a zero-width joiner still belongs to the cluster, but String.graphemes/1 starts a new one.

  • A spacing mark between an emoji and a zero-width joiner ends the emoji sequence, so the next emoji starts a new cluster, but String.graphemes/1 joins it to the previous one.

iex> Unicode.String.split("क्\u200Cष", break: :grapheme)
["क्\u200C", "ष"]

iex> Unicode.String.split("👍\u200D🏽", break: :grapheme)
["👍\u200D🏽"]

# On OTP 28 and later String.graphemes/1 returns ["क्\u200Cष"] for the
# first. Every OTP release returns ["👍\u200D", "🏽"] for the second.

The two can also differ because of the Unicode version. This library implements Unicode 18.0, while String.graphemes/1 follows the version bundled with your OTP release, which is 17.0 in OTP 29. Unicode 18.0 revised GB9c so that a virama joins the following consonant even when no consonant precedes it, and added new characters.

Outside these cases String.graphemes/1 is fine.

Emoji

Emoji sequences are the most common reason a grapheme cluster spans many codepoints. A family emoji is several people joined by zero-width joiners, and a skin tone is a modifier attached to the preceding character.

iex> Unicode.String.split("👨‍👩‍👧‍👦", break: :grapheme) |> length()
1

iex> String.length("👨‍👩‍👧‍👦")
1

Flags are pairs of regional indicators, and the rule counts them in pairs, so an odd trailing indicator stands alone:

iex> Unicode.String.split("🇦🇺🇳🇿", break: :grapheme) |> length()
2

Combining marks

A base character and the marks that follow it form one cluster, however many marks there are. Here e is followed by U+0301 COMBINING ACUTE ACCENT — two codepoints, one cluster:

iex> "e" <> <<0x0301::utf8>> |> Unicode.String.split(break: :grapheme) |> length()
1

Normalising with String.normalize/2 would turn that into the single codepoint U+00E9, but segmentation does not require you to — the decomposed form is one grapheme cluster either way.

Performance

Grapheme breaking has an ASCII fast path: two printable ASCII bytes in a row are always a boundary, which is decided without decoding a codepoint or consulting a property table. Text that is mostly Latin costs very little.

If you are segmenting a very large string and only need part of it, stream/2 avoids building the whole list:

iex> "abcdef" |> Unicode.String.stream(break: :grapheme) |> Enum.take(3)
["a", "b", "c"]