User guide

unicode_set parses Unicode Set expressions such as [[:Lu:]&[:Latn:]] and turns them into things Elixir can use directly: function guards, compiled binary patterns for String.split/3, code point lists for nimble_parsec, and character classes for regular expressions. This guide walks through the expression syntax and each of those uses.

Every function returns {:ok, result} or {:error, {exception, reason}}. Each has a ! variant that returns the bare result or raises Unicode.Set.ParseError.

Installation

Add unicode_set to your dependencies:

def deps do
  [
    {:unicode_set, "~> 1.9"}
  ]
end

The library depends on unicode, which carries the Unicode Character Database. Both follow the current Unicode release, 18.0 at the time of writing.

Writing a Unicode Set

A Unicode Set is a set of code points and, optionally, of strings. It is written between square brackets.

Characters and ranges

iex> Unicode.Set.parse_and_reduce!("[abc]").parsed
{:in, [{97, 99}]}

iex> Unicode.Set.parse_and_reduce!("[a-z]").parsed
{:in, [{97, 122}]}

iex> Unicode.Set.parse_and_reduce!("[a-zA-Z0-9_]").parsed
{:in, [{48, 57}, {65, 90}, {95, 95}, {97, 122}]}

The :parsed field holds the evaluated set as a sorted list of {first, last} code point ranges. Juxtaposed elements are unioned.

Any character can be written literally except the set-syntax characters [ ] { } & - ^ \ $, which must be escaped with a backslash:

iex> Unicode.Set.parse_and_reduce!("[\\[\\]\\-]").parsed
{:in, [{45, 45}, {91, 91}, {93, 93}]}

A hyphen at the start or end of a set is a literal hyphen, so [-a], [a-] and [-] all contain U+002D. [] is the empty set.

Escapes

All the usual escapes are available, and any other backslashed character is itself:

iex> Unicode.Set.parse_and_reduce!("[\\u0041\\x{1F600}\\N{EURO SIGN}\\N{NULL}]").parsed
{:in, [{0, 0}, {65, 65}, {8364, 8364}, {128512, 128512}]}

Character names are matched loosely: case, spaces, hyphens and underscores are ignored, so \N{latin_small_letter_a} and \N{LATIN SMALL LETTER A} are the same character. Name aliases of every kind resolve, which is how \N{NULL}, \N{LF} and \N{BYTE ORDER MARK} work.

Properties

The real power of a Unicode Set is naming characters by their Unicode properties. There are two spellings, POSIX-style [:…:] and Perl-style \p{…}, which mean the same thing:

iex> Unicode.Set.parse_and_reduce!("[:Lu:]").parsed == Unicode.Set.parse_and_reduce!("\\p{Lu}").parsed
true

A bare name is a General_Category value, a Script value or a binary property:

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("\\p{Lu}").parsed
iex> hd(ranges)
{65, 90}

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("\\p{Greek}").parsed
iex> hd(ranges)
{880, 883}

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("\\p{Emoji}").parsed
iex> hd(ranges)
{35, 35}

Any other property is written property=value, and ≠ (U+2260) selects everything else:

iex> Unicode.Set.parse_and_reduce!("\\p{Block=Basic Latin}").parsed
{:in, [{0, 127}]}

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("\\p{Line_Break=OP}").parsed
iex> hd(ranges)
{40, 40}

iex> Unicode.Set.parse_and_reduce!("\\p{gc≠Cn}").parsed == Unicode.Set.parse_and_reduce!("\\P{Cn}").parsed
true

Property and value names are matched loosely too, so \p{General Category=uppercase letter}, \p{gc=Lu} and \p{GC=LU} are all the same query. The General_Category groupings L, LC, M, N, P, S, Z and C work, as do the POSIX names alpha, digit, punct, space and so on.

\P{…} and [:^…:] are the complement of the property, kept in {:not_in, ranges} form:

iex> {:not_in, ranges} = Unicode.Set.parse_and_reduce!("\\P{L}").parsed
iex> hd(ranges)
{65, 90}

Three properties take special values. Age is cumulative, so \p{Age=6.0} is every character that existed in Unicode 6.0. Numeric_Value takes NaN, a fraction or a decimal:

iex> Unicode.Set.parse_and_reduce!("\\p{nv=1/6}").parsed
{:in, [{8537, 8537}, {68087, 68087}, {74849, 74849}, {126269, 126269}]}

And Name and Name_Alias select a single character:

iex> Unicode.Set.parse_and_reduce!("\\p{Name=EURO SIGN}").parsed
{:in, [{8364, 8364}]}

The README lists every supported property. The UTS #61 conformance guide records exactly how the syntax relates to the standard.

Set operations

Sets combine by juxtaposition (union), & (intersection) and - (difference). The operators have equal precedence and bind left to right, so nest brackets to group:

iex> Unicode.Set.parse_and_reduce!("[[a-z][A-Z]]").parsed
{:in, [{65, 90}, {97, 122}]}

iex> Unicode.Set.parse_and_reduce!("[[a-z]&[m-z]]").parsed
{:in, [{109, 122}]}

iex> Unicode.Set.parse_and_reduce!("[[a-z]-[aeiou]]").parsed
{:in, [{98, 100}, {102, 104}, {106, 110}, {112, 116}, {118, 122}]}

iex> Unicode.Set.parse_and_reduce!("[[a-z]-[c]&[d]]").parsed
{:in, [{100, 100}]}

iex> Unicode.Set.parse_and_reduce!("[[a-z]-[[c]&[d]]]").parsed
{:in, [{97, 122}]}

The right-hand side of & or - must be a bracketed set or a property query. The most common use is narrowing a property:

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("[[:Lu:]&[:Greek:]]").parsed
iex> hd(ranges)
{880, 880}

Complement

A leading ^ complements the set:

iex> Unicode.Set.parse_and_reduce!("[^a-z]").parsed
{:not_in, [{97, 122}]}

A complement is kept in {:not_in, ranges} form wherever possible, because guards, regexes and nimble_parsec can all express "not these" directly. It is only expanded into positive ranges when an intersection or difference forces it.

Strings

A set may also contain strings, written in braces. {} is the empty string.

iex> Unicode.Set.parse_and_reduce!("[a{bc}{def}]").parsed
{:in, [{97, 97}, {~c"bc", ~c"bc"}, {~c"def", ~c"def"}]}

Inside braces every character other than \ and } is literal, including spaces and the set-syntax characters. String members appear in the range list as {charlist, charlist} pairs, and are only supported by the functions that can match more than one code point: to_pattern/1, compile_pattern/1, to_regex_string/1 and the non-guard form of match?/2.

White space

White space between elements is ignored, so long expressions can be laid out for reading:

iex> {:in, ranges} = Unicode.Set.parse_and_reduce!("[ [:L:] & [:Latn:] - [a-z] ]").parsed
iex> hd(ranges)
{65, 90}

Parsing

parse/1 parses an expression without evaluating its set operations; parse_and_reduce/1 evaluates them and leaves the range list shown throughout this guide. Both return a Unicode.Set struct:

iex> {:ok, set} = Unicode.Set.parse_and_reduce("[a-z]")
iex> set.set
"[a-z]"
iex> set.parsed
{:in, [{97, 122}]}
iex> set.state
:reduced

Invalid expressions return an error rather than raising:

iex> {:error, {Unicode.Set.ParseError, message}} = Unicode.Set.parse("[:nonesuch:]")
iex> message
"Unable to parse \"[:nonesuch:]\". The unicode script, category or property \"nonesuch\" is not known."

The ~u sigil parses at compile time:

iex> import Unicode.Set.Sigil
iex> ~u"[[:Lu:]&[:thai:]]"
#Unicode.Set<[[:Lu:]&[:thai:]]>

Function guards

Unicode.Set.match?/2 is a macro. In a guard it expands, at compile time, into plain comparisons on the code point, so there is no runtime cost beyond a few integer tests:

defmodule Guards do
  require Unicode.Set

  defguard is_digit(codepoint) when Unicode.Set.match?(codepoint, "[[:Nd:]]")
  defguard is_greek_letter(codepoint) when Unicode.Set.match?(codepoint, "[[:L:]&[:Greek:]]")
end

defmodule Classifier do
  require Guards

  def classify(<<codepoint::utf8, _::binary>>) when Guards.is_digit(codepoint), do: :digit
  def classify(<<codepoint::utf8, _::binary>>) when Guards.is_greek_letter(codepoint), do: :greek
  def classify(_), do: :other
end

The set must be a literal string, because it is parsed when the module compiles. Outside a guard, match?/2 builds a search tree at compile time and consults it at runtime, and there it also accepts sets with string members:

iex> require Unicode.Set
iex> Unicode.Set.match?(?๓, "[[:digit:]]")
true
iex> Unicode.Set.match?(?๓, "[[:digit:]-[:Thai:]]")
false

Binary patterns

to_pattern/1 returns the set as a list of strings, and compile_pattern/1 compiles that list with :binary.compile_pattern/1. Both can be passed to String.split/3, String.replace/3 and String.contains?/2, and the compiled form is the faster of the two:

iex> Unicode.Set.to_pattern!("[0-9]")
["0", "1", "2", "3", "4", "5", "6", "7", "8", "9"]

iex> pattern = Unicode.Set.compile_pattern!("[[:Nd:]]")
iex> String.split("abc1def2ghi3jkl", pattern)
["abc", "def", "ghi", "jkl"]

A pattern enumerates every member of the set, so keep the set small; a complement or a very large class is better handled as a regex. compile_pattern/1 returns an error for a complement, for the empty set and for a set whose only member is the empty string, since binary patterns cannot express those.

NimbleParsec ranges

to_utf8_char/1 returns the set in the form NimbleParsec.utf8_char/1 and its siblings accept: integers, ranges and {:not, …} exclusions:

iex> Unicode.Set.to_utf8_char!("[[^abcd][mnb]]")
[98, 109..110, {:not, 97..100}]

defmodule Combinators do
  import NimbleParsec

  @digit Unicode.Set.to_utf8_char!("[[:Nd:]]")

  defparsec :digit, utf8_char(@digit) |> label("a digit in any script")
end

String members cannot be expressed as a code point list, so to_utf8_char/1 raises ArgumentError for a set that contains them.

Regular expressions

to_regex_string/1 renders the set as a PCRE character class, with string members as an alternation in front of it:

iex> Unicode.Set.to_regex_string!("[a-z{ch}]")
"(?:ch|[\\x{61}-\\x{7A}])"

More useful is Unicode.Regex, whose compile/2, compile!/2 and match?/3 take an ordinary regular expression, expand every Unicode Set in it, and hand the result to Regex. This gives regular expressions the full property syntax, including intersections and differences that PCRE cannot express, and it tracks the current Unicode release rather than the one the VM was built with:

iex> {:ok, regex} = Unicode.Regex.compile("^[[:L:]&[:Greek:]]+$")
iex> Regex.match?(regex, "Ελληνικά")
true
iex> Regex.match?(regex, "Greek")
false

iex> Unicode.Regex.match?("[[:Sc:]]", "$")
true

A set in a regex must be written as a class, so [[:Lu:]] or \p{Lu}, and property names are resolved by this library rather than by PCRE.

Errors

Every parsing function returns {:error, {exception, reason}} for an expression it cannot handle, where exception is the module to raise and reason is a message naming the problem and where it was detected:

iex> Unicode.Set.parse("[z-a]")
{:error,
 {Unicode.Set.ParseError,
  "Unable to parse \"[z-a]\". Character range starts at 122 which is after its end 97. Detected at \"]\"."}}

The ! variants raise Unicode.Set.ParseError with the same message. Because the sets inside match?/2 and ~u are parsed at compile time, an invalid set there is a compile error.

Performance notes

• Everything that can be done at compile time is. match?/2 in a guard, and ~u, cost nothing at runtime beyond the generated comparisons.
• parse_and_reduce/1 resolves property queries against the unicode library's tables, which are compiled into that library, so evaluation is fast but not free; cache the result rather than re-parsing in a loop.
• Prefer compile_pattern/1 over to_pattern/1 when the pattern is reused.
• A regex compiled by Unicode.Regex.compile/2 is an ordinary Regex and can be stored in a module attribute.