Search documentation of Localize

# Localize.Message.Highlighter (Localize v0.39.0)

Copy Markdown View Source

Produces a classified token stream from an MF2 AST for syntax highlighting.

The output is a flat list of `{class :: atom, text :: String.t()}` tuples. Each tuple represents one syntactic region of the canonical message — punctuation, a variable reference, a function name, a literal, plain text, etc. Formatters (`Localize.Message.Formatter.HTML`, `Localize.Message.Formatter.ANSI`, `Localize.Message.Formatter.Plain`) consume the token stream and produce rendered output.

Concatenating every token's text field yields the canonical MF2 message (the same output as `Localize.Message.Print.to_string/1`) — the `Plain` formatter does exactly that. This property is verified by the test suite.

### Token classes

Classes follow the same taxonomy as tree-sitter highlight captures, so one stylesheet can style both server-rendered HTML and the browser-side `mf2_wasm_editor`. Atoms carry underscores; the HTML formatter converts them to hyphens on emission (so `:punctuation_bracket` becomes `.mf2-punctuation-bracket`).

- `:punctuation_bracket` — `{`, `}`, `{{`, `}}`, `|`, `=`, `/`, whitespace inside expressions, and similar structural characters.

- `:variable` — `$name` (entire token including `$`).

- `:function` — `:number`, `:date`, custom function names (entire token including `:`).

- `:keyword` — `.input`, `.local`, `.match` keywords.

- `:tag` — markup tag names (`bold` in `{#bold}`).

- `:attribute` — attribute names (`@translate`).

- `:property` — option names (`style` in `style=|short|`).

- `:string` — quoted literal content including the `|` delimiters.

- `:number` — numeric literals (integer and float).

- `:text` — pattern text (plain message content).

- `:string_escape` — `\{`, `\|`, `\\` escape sequences.

- `:constant_builtin` — `*` (catchall variant key).

## Summary

### Types

class()
token()

### Functions

to_tokens(ast)

Walks the AST and returns a list of classified tokens.

## Types

# class()

    @type class() ::
      :punctuation_bracket
      | :variable
      | :function
      | :keyword
      | :tag
      | :attribute
      | :property
      | :string
      | :number
      | :text
      | :string_escape
      | :constant_builtin

# token()

    @type token() :: {class(), String.t()}

## Functions

# to_tokens(ast)

    @spec to_tokens(list() | tuple()) :: [token()]

Walks the AST and returns a list of classified tokens.

### Arguments

- `ast` is a parsed MF2 AST as returned by `Localize.Message.Parser.parse/1`.

### Returns

- A list of `{class, text}` tuples.