Aether Reference

Aether is Ichor's own grammar language. This page documents every feature in detail. If you're new to Aether, read the tutorial first — this page is a reference, not an introduction.

File structure

Every grammar starts with two required pragmas, in order:

@grammar "name"
@root rule_name

@grammar names the grammar (informational). @root names the rule where matching begins. After those, zero or more of the optional pragmas below may follow, in any order, each at most once. Then the grammar's token and rule definitions follow, in any order — a definition may reference a name declared later in the file.

Comments run from ; to the end of the line.

Tokens vs. rules

This is the one syntactic distinction Aether's entire design hinges on:

• A name in ALL_CAPS (matching [A-Z_][A-Z0-9_]*) declares a token. Tokens are matched by the lexer, directly against raw characters.
• A name in snake-case or kebab-case (matching [a-z][a-z0-9_-]*) declares a rule. Rules are matched by the parser, over the token stream the lexer already produced — never over raw characters directly.

Every Aether grammar compiles to a genuine two-stage Lexer -> Parser, never a single scannerless pass. This has real consequences:

• A rule body may only reference tokens and other rules — never an inline character class, /pattern/ regex literal, or bare . ("any character"). Those are lexer-only constructs; give the pattern a name as a token instead, and reference that token from the rule.
• A token body may reference other tokens, but never a rule.
• Writing a bare string literal ("SELECT") directly inside a rule body is allowed as a convenience — Aether automatically promotes it to a compiler-generated anonymous token behind the scenes, since only tokens ever participate in lexing. The same literal text (with the same case-sensitivity) reuses the same anonymous token wherever it appears.
• A bare string literal directly inside a token body is just an ordinary sub-match — no promotion needed, since it's already inside a token.

Maximal munch

At every position, the lexer tries every token the grammar's rules can actually reach (directly referenced by some rule, or the active @skip token) and picks the longest match. Ties are broken by declaration order — the token declared first in the file wins. A token that can match zero characters is never accepted as "the next token" (it would never advance, and tokenizing would never finish); such a token can still exist and be referenced from inside another token's own definition, it just can never win the top-level race on its own.

This is why declaration order sometimes matters even between tokens that look unrelated: if two tokens can match the same text at the same position with the same length, whichever is declared first in the file wins.

Rule parsing: ordered PEG choice

Once tokenized, rules are matched using ordinary PEG semantics: within a first | second | third, the first alternative that matches wins, even if a later alternative could also have matched (possibly matching more, or less). There's no most-specific-wins or longest-match rule at the parser level the way there is at the lexer level — order in the source is the only thing that decides among rule alternatives, so put more specific alternatives first.

Expression syntax

From lowest to highest precedence:

choice      := sequence ("|" sequence)*
sequence    := term+
term        := ("~")? (name ":")? postfix
postfix     := ("&" | "!")? primary ("*" | "+" | "?" | "{" bound "}")?
primary     := STRING | CHAR_CLASS | REGEX | "." | UPPER_IDENT | lower_ident
             | "(" choice ")" | "@indent(" choice ")" | "@samecol(" choice ")"

Quantifiers, predicates, and grouping all work identically in both token and rule bodies (subject to the token/rule leaf restrictions above).

Whitespace: @skip / @noskip / ~

By default (no @skip/@noskip pragma given at all), every rule's sequence elements — after the first — get an automatic SPACE* spliced in front of them, where SPACE is the built-in predefined token (overridable). This is what lets expr := term "+" term tolerate "1 + 2" and "1+2" alike without writing whitespace tolerance into every single rule by hand.

• @skip TOKEN uses a different token in place of SPACE — useful when comments should also be treated as skippable trivia (a common pattern: declare a TRIVIA := (SPACE | COMMENT)* token and @skip TRIVIA).
• @noskip disables splicing entirely. Required for any grammar where whitespace is meaningful syntax, not filler — HTTP headers, regex patterns, Markdown.
• ~name (only valid in rule bodies, only directly prefixing a bare token or rule reference with no capture and no quantifier) suppresses splicing before that one specific term, for the rare case where two adjacent things must not have skippable trivia between them even though the rest of the rule does.

Splicing also applies to the repeated body of */+/{n,m} — form* tolerates whitespace between repetitions, not just around the whole group, since each repetition is itself another sequence element.

Predefined tokens

Five tokens exist in every grammar even if never declared:

Any of the five may be redeclared with NAME := ..., as long as the redeclaration happens before the token is used anywhere in the file (including implicitly, e.g. @skip's own default use of SPACE, or a POSIX bracket class referencing one of the other four). Using one first and then trying to redeclare it is a compile error — the two orderings would otherwise mean different things depending on where in the file the use occurred, which Aether rejects outright rather than resolving by File order.

POSIX bracket classes

Inside a character class ([...]), [:alpha:], [:alnum:], [:digit:], [:space:], and [:hex:] reference the identically-named predefined token, and can be freely mixed with ordinary ranges and characters in the same brackets:

IDENT := [[:alpha:]_][[:alnum:]_]*

Character classes

[...] matches one character from the given set; [^...] negates it (matches one character not in the set — but still requires a character to be present; it never matches at end of input). Contents are a mix of single characters, a-z-style ranges, POSIX names, and escape sequences. A - immediately before the closing ] is a literal dash, not a range operator.

String literals

Double-quoted, with these escapes: \n \r \t \\ \" \[ \] \/, \xHH (exactly two hex digits), \u{H+} (one or more hex digits), and any other punctuation character escapes to itself (\^, \-, \*, and so on) — convenient for writing character-class-like escapes consistently in both contexts.

A quote suffix controls case sensitivity for that one literal, overriding the grammar-level @case_insensitive setting either way:

• "lit" — case-sensitive, unless @case_insensitive is set for the whole grammar (then case-insensitive).
• "lit"i — always case-insensitive, regardless of the grammar-level setting.
• "lit"cs — always case-sensitive, regardless of the grammar-level setting.

@case_insensitive

A grammar-wide pragma: every bare ("lit", no suffix) string literal matches either case. Doesn't affect character classes or [:posix:] classes — only quoted string literals.

Regex literals

Token bodies only (never valid in a rule body) — /pattern/ is a convenience shorthand that desugars entirely into ordinary Aether primitives before reaching Grammar.IR. It is not a real regex engine wired in underneath; support is deliberately partial:

Supported: literal characters, . (any character), character classes ([...]/[^...], POSIX names included), alternation (|), grouping ((...)), quantifiers (* + ? {n} {n,} {n,m}), positive/ negative lookahead ((?=...) / (?!...)), and the shorthand classes \d \w \s \h (referencing DIGIT/ALNUM/SPACE/HEX) with their negated forms \D \W \S \H.

Explicitly rejected, with a normal Ichor.Error rather than silently doing the wrong thing: anchors (^ $), backreferences (\1-\9), and named/unnamed capturing groups ((?<name>...)) — none of these have a meaningful desugaring into PEG-style matching primitives, so Aether refuses them outright instead of pretending to support them.

NUMBER := /\d+(\.\d+)?/

Indentation sensitivity

Two pragmas, valid only inside rule bodies, for layout-sensitive grammars (YAML-style structure) without requiring a scannerless parser to special-case columns everywhere:

• @indent(expr) — expr must start at a column strictly greater than the currently active reference column, and its own starting column becomes the new reference column for anything nested inside it.
• @samecol(expr) — expr must start at exactly the currently active reference column (no descending further).

Both also accept a single bare term with no parens as sugar for wrapping just that one term: @samecol pair is shorthand for @samecol(pair).

@samecol always checks against the nearest enclosing @indent's reference column — with no enclosing @indent anywhere, the reference column defaults to 0, so @samecol at the very top of a grammar only succeeds on something starting in the first column. In practice @samecol is used nested inside an @indent, as below, not standalone.

mapping := pair (NEWLINE @samecol pair)*
pair     := SCALAR COLON (inline_value | NEWLINE @indent(block_value))

Named captures

name:expr captures expr's matched value under name, available to an Ichor.Actions module as captures.name (as a %{name => Ichor.Capture.t()} — see the tutorial for how Actions modules consume these). A bare reference to a named token or rule (no explicit name: prefix) is implicitly captured under its own name automatically — factor in expr := term (op:("+" | "-") term)* needs no term:term, it already shows up as :term. A bare reference to a compiler-generated anonymous token (an inline literal like "(", never a name the grammar author chose) gets no implicit capture — writing a bare literal is precisely how a grammar author says "I don't care about matching this," and an explicit paren:"(" still works if that literal's value actually is wanted.

A capture whose expression sits under a */+/{n,m} anywhere in the rule always arrives as a list in Ichor.Actions, even if it matched zero or exactly one time — never a bare value, and never a missing key, so pattern-matching on capture shape is reliable regardless of how many times something actually matched.

@engine

A grammar-wide pragma picking the parsing algorithm for @root:

@engine peg   ; the default -- ordered-choice recursive descent
@engine lr    ; deterministic shift-reduce, requires a conflict-free grammar
@engine glr   ; Tomita-style GLR, forks at genuine ambiguity

Independent of which backend runs the grammar (Grammar.VM or Grammar.Native, i.e. use Ichor) — both support all three engines. peg (the default, so every grammar written before @engine existed keeps behaving identically) is ordinary top-down recursive descent: the first matching alternative wins, and left recursion is rewritten away by Grammar.Analysis automatically. lr/glr are bottom-up shift-reduce parsers built from a from-scratch SLR(1) table instead — the natural fit for left-recursive rules (expr := expr "+" term | term, not the PEG-idiomatic term ("+" term)*), which PEG can't parse at all but bottom-up parsing handles natively. Grammar.Analysis skips its left-recursion rewrite entirely for lr/glr grammars, since it would undo the very shape they want.

• lr requires the resulting table be conflict-free — an ambiguous or genuinely context-free (non-regular-per-rule) grammar is rejected at compile/analysis time, naming the conflicting state and productions, rather than silently picking one arbitrarily.
• glr accepts conflicts and forks at them, exploring every viable parse in parallel via a graph-structured stack that shares/merges history at points where forks reconverge — this is what recovers inputs plain PEG's greedy, never-reconsidered commit can't (a rule := "ab" | "a" alternative that "wins" too early, with nothing left over for what follows), and what lets a grammar express genuine ambiguity (natural-language-style, or the classic dangling-else problem) instead of being forced to silently favor one reading. When more than one derivation survives to the end of input, the earliest point the derivations disagree decides the winner, by declared order — same convention ordinary PEG choice already uses (whichever alternative/production was written first in the source wins), with one refinement: at a conflict cell offering both a shift and a reduce, the shift always wins (matching yacc/bison's own default), which is what makes the dangling-else case resolve the way every real language resolves it (else binds to the nearest open if) with no special-casing anywhere.

Only Seq/Choice/Star/Plus/Opt/Rep/RuleRef/Capture are supported inside an lr/glr rule — &/! lookahead predicates, @indent/@samecol, and rule-position @native(...) all need arbitrary backtracking or side-channel state a shift-reduce table has no room for, and are rejected with a build-time error naming the rule and construct. Token-level lexing (including token-position @native(...)) is unaffected either way — lr/glr still parse the same Tokenizer/Lexer-produced token stream a peg grammar would.

@grammar "calculator"
@root expr
@engine lr

NUMBER := /\d+(\.\d+)?/
PLUS   := "+"

expr := expr PLUS term | term
term  := NUMBER

See Grammar.LR/Grammar.GLR (interpreted) and Grammar.Native.LR/ Grammar.Native.GLR (compiled) for the engines themselves, and Grammar.LRTable for the shared SLR(1) table construction both draw on.

@keywords / @refine

Reclassify a token after the Tokenizer matches it, before the Parser ever sees it — for a lexical distinction maximal munch alone can't make, because it depends on the token's own text (a keyword vs. an ordinary identifier with the same shape) or on what came immediately before it (JavaScript's /, which starts either a regex literal or a division operator depending on the preceding token).

@keywords is sugar for the common table-lookup case:

WORD := [a-zA-Z_][a-zA-Z0-9_]*
@keywords WORD { "if" -> KEYWORD_IF, "return" -> KEYWORD_RETURN }

Every WORD token whose exact text matches a key in the table is renamed to the corresponding target token (KEYWORD_IF, KEYWORD_RETURN); anything else stays WORD. The target names never need their own TOKEN := ... declaration — @keywords is what tells Grammar.Analysis and the lexable-candidate-set computation they exist.

@refine("Module", "function", ...) is the escape hatch for anything needing real logic instead of a fixed table — attached directly to a token's own definition:

SLASH := "/" @refine("JS.SlashDisambiguator", "refine", DIV, REGEX_START)

The callback (Ichor.TokenRefiner.refine/4: raw_name, raw_text, pos :: {line, column}, and preceding — every token already reclassified so far, in final form, which is what makes the lookbehind-dependent JS case work) returns {:ok, new_name, value} (the new token name plus a capture-value override, e.g. a decoded escape sequence reaching Ichor.Actions instead of raw source text) or {:error, reason}, surfaced as an Ichor.Error with stage: :lexer.

@native / @hint: the escape hatch

For the rare case a grammar's own meaning is mutated mid-file by something declared earlier — Prolog's runtime-extensible operator table (op/3), Haskell fixity declarations, a heredoc whose terminator is read off the source itself, a string literal with embedded interpolated expressions — no static grammar (PEG, LR, or GLR) can express the rule or token directly. @native("Module", "function", dep, ...) hands that one rule or token to hand-written Elixir instead.

At rule position (Grammar.IR.Custom)

expr := @native("Prolog.Grammar", "parse_term", primary) @hint(nullable: false, leading: (primary))

deps (primary above) lists the only other rules the callback is allowed to call back into, via a rule_matchers map restricted to exactly those names — kept explicit so Grammar.Analysis's reference check still sees this node's real dependencies even though its own body is opaque Elixir code. The callback (matching Ichor.CustomRule.match/4, though it doesn't have to be named match — @native's own second argument names it explicitly) is:

@callback match(stream, pos, context, rule_matchers) ::
            {:ok, new_pos, Ichor.Capture.node_t()} | :fail

• stream/pos — the same compiled token stream and position any ordinary rule matcher works over.
• context — read-only: whatever the previous top-level form's own evaluation produced (via Grammar.VM.run_sequence/4/ Grammar.Native's generated run_sequence/2), or initial_context outside a sequence. This is how a :- op(700, xfx, is). directive parsed earlier in the same file can extend the operator table a later clause's own @native callback parses against.

• rule_matchers — %{primary: (stream, pos -> {:ok, new_pos, node} | :fail)} for the example above: call straight back into the grammar's own compiled primary rule.

@hint(nullable: bool, leading: (dep, ...)) supplies the two facts Grammar.Analysis would otherwise compute by walking the rule's own structure, impossible here since the body is arbitrary code: nullable defaults to false (most custom recognizers consume something), leading defaults to every declared dependency (conservative, for left-recursion-cycle detection). Ichor.Toolkit.Pratt is built exactly for writing this kind of callback — precedence-climbing over a runtime-mutable operator table is the single most common reason to reach for rule-position @native at all.

At token position (Grammar.IR.CustomLexeme)

STRING := @native("JS.StringInterp", "scan", expr) @hint(nullable: false)

Always stands alone as an entire token definition — it can't be referenced from inside another token's body, and can't be composed inside a larger token expression. deps here names rules (never other tokens — the one narrow exception to "a token body may only reference other tokens"), reachable through a re-lex-from-a-character-position primitive rather than the token-stream-based rule_matchers rule position gets:

@callback scan(input, context, rule_matchers) ::
            {:ok, text, rest, Ichor.Capture.node_t() | nil} | :fail

• input — the remaining suffix of the source text (not a position into some original string).
• context — same meaning as Ichor.CustomRule.match/4's.

• rule_matchers — %{expr: (input -> {:ok, text, rest, capture} | :fail)} for the example above: re-lexes input from scratch and matches the named rule, for a token that needs to recurse into full rule-level parsing mid-scan (an embedded #{expr} inside a string literal) — something a fixed maximal-munch tokenizer has no way to express on its own.

Returns {:ok, text, rest, capture} with text <> rest == input. capture is normally nil (this token behaves like any other — a plain {:token, name, text} downstream); a string-interpolation token overrides it with an explicit Ichor.Capture.node_t() instead, so its embedded expressions' own parsed structure reaches Ichor.Actions intact rather than collapsing into flat text. @hint(nullable: bool) is the only hint here (false by default) — there's no leading, since left-recursion-cycle detection is a rule-level concept only.