Changelog

Copy Markdown View Source

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

[0.7.2] - 2026-10-23

Fixed

  • @engine lr accepts an optional element between two others under @skip: "[" WORD* "]" was refused as a conflict on the skip token, because the spliced SPACE* before and after an empty WORD* stood side by side. Under @engine lr and glr the skip token is now dropped from the stream before parsing instead of being spliced into the rules, by both backends and in lossless mode, which keeps every byte as before. The same input is accepted. A rule there naming the skip token itself, and ~ there, are now refused with an error saying why, rather than meaning something a parse cannot do.
  • The install lines in the README and guides name the current versions (ichor_runtime ~> 0.6.1, ichor ~> 0.7); they still said ~> 0.5 and ~> 0.6.

[0.7.1] - 2026-10-21

Fixed

  • A generated parser tokenizes several times faster when a grammar has large character classes, such as Unicode identifier classes: each class is prepared once (Tokenizer.char_class/1, ichor_runtime 0.6.1) instead of being scanned range by range for every character, by the generated parser and by the interpreter alike. Regenerate a checked-in parser to get it; on a 5 KB Dextrin document, tokenizing takes 2.0 ms instead of 12.4 ms.

[0.7.0] - 2026-10-21

Fixed

  • A failed PEG parse says where it went wrong, by both backends and in run_sequence: at the furthest token any alternative reached, with what was found there and what would have been accepted -- unexpected "}" -- expected one of: "[", "]", INT, or unexpected end of input -- ... -- in found and expected. It used to say only that the root did not match, at the first token. What a lookahead failed to match is not counted, and input left over after a complete match is reported at its first token rather than at the trivia before it. Every engine now agrees: LR and GLR already stopped at the failing token.

Changed

  • The README warns, at the top, that this library is under heavy development and not for production use before 1.0.0: things can and will break, and the specification can and will change, between any two releases until then.
  • The warning reaches hex.pm and HexDocs too: the package description starts with it, so search results show it, and HexDocs shows the README's warning as a warning box -- GitHub's > [!WARNING] alert, which ExDoc does not render, is rewritten into ExDoc's own admonition when the docs are built (the docs alias; the README is unchanged).

[0.6.0] - 2026-10-20

Fixed

  • A capture owns the reference it directly wraps, by every engine: in sign:SIGN?, items:item* or n:X{1,3} the reference is captured under the capture's name only. The PEG backends also captured it under its own name, so sign:sign? came out as a list of two -- the rule's node and the text -- and sign:SIGN? handed an action both sign and SIGN. The lossless tree still has a node for the rule.

  • Every engine keeps the captures inside a captured group, as the PEG backends always did: all:(k:A B) gives all, k and B. LR and GLR dropped them, and reported a repeated reference inside one (many:A*) as an empty A: [].

  • The Aether guides: Grammar.VM runs only @engine peg (Grammar.LR and Grammar.GLR run the others, interpreted), and TRIVIA? at the ends of a grammar is no longer needed since every engine trims the skip tokens around the input.

  • An action's error says where, by every engine and both backends, lossless or not, and in run_sequence: an error a handle_rule/3 or handle_token/3 returns with no line is placed where that rule or token matched, and shown with its line and a caret. Every parser records where each capture matched; it used to keep only the text, so the runtime had nothing to place such an error with.

Added

  • Captures and default nodes have spans: an action reads where a capture matched as capture.span, {line, column, length}, and a default %Ichor.Node{} has its rule's.

Changed

  • Needs ichor_runtime 0.5, for the recorded positions (Ichor.Capture.at/3, Grammar.Source.Locator). A PEG parser generated by an earlier ichor still runs on it, without positions; regenerate an LR or GLR one, whose parse/2 would otherwise hand back the recorded positions with its captures.
  • parse/2 of a generated module returns the captures as before, without the recorded positions; a @native(...) rule's or lexeme's rule_matchers hand it plain nodes, as they always have.

[0.5.1] - 2026-10-19

Added

  • mix ichor.conformance answers lossless: the lossless tree of a text, as the Ichor Conformance Suite writes it -- a node as {:rule, name, offset, size, line, column, children}, a token as {:token, name, text, trivia, offset, line, column}, an inline literal's token named nil -- through either backend, so every implementation is held to the same tree, offsets included.

[0.5.0] - 2026-10-19

Added

  • The lossless mode (design/lossless-mode.md): a parser generated with lossless: true (use Ichor, Ichor.generate/4, Ichor.generate_from_grammar/3) or mix ichor.gen --lossless gains parse_lossless/1,2 and run_lossless/1,2. They return the lossless tree of an input (Ichor.Syntax, in ichor_runtime): every byte, trivia and punctuation included, under every grammar rule that matched, with byte offsets, lines and columns; run_lossless returns the ordinary value beside it. For every engine and both backends -- Grammar.VM, Grammar.LR and Grammar.GLR have parse_lossless/3 and run_lossless/4 -- and the same tree from each. Without the option, the generated code is what it was; with it, only the module grows.

Fixed

  • An LR or GLR rule repeated three times or more is a flat list (through ichor_runtime 0.4.0): doc := entry+ with three entries gave [a, [b, c]], which Ichor.Actions could not evaluate.

[0.4.0] - 2026-10-18

Added

  • mix ichor.conformance interpreted|compiled REQUEST ANSWER, this implementation's adapter for the Ichor Conformance Suite (Ichor.Conformance). It reads the suite's JSON request, makes a parser from each case's grammar through the named backend -- Grammar.VM/Grammar.LR/Grammar.GLR, or a module generated by Ichor.generate_from_grammar/2 -- and answers every case with its default tree, or its error's stage, line and column, as the suite's typed JSON. The suite holds each backend to its cases separately, which is what shows the two agree. A case's grammar may be written in any notation Ichor reads -- Aether, ABNF, BNF, ISO EBNF, PEG -- told by its extension or an @style line.
  • Grammar.VM.tokenize/3: a grammar's token stream for an input, interpreted, for every engine -- what a generated module's own tokenize/2 answers. Grammar.LR and Grammar.GLR lex through it.
  • mix ichor.conformance answers the toolkit cases: "pratt", "unify", "solve" and "walk" run Ichor.Toolkit.Pratt, Ichor.Backtrack.Bindings, Ichor.Backtrack.Tree and Ichor.Toolkit.TermWalk on data a case gives, through one standard set of callbacks and one term module (Ichor.Conformance.Toolkits).
  • mix ichor.conformance answers "render": the first error exactly as Ichor.Error.format/1 writes it, or the tree when the text parses.
  • mix ichor.conformance answers "tokens" as well as "parse": the token stream, trivia included, as {name, text, line, column}, an inline literal's generated token named nil.
  • W3C EBNF can be imported (:ebnf_w3c), as the other four notations always could: Ichor.EBNF.W3C read it but Ichor.GrammarImport did not offer it, which the Praxis port already called an oversight. A file says which EBNF it is by .w3c.ebnf or .iso.ebnf, or @style ebnf-w3c / @style ebnf-iso; a plain .ebnf is ISO, as before.
  • Ichor.GrammarImport.styled/2: which notation a grammar file is written in, by its @style line or its extension, and the source to hand that notation's reader. mix ichor.gen and the conformance adapter both use it, so they never disagree about what a file is.

Changed

  • Licensed under the Apache License 2.0 (Apache-2.0); it was MIT.
  • A rule that captures nothing is the text it matched. Its body is captured as :text -- the input from its first token to its last -- so its default value is that string, and an Actions module finds it in %{text: ...}. It used to be an empty node, and what it matched was lost to whatever the parse was fed to; in an imported ABNF, BNF, EBNF or PEG grammar, which have no capture syntax, every terminal was. Decided by the grammar, never by the input (Grammar.Analysis). A generated parser changes when it is regenerated: its capture-less rules come out as text, where they were empty nodes.

Fixed

  • Trivia around the input is skipped by every engine and both backends. run/2 refused a leading @skip token that run_sequence/2 skipped, so " 2" parsed through one and not the other, and an @engine lr grammar refused trailing trivia that PEG and GLR accepted. The stream is now trimmed of @skip tokens before its first other token and after its last (Grammar.Source.trimmed/2); between two tokens they are the grammar's, as before.
  • Every parse error says where. A root that does not match is placed at the token it was tried at, and "unexpected end of input" at the end of the input -- line 1, column 1 for no input at all. Both used to carry no line or column.
  • Both found by the Ichor Conformance Suite. Needs ichor_runtime 0.3.

[0.3.1] - 2026-10-07

Added

  • guides/aether/AETHER_GRAMMAR.md: Aether's own syntax, written as an Aether grammar. Aether.Reader stays hand-written; the guide is a description, and test/aether/grammar_doc_test.exs keeps it honest by compiling the guide's grammar and checking it accepts exactly what Aether.Reader accepts, across every .aether file in the repository.
  • Guides for the two Mix tasks, guides/mix_tasks/ICHOR_GEN.md and ICHOR_TOKENS.md, in a "Mix Tasks" group of their own in the docs.
  • The package links its GitHub Pages documentation beside the repository.

Fixed

  • A name: capture inside a token body is now a reader error at the capture's name: "captures are only valid in rule bodies". It used to be accepted, after which Grammar.VM crashed with a FunctionClauseError on the first match. Captures were always documented as rule-only.
  • References to types in the docs resolve. They were written as if they named functions, and ExDoc warned that no such function exists; they carry the t: prefix now.

Changed

  • mix.lock is no longer checked in, as befits a library: it only ever pinned what this repository's own CI resolved, never what a user gets. CI runs Elixir 1.19.5.

[0.3.0] - 2026-08-03

Changed

[0.2.1] - 2026-07-29

Changed

  • Documentation reframed around three ways to run a grammar, with mix ichor.gen established as the recommended default for anything shipping to production: README gained a "Three ways to run a grammar" section (mix ichor.gen / use Ichor / the raw Aether.Parser + Grammar.Analysis + Grammar.VM pipeline) spelling out the concrete reason for that recommendation -- use Ichor needs the real Ichor module present at compile time in whatever environment you compile in, so ichor can't be only: :dev, runtime: false in any app that still has one, including a mix release build (which normally compiles under MIX_ENV=prod); mix ichor.gen output has no macro dependency on ichor at all, so it's the only path that actually allows the dev-only split ichor_runtime exists to enable. The tutorial gained a new §8, "Loading a grammar at runtime", with a worked plugin-loader example for the third path (previously only mentioned in passing, never actually demonstrated); the cheatsheet gained a "Which one do I want?" comparison table and was reordered to match; the Ichor/Mix.Tasks.Ichor.Gen/Grammar.VM moduledocs now name and cross-reference all three paths for readers landing there directly via ExDoc/hex search.

Fixed

  • Three pre-existing broken anchor links across the guides (guides/ TUTORIAL.md, guides/CHEATSHEET.md, guides/aether/TUTORIAL.md, guides/EXAMPLES.md) -- ExDoc's markdown renderer turns a period inside a backtick-quoted heading (`mix ichor.gen`, `Ichor.Actions`) into an extra hyphen when generating the heading's own anchor id (mix-ichor-gen, ichor-actions), not just stripping it, which none of the existing cross-reference links accounted for. ExDoc doesn't validate these at build time, so they'd been silently dead since introduced.

[0.2.0] - 2026-07-29

Added

  • mix ichor.gen: runs the parse/analyze/native-codegen pipeline ahead of time and writes the result to a plain .ex file, instead of use Ichor re-running it inside a macro expansion on every compile. Every generated helper function is emitted as def rather than defp (macro-originated code is exempt from Elixir's unused-function check; written out as ordinary source, it isn't). Ichor.generate/3 is the new public entry point both __using__/1 and the task share.
  • Ichor.GrammarImport: turns an ABNF/BNF/ISO-EBNF/PEG ruleset (as Ichor.ABNF/Ichor.BNF/Ichor.EBNF.ISO/Ichor.PEG produce them -- not a runnable Aether.Grammar on their own) into one that is: picks a root (the source's own first-declared rule, overridable), always @noskip, auto-promotes bare literals/char-classes into synthetic tokens, and -- for ABNF specifically -- fills in whichever of RFC 5234 Appendix B's "core rules" (ALPHA, DIGIT, CRLF, ...) the source references but never defines, restricted to their transitive closure so unreferenced, overlapping core rules (CHAR/OCTET overlapping DIGIT, e.g.) can't win a maximal-munch tie against a token a rule actually meant. mix ichor.gen now accepts any of these formats directly (.abnf/.bnf/.ebnf/.peg, auto-detected from the file extension, or forced with an @style pragma line), with a new --root NAME flag for the non-Aether entry-rule override.
  • ichor_runtime: split out of ichor proper -- the ~24 modules a generated parser (from mix ichor.gen or use Ichor) actually calls at runtime (Ichor.Actions, Ichor.Error, the compiled Tokenizer/Parser combinators, the LR/GLR shift-reduce/GSS runtime, and standalone Ichor.Toolkit.Pratt/TermWalk/Ichor.Backtrack), with everything dev-time-only (the Aether front-end, format importers, Grammar.Analysis, the LR/GLR table builder, both codegen backends) staying in ichor. Grammar.LRTable was split in the process: the struct plus current_terminal/3/unexpected_error/2 (genuinely called at match time) moved to ichor_runtime; table construction (build/1/conflicts/1, which need Automaton/Desugar/Sets) stays in ichor as the new Grammar.LRTable.Builder. Independently published (ichor_runtime 0.1.0) and maintained -- its own repository, its own release cadence -- not a subdirectory of this one; ichor depends on it via a normal ~> 0.1.0 Hex version constraint (its own interpreted backends need it too).

Fixed

  • evaluate_node/3 moved from the top-level Ichor module to Ichor.Actions.evaluate_node/3 -- found while migrating a real downstream consumer (dextrin) to the ichor/ichor_runtime split: it's a genuine runtime entry point (any grammar with macro-like features needs it to evaluate its own expansion, not just LISP's worked example), not a compile-time-only concern like generate/3/ __using__/1, which are the only two things that actually belong in the dev-only ichor package. Left in Ichor proper, it would have been unavailable at runtime to anything depending on ichor_runtime alone, exactly as the split intends.
  • Ichor.Toolkit.Pratt, Ichor.Toolkit.TermWalk, and Ichor.Backtrack (+ .Bindings/.Term/.Tree) moved from ichor to ichor_runtime -- the same class of gap as evaluate_node/3 above, found migrating another real downstream consumer (Aletheia): a @native(...) rule's own callback module (Prolog-style operator-precedence parsing, the worked example this whole feature is motivated by -- see test/prolog/) calls Ichor.Toolkit.Pratt.parse/4 on every single parse, not just once at grammar-generation time, and a consuming engine built on Ichor.Backtrack calls into Bindings/Tree on every resolution step. None of the three have any dev-only dependency of their own (Pratt operates on a plain map, Backtrack is self-contained, TermWalk only needs Backtrack.Term), so keeping them in the dev-only package was an oversight in the original split, not a deliberate exclusion. Ichor.Toolkit.TypeScheme (which stays in ichor -- it's a compile-time type-checking concern) depends on both Bindings and TermWalk; unaffected by the move since ichor already depends on ichor_runtime.

[0.1.1] - 2026-07-28

Added

  • @native(...)/@hint(...) at rule position (Grammar.IR.Custom): an escape hatch for grammar rules plain PEG can't express — most notably a runtime-mutable operator-precedence table (Prolog's op/3, Haskell fixity declarations, Coq/Agda/Lean mixfix notation), dispatching to an author-supplied Ichor.CustomRule.match/4-shaped callback with access to the token stream, the current position, the grammar's own context (read-only, threaded via run_sequence/4 across top-level forms), and rule_matchers restricted to the node's own declared dependencies.
  • @native(...)/@hint(...) at token position (Grammar.IR.CustomLexeme/Ichor.CustomLexeme): the analogous escape hatch for lexing that fixed maximal-munch tokenization can't express — heredocs (the terminator is read off the source itself), string interpolation (a token needing to recurse into a rule mid-scan), and in principle TeX-style \catcode mutation. Includes a new "re-lex and match rule R starting at a given character position" primitive (rule_matchers.name.(input, char_pos)), supported by both backends.

  • @engine peg | lr | glr pragma (default peg, fully backward compatible): selects the parsing algorithm for @root, independent of execution strategy (interpreted vs. compiled):

    • Grammar.LR (and Grammar.Native.LR for compiled codegen): a from-scratch SLR(1) table (Grammar.LRTable) plus a deterministic shift-reduce engine, for grammars an author has already verified are conflict-free — zero GSS/backtracking bookkeeping, and (compiled) fully inlined into one Elixir function per automaton state.
    • Grammar.GLR (and Grammar.Native.GLR): genuine Tomita-style GLR over the same SLR(1) table, forking at every real conflict via a graph-structured stack (Grammar.GLR.GSS) that shares/merges history instead of naively duplicating parallel stacks, resolving multiple valid derivations via a declared-order tie-break (Grammar.LRTable.Automaton's own canonical "shift before reduce, reduces by production id" conflict-cell ordering) — this is what lets it parse inputs plain PEG's greedy, non-backtracking commit can never recover from, and pick a deterministic winner for genuine CFG-level ambiguity a PEG grammar could never even express.
    • Grammar.Analysis skips its PEG-specific left-recursion rewrite for non-peg grammars (left recursion is what bottom-up shift-reduce parsing wants, not a hazard); both backends guard against running a mismatched-@engine grammar through the wrong entry point.
  • @keywords / @refine: token-stream reclassification after tokenization, before parsing (Grammar.Lexer.reclassify/2) — @keywords WORD { "if" -> KEYWORD_IF, ... } is table-lookup sugar for the general case, @refine(Module, :function) dispatches to an author-supplied callback with the already-reclassified stream so far (enabling lookbehind-dependent reclassification, e.g. JavaScript's regex-literal-vs-division-operator ambiguity).
  • Ichor.Backtrack: a generic lazy-search-plus-unification substrate for logic-language evaluation models (SLD-resolution and friends) — Ichor.Backtrack.Tree (unit/fail/disjunction/ conjunction/once, pulled lazily via next/1, depth-first/ left-to-right like Prolog's own clause-order backtracking), Ichor.Backtrack.Bindings (a substitution: bind/resolve/unify/ unify_occurs_check, over any Ichor.Backtrack.Term implementation an author supplies for their own term representation).
  • Ichor.Toolkit, small IR-agnostic building blocks extracted from patterns Ichor's own compiler internals had already independently hand-rolled more than once — useful to a grammar author's own downstream semantic analysis/type inference/codegen, never run by Ichor itself:
    • Fixpoint/Graph — "iterate a step function to convergence," and reachability/cycle detection over an arbitrary graph.
    • Scope — a generic nested lexical scope/symbol table.
    • TypeScheme — Hindley-Milner-style let-polymorphism (generalize/instantiate/resolve_deep) built on Ichor.Backtrack.Bindings.unify_occurs_check/4.
    • Codegen — small quote/unquote mechanics (fresh/vars/ indexed_vars/capture/clause) for authors hand-writing their own Elixir-codegen backend.
    • Result — reduce_ok/map_ok, an Enum.reduce-with-early-exit- on-failure pair for the "walk a collection, thread an accumulator through a fallible step" idiom.
    • Pratt — precedence-climbing (Pratt) expression parsing over a runtime-mutable operator table (prefix/infix/postfix, including the genuinely ambiguous case of an operator registered as both infix and postfix at once).
    • Layout — the Python/Haskell/F#-style off-side-rule (INDENT/ DEDENT token synthesis) algorithm, an alternative strategy to Aether's own @indent/@samecol.
    • TermWalk — generic fold/rewrite recursion over any Ichor.Backtrack.Term implementation.
  • Native GLR/LR codegen (Grammar.Native.LR/Grammar.Native.GLR): the SLR(1) table is built once, at Elixir-compile-time, instead of on every call; Grammar.Native.LR fully inlines shift/reduce dispatch into one Elixir function per automaton state (a genuine Bison-style compiled parser), Grammar.Native.GLR compiles the action/goto lookup tables while sharing one runtime GSS engine (Grammar.GLR.Runtime) with the interpreted backend.
  • Five more worked-example grammars, chosen specifically to close gaps where a feature had only ever been exercised by a synthetic/textbook fixture, never anything relating to a real programming language, file format, or protocol:
    • Prolog (test/prolog/) — real clause/rule/directive syntax (facts, rules with conjunction, :- op(Prec, Type, Name). directives genuinely mutating the operator table mid-file via Ichor.Toolkit.Pratt + run_sequence/4's context threading), closing the loop with Ichor.Backtrack: a parsed fact unifies directly against a hand-built query term through the same substrate.
    • The classic dangling-else ambiguity (test/dangling_else/) — real, present in C/Java/JavaScript/PHP, resolved correctly (else binds to the nearest if) with no special-casing needed.
    • CommonMark thematic breaks (--- -> <hr>, added to the existing Markdown grammar) — the first real use of &expr (Grammar.IR.AndPred) anywhere in the test suite.
    • A classic-BNF LISP fixture (test/lisp/lisp.bnf) — Ichor.BNF's first real-world-named fixture.
    • A W3C-EBNF XML subset (test/xml/) — Ichor.EBNF.W3C's first real-world-named fixture, keeping the real XML 1.0 spec's own production names.

Changed

  • Split Aether.Parser into Aether.Reader (pure-syntax parsing of .aether source into a concrete syntax tree, no desugaring) and Aether.Eval (that CST into Grammar.IR/Aether.Grammar, owning predefined-token override/use tracking, case-insensitivity resolution, character-class/regex desugaring, inline-literal promotion, and @skip splicing). Mirrors the Reader/Eval split generated grammar code already has (parse/1 vs. run/1,2) and the ABNF/BNF/EBNF/PEG importers already inherit from it. Aether.Parser.parse/2 remains the combined entry point; observable behavior is unchanged.
  • Split the tokenizer/lexer/parser pipeline into four explicit stages, shared identically by both backends: Grammar.Source (UTF-8 validation), Grammar.VM.Tokenizer/Grammar.Native.Runtime.Tokenizer (renamed from Lexer/Grammar.Native.Runtime — maximal-munch char-level matching), Grammar.Lexer (new — @keywords/@refine reclassification, shared by both backends since it's pure post-tokenization data processing), and the existing token-level Parser stage (Grammar.Native.Runtime.Parser, split out of Grammar.Native.Runtime).
  • Aether.Grammar gained rule_order (mirroring the existing token_order) and an engine field (:peg by default).
  • Renamed LICENSE.txt to LICENSE.

[0.1.0] - 2026-07-26

Added

  • Aether, Ichor's own grammar language: ALL_CAPS tokens matched by maximal munch, snake-case/kebab-case rules matched by ordered PEG choice, string/character-class/regex-literal primitives, quantifiers (*/+/?/{n,m}), lookahead predicates (&/!), named captures, automatic @skip whitespace-splicing (with @noskip and ~ to opt out), @case_insensitive, POSIX bracket classes, and @indent/ @samecol for layout-sensitive grammars.
  • Grammar.IR, the normalized AST every front-end compiles to and every backend consumes.
  • Grammar.Analysis: reference checks, a direct-left-recursion rewrite, an empty-repetition hazard check, and a duplicate-alternative lint.
  • Grammar.VM, an interpreted bytecode backend (LPeg-style compiled Lexer + Parser).
  • Grammar.Native, a compile-time codegen backend (use Ichor), generating direct Elixir function calls instead of bytecode.
  • Ichor.Actions, the behaviour connecting a parsed grammar to real evaluation, transpilation, or execution: handle_rule/3, handle_token/3, optional finalize/1, and a default fallback for anything a grammar's own Actions module doesn't implement.
  • Format importers reading ABNF (RFC 5234 + RFC 7405), classical BNF (ALGOL 60 Report convention), ISO/IEC 14977 EBNF, W3C-style EBNF (the notation the XML 1.0 spec's own section 6 uses), and PEG (Ford's paper / pest / PEG.js convention) — each producing the same Grammar.IR-based ruleset shape every other front-end produces.
  • mix ichor.tokens, listing every token a grammar declares in maximal-munch tie-break order.
  • Nine worked-example grammars exercising the library end to end: a calculator, a LISP dialect, YAML, LogQL, SQL, HTTP, regex, Forth, and a Markdown-to-HTML transpiler.
  • Cross-format validation: six of those worked examples (calculator, LISP, LogQL, SQL, HTTP, regex) independently re-expressed in ABNF, ISO EBNF, and PEG, verified to produce equivalent tokens and identical accept/reject behavior to their native Aether counterparts.