Logos.
Copy Markdown
The public reader: turns Logos source text into plain Logos.Form.t()
data, via priv/grammar/logos.aether (an Aether grammar) and
Logos.Reader.Actions (pure reification, see that module's docs).
Delegates to Logos.Reader.Generated (@moduledoc false, so it's
intentionally absent from these docs), a plain .ex file checked into
the repo and produced ahead of time by mix ichor.gen (see mix.exs
for the exact regeneration command) rather than spliced in at compile
time via use Ichor. This is what lets logos depend on the small
ichor_runtime package at runtime and keep the full ichor compiler
(grammar parsing, analysis, codegen) a only: :dev, runtime: false
dependency -- ichor itself never ships, including in a mix release
build. Regenerate that module whenever
priv/grammar/logos.aether changes; nothing here needs to change in
step with it since only run_sequence/2's and tokenize/2's
signatures are depended on, and both are stable across any grammar
edit that doesn't change the reader's own public API.
Why read/1/read_all/1 are both built on run_sequence/2
The grammar's @root can only name one rule (form), and a bare
run/2 call (built on parse/1) only matches the root rule once, at
position 0, with no leading/trailing trivia handling at all -- great
for something like a calculator grammar where the whole input really
is one expression, wrong for a Lisp source file that's a sequence of
top-level forms separated (and surrounded) by whitespace/comments.
run_sequence/2 (already generated by Grammar.Native, built for
exactly this "load a file one top-level form at a time" case) handles
leading, trailing, and inter-form trivia correctly by calling
Runtime.skip_leading_trivia/3 before every match attempt, including
the first and the one that discovers there's nothing left. Rather than
add a second @root (Aether only allows one) or a sibling use Ichor
module just for a program := form* rule, both read/1 and
read_all/1 are thin wrappers over the same run_sequence/2: read/1
takes just the first form (mirroring Clojure's read-string, which
reads one form and ignores anything after it); read_all/1 returns
every form.
Both filter out Logos.Reader.Actions.discard/0 sentinels -- a
top-level #_form datum comment must vanish from the result, not show
up as a phantom top-level form.
tokenize/1
Logos.Format and Logos.Repl need the raw, position-preserving
token stream (including :TRIVIA, i.e. comments/whitespace) rather
than reified Logos.Form.t() data, so tokenize/1 re-exports the
generated module's own tokenize/2 directly -- see Logos.Format's
own moduledoc for why.
Summary
Functions
Reads exactly the first top-level form out of source; anything after
it is ignored, matching read-string semantics.
Reads every top-level form out of source, in order. runtime -- see read/2's doc.
Tokenizes source into the complete, position-preserving raw token stream (including :TRIVIA). See Logos.Format's moduledoc for why this exists alongside read/1/read_all/1.
Functions
@spec read(String.t(), Logos.Runtime.t() | nil) :: {:ok, Logos.Form.t()} | {:error, term()}
Reads exactly the first top-level form out of source; anything after
it is ignored, matching read-string semantics.
runtime (optional, defaults to nil) serves two purposes: syntax-
quote auto-qualification (it tells Logos.Reader.Actions what the
"current defining namespace" is when desugaring a ` anywhere in
the form), and resolving a #tag value tagged literal against
Logos.Runtime's :data_readers registry (handle_rule(:tagged_literal, ...)). Passing nil (the default, and what every runtime-unaware
caller/test still gets) skips auto-qualification gracefully (a bare
symbol inside a syntax-quote is left unqualified rather than erroring)
but makes reading a tagged literal a hard error -- there's no sensible
degraded fallback for "resolve this tag" the way there is for
"qualify this symbol."
@spec read_all(String.t(), Logos.Runtime.t() | nil) :: {:ok, [Logos.Form.t()]} | {:error, term()}
Reads every top-level form out of source, in order. runtime -- see read/2's doc.
@spec tokenize(String.t()) :: {:ok, [Grammar.VM.Token.t()]} | {:error, term()}
Tokenizes source into the complete, position-preserving raw token stream (including :TRIVIA). See Logos.Format's moduledoc for why this exists alongside read/1/read_all/1.