The grammar's fixed vocabulary, enumerated for editor tooling.
An expression editor that offers completion needs to know what the language
accepts: which operators exist, which words are reserved, which duration
units follow a number, and which functions are callable. Function names have
always been reachable - Predicator.Context.new/2 resolves them from
Predicator.FunctionProvider modules - but the operator and keyword lexemes
lived only inside Predicator.Lexer's private clause heads, so a consumer
wanting completion had to hard-code a second copy of the grammar and keep it
in sync by hand. This module is the first copy made public instead.
It is a reading surface, not a parsing one. Nothing here participates in
lexing, parsing, compiling, or evaluating; the entries describe the grammar
the lexer already implements, and adding a category or a doc string changes
no program's meaning. test/predicator/vocabulary_sync_test.exs binds the
two together: it round-trips every enumerated lexeme through
Predicator.Lexer.tokenize/1, and it checks the enumeration against the
lexer's own Predicator.Lexer.token/0 union, its classify_identifier/1
clause heads, and its duration_unit?/1 clause heads, so a token added to
the lexer without a matching entry here turns the suite red rather than
quietly shipping an editor that cannot complete it.
Entries
Every static entry is a map with five keys:
:lexeme- the exact source text, e.g.">=","contains":token_type- thePredicator.Lexer.token/0tag it lexes to:category- one ofcategories/0:display- what an editor shows in a completion list, which differs from the lexeme only where the bare lexeme reads badly on its own ("a + b"for"+","1d"for the duration unit"d"):doc- one line of prose, sentence case, no trailing period
A function entry carries the same five keys plus :arity, since a function's
arity is part of what an editor needs to complete a call. Its :doc is
nil: a provider binds a name to {arity, atom} and carries no description,
so there is nothing truthful to put there, and an invented sentence in a
documented field is worse than an absent one.
Case
The word operators are accepted in two cases - and and AND both lex to
:and_op - and both are enumerated, as separate entries with the same
:token_type. An editor offering only one of them would be offering a house
style the grammar does not have. if, else, while, and the temporal
words are lower-case only, and are enumerated only that way.
Examples
iex> Predicator.Vocabulary.by_category(:arithmetic) |> Enum.map(& &1.lexeme)
["+", "-", "*", "/", "%"]
iex> Predicator.Vocabulary.tokens() |> Enum.find(&(&1.lexeme == ">=")) |> Map.take([:token_type, :category])
%{token_type: :gte, category: :comparison}
iex> Predicator.Vocabulary.functions() |> Enum.any?(&(&1.lexeme == "len"))
trueSummary
Types
The kind of thing an entry is, which is what an editor groups a completion list by.
A fixed lexeme of the grammar - an operator, keyword, literal word, separator, or duration unit.
A callable function, resolved from the providers rather than from the lexer.
Functions
Every entry: tokens/0 followed by functions/1 on the same opts.
The entries in one category.
Every category an entry can carry, in the order tokens/0 groups them.
The callable functions, resolved the same way Predicator.Context.new/2
resolves them.
The word-shaped entries: everything an editor must not offer as a plain identifier, because the lexer classifies it as something else.
The entries that combine or compare values: the comparison, logical, arithmetic, membership, temporal, and cast categories.
Every fixed lexeme of the grammar: operators, keywords, literal words, brackets, separators, and duration units.
Types
@type category() ::
:comparison
| :logical
| :arithmetic
| :membership
| :temporal
| :control
| :literal
| :grouping
| :punctuation
| :cast
| :duration_unit
| :function
The kind of thing an entry is, which is what an editor groups a completion list by.
:cast holds the single :: postfix operator (ADR-0011); :grouping holds
the bracket pairs; :punctuation holds the separators that are neither.
@type entry() :: %{ lexeme: binary(), token_type: atom(), category: category(), display: binary(), doc: binary() }
A fixed lexeme of the grammar - an operator, keyword, literal word, separator, or duration unit.
@type function_entry() :: %{ lexeme: binary(), token_type: :function_name | :qualified_function_name, category: :function, display: binary(), doc: nil, arity: Predicator.Evaluator.function_arity() }
A callable function, resolved from the providers rather than from the lexer.
:token_type is :qualified_function_name for a namespaced name
("Math.abs") and :function_name for a bare one ("len"), matching what
the lexer produces for a call to it.
Functions
@spec all(keyword()) :: [entry() | function_entry(), ...]
Every entry: tokens/0 followed by functions/1 on the same opts.
Examples
iex> length(Predicator.Vocabulary.all()) == length(Predicator.Vocabulary.tokens()) + length(Predicator.Vocabulary.functions())
true@spec by_category(category()) :: [entry() | function_entry()]
The entries in one category.
The argument is guarded against categories/0, so a misspelled category
raises FunctionClauseError rather than returning an empty list that reads
like a category the grammar happens not to use.
Examples
iex> Predicator.Vocabulary.by_category(:cast) |> Enum.map(& &1.lexeme)
["::"]
iex> Predicator.Vocabulary.by_category(:literal) |> Enum.map(& &1.lexeme)
["true", "false", "null", "undefined"]@spec categories() :: [category(), ...]
Every category an entry can carry, in the order tokens/0 groups them.
Examples
iex> :duration_unit in Predicator.Vocabulary.categories()
true@spec functions(keyword()) :: [function_entry()]
The callable functions, resolved the same way Predicator.Context.new/2
resolves them.
opts takes :builtins, :providers, and :functions, and is passed
straight to Predicator.Context.resolve_functions/1, so the names returned
here are exactly the names a context built with the same options will
accept - including a host's own providers, which is the case an editor
embedded in a host application actually has.
Entries are sorted by name, since a dispatch map has no order and a completion list needs one.
Examples
iex> Predicator.Vocabulary.functions(builtins: false)
[]
iex> Predicator.Vocabulary.functions() |> Enum.find(&(&1.lexeme == "len")) |> Map.take([:display, :arity, :doc])
%{display: "len(...)", arity: 1, doc: nil}
iex> Predicator.Vocabulary.functions() |> Enum.find(&(&1.lexeme == "Math.abs")) |> Map.fetch!(:token_type)
:qualified_function_name@spec keywords() :: [entry(), ...]
The word-shaped entries: everything an editor must not offer as a plain identifier, because the lexer classifies it as something else.
Duration units are excluded. d is an ordinary identifier everywhere except
immediately after a number, so treating it as a reserved word would be
wrong; by_category(:duration_unit) is where those live.
Examples
iex> Predicator.Vocabulary.keywords() |> Enum.map(& &1.lexeme) |> Enum.member?("contains")
true
iex> Predicator.Vocabulary.keywords() |> Enum.map(& &1.lexeme) |> Enum.member?("d")
false@spec operators() :: [entry(), ...]
The entries that combine or compare values: the comparison, logical, arithmetic, membership, temporal, and cast categories.
Examples
iex> Predicator.Vocabulary.operators() |> Enum.all?(&(&1.category != :literal))
true@spec tokens() :: [entry(), ...]
Every fixed lexeme of the grammar: operators, keywords, literal words, brackets, separators, and duration units.
Functions are not here - they depend on which providers a caller resolves,
so they come from functions/0 and functions/1 instead. all/0 is the
two lists together.
Examples
iex> Predicator.Vocabulary.tokens() |> Enum.map(& &1.category) |> Enum.uniq() |> Enum.member?(:function)
false