The grammar's fixed vocabulary, enumerated for editor tooling.

An expression editor that offers completion needs to know what the language accepts: which operators exist, which words are reserved, which duration units follow a number, and which functions are callable. Function names have always been reachable - Predicator.Context.new/2 resolves them from Predicator.FunctionProvider modules - but the operator and keyword lexemes lived only inside Predicator.Lexer's private clause heads, so a consumer wanting completion had to hard-code a second copy of the grammar and keep it in sync by hand. This module is the first copy made public instead.

It is a reading surface, not a parsing one. Nothing here participates in lexing, parsing, compiling, or evaluating; the entries describe the grammar the lexer already implements, and adding a category or a doc string changes no program's meaning. test/predicator/vocabulary_sync_test.exs binds the two together: it round-trips every enumerated lexeme through Predicator.Lexer.tokenize/1, and it checks the enumeration against the lexer's own Predicator.Lexer.token/0 union, its classify_identifier/1 clause heads, and its duration_unit?/1 clause heads, so a token added to the lexer without a matching entry here turns the suite red rather than quietly shipping an editor that cannot complete it.

Entries

Every static entry is a map with at least these five keys:

• :lexeme - the exact source text, e.g. ">=", "contains"
• :token_type - the Predicator.Lexer.token/0 tag it lexes to
• :category - one of categories/0
• :display - what an editor shows in a completion list, which differs from the lexeme only where the bare lexeme reads badly on its own ("a + b" for "+", "1d" for the duration unit "d")
• :doc - one line of prose, sentence case, no trailing period

A function entry carries the same five keys plus :arity, since a function's arity is part of what an editor needs to complete a call. Its :doc is nil: a provider binds a name to {arity, atom} and carries no description, so there is nothing truthful to put there, and an invented sentence in a documented field is worse than an absent one.

Operator entries

An entry in an operator category - the six categories operators/0 returns - carries five more keys, because a structured editor needs more about an operator than a completion list does. A picklist has to name the operator in a sentence, know how many operands it takes, know which atom the parser will put in the AST for it, know which kinds of value it is worth offering it for at all, and - where the same operator is enumerated twice - know which of the two spellings to write:

• :label - a short phrase read in place of the operator, e.g. "is at least" for ">=". :display is a template and :doc is a sentence of prose about the semantics; neither is a UI string, which is why this is its own key
• :arity - how many operands the operator takes. "-" takes [1, 2], since it is both subtraction and negation
• :ast_op - the atom Predicator.Parser.parse/2 puts in the node it builds, which differs from :token_type for three operators (:in_op becomes :in, :contains_op becomes :contains, :strict_equal becomes :strict_eq). nil where no node carries the operator as an atom of its own
• :value_kinds - the value_kind/0s the operator is worth offering for, as the kind of the value on its right. nil for an operator that does not compare a field against a value at all - the logical, arithmetic, temporal and cast categories
• :canonical - whether this spelling is the one Predicator.decompile/2 writes for the operator. See "Case" below: a word operator is enumerated in both cases, and only one of the two entries is the spelling a caller that means to render source should offer

Entries outside an operator category carry none of the five. That is the shape function_entry/0 already has against entry/0: an entry carries the keys its kind of thing has, and no placeholder keys for the ones it does not.

What :value_kinds admits, and two deliberate exclusions

The admissions follow Predicator.Evaluator's own comparison semantics. Ordered comparison (>, >=, <, <=) is admitted where the evaluator orders the kind meaningfully: numbers and strings through its types_match guard, dates and datetimes chronologically, durations as the numbers they reduce to. Equality is admitted for every scalar kind. in takes a list on its right and nothing else; contains takes a list on its left, so the value beside it is a scalar of any kind.

Two things the evaluator does answer are still not admitted, because a picklist offering them would be offering nonsense:

Neither exclusion narrows the grammar. Both are the kind of judgement Predicator.Simple already records for the shapes it leaves out: the language still accepts them, and the structured surface still does not offer them.

Case

The word operators are accepted in two cases - and and AND both lex to :and_op - and both are enumerated, as separate entries with the same :token_type. An editor offering only one of them would be offering a house style the grammar does not have. if, else, while, and the temporal words are lower-case only, and are enumerated only that way.

Enumerating both cases leaves a caller that renders source with a question the enumeration alone cannot answer: of the two entries for one operator, which spelling does Predicator.decompile/2 write? That is a fact about this vocabulary, so :canonical answers it here rather than leaving each caller to re-derive it. It is false on exactly one thing: the lower-case entry of a word operator enumerated in both cases. Every other operator entry is the only spelling there is and is therefore its own canonical one - ">=", and the lower-case-only temporal words alike. It says nothing about what parses: both spellings lex to the same token type, and an editor offering completions wants both.

Examples

iex> Predicator.Vocabulary.by_category(:arithmetic) |> Enum.map(& &1.lexeme)
["+", "-", "*", "/", "%"]

iex> Predicator.Vocabulary.tokens() |> Enum.find(&(&1.lexeme == ">=")) |> Map.take([:token_type, :category])
%{token_type: :gte, category: :comparison}

iex> Predicator.Vocabulary.functions() |> Enum.any?(&(&1.lexeme == "len"))
true