Predicator.Parser (predicator v3.7.0)

Recursive descent parser for predicator expressions.

The parser converts a stream of tokens from the lexer into an Abstract Syntax Tree (AST) with comprehensive error reporting including exact position information.

Grammar

The parser implements this grammar with proper operator precedence:

expression   → logical_or
logical_or   → logical_and ( "OR" | "||" logical_and )*
logical_and  → logical_not ( "AND" | "&&" logical_not )*
logical_not  → "NOT" | "!" logical_not | comparison
comparison   → addition ( ( ">" | "<" | ">=" | "<=" | "=" (deprecated) | "==" | "!=" | "===" | "!==" | "in" | "contains" ) addition )?
addition     → multiplication ( ( "+" | "-" ) multiplication )*
multiplication → unary ( ( "*" | "/" | "%" ) unary )*
unary        → ( "-" | "!" ) unary | postfix
postfix      → primary ( "[" expression "]" | "." IDENTIFIER )*
primary      → NUMBER | FLOAT | STRING | BOOLEAN | DATE | DATETIME | IDENTIFIER | duration | relative_date | function_call | list | object | "(" expression ")"
function_call → FUNCTION_NAME "(" ( expression ( "," expression )* )? ")"
list         → "[" ( expression ( "," expression )* )? "]"
object       → "{" ( object_entry ( "," object_entry )* )? "}"
object_entry → object_key ":" expression
object_key   → IDENTIFIER | STRING
duration     → NUMBER UNIT+
relative_date → duration "ago" | duration "from" "now" | "next" duration | "last" duration

Deprecated: `=` as equality

Using = as an equality operator is deprecated. It still parses and still compiles to ["compare", "EQ"], but parsing one emits a deprecation warning, and Predicator 4.0 makes expression-position = a parse error. Use == instead.

Source positions

Every AST node carries a trailing {line, column} giving the 1-based position of the token that defines it. Leaves point at their own token; operators point at the operator token, so the arithmetic node in a * true reports column 3.

Use strip_positions/1 to recover the position-free shape Predicator 3.6 produced, and ensure_positions/1 to bring a position-free AST up to the shape the visitors expect.

Examples

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("(age >= 18)")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gte, {:identifier, "age", {1, 2}}, {:literal, 18, {1, 9}}, {1, 6}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85 AND age >= 18")
iex> Predicator.Parser.parse(tokens)
{:ok, {:logical_and, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}, {:comparison, :gte, {:identifier, "age", {1, 16}}, {:literal, 18, {1, 23}}, {1, 20}}, {1, 12}}}

Summary

Types

arithmetic_op()

Arithmetic operators in the AST.

ast()

Abstract Syntax Tree node types.

bare_ast()

The position-free AST shape Predicator 3.6 produced, as returned by strip_positions/1. The visitors still consume this shape; they move to ast/0 when they learn to carry positions.

bare_object_key()

A position-free object key, as returned by strip_positions/1.

comparison_op()

Comparison operators in the AST.

membership_op()

Membership operators in the AST.

object_entry()

An object entry (key-value pair) in an object literal.

object_key()

A key in an object literal - either an identifier or string literal.

parser_state()

Internal parser state for tracking position and tokens.

position()

A node's source position, or nil when the node was not produced by the parser.

relative_direction()

Relative date directions in the AST.

result()

Parser result - either success with AST or error with details.

unary_op()

Unary operators in the AST.

value()

A value that can appear in literals.

Functions

ensure_positions(node)

Appends a nil position to any node that lacks one, producing the shape visitors expect.

parse(tokens)

Parses a list of tokens into an Abstract Syntax Tree.

strip_positions(node)

Removes source positions from an AST, producing the position-free shape Predicator 3.6 used.

Types

arithmetic_op()

@type arithmetic_op() :: :add | :subtract | :multiply | :divide | :modulo

Arithmetic operators in the AST.

ast()

@type ast() ::
  {:literal, value(), position()}
  | {:string_literal, binary(), :double | :single, position()}
  | {:identifier, binary(), position()}
  | {:comparison, comparison_op(), ast(), ast(), position()}
  | {:arithmetic, arithmetic_op(), ast(), ast(), position()}
  | {:unary, unary_op(), ast(), position()}
  | {:membership, membership_op(), ast(), ast(), position()}
  | {:logical_and, ast(), ast(), position()}
  | {:logical_or, ast(), ast(), position()}
  | {:logical_not, ast(), position()}
  | {:list, [ast()], position()}
  | {:object, [object_entry()], position()}
  | {:function_call, binary(), [ast()], position()}
  | {:bracket_access, ast(), ast(), position()}
  | {:property_access, ast(), binary(), position()}
  | {:duration, [{integer(), binary()}], position()}
  | {:relative_date, ast(), relative_direction(), position()}

Abstract Syntax Tree node types.

Every node carries a trailing source position - the {line, column} of the token that defines it, or nil for a node built by a caller rather than parsed.

{:literal, value, pos} - A literal value (number, boolean, list, date, datetime, duration)
{:string_literal, value, quote_type, pos} - A string literal with quote type information
{:identifier, name, pos} - A variable reference
{:comparison, operator, left, right, pos} - A comparison expression (including equality)
{:arithmetic, operator, left, right, pos} - An arithmetic expression (+, -, *, /, %)
{:unary, operator, operand, pos} - A unary expression (-, !)
{:logical_and, left, right, pos} - A logical AND expression
{:logical_or, left, right, pos} - A logical OR expression
{:logical_not, operand, pos} - A logical NOT expression
{:list, elements, pos} - A list literal
{:object, entries, pos} - An object literal with key-value pairs
{:membership, operator, left, right, pos} - A membership operation (in/contains)
{:function_call, name, arguments, pos} - A function call with arguments
{:bracket_access, object, key, pos} - A bracket access expression (obj[key])
{:property_access, object, property, pos} - A property access expression (obj.prop)
{:duration, units, pos} - A duration literal (e.g., 3d8h)
{:relative_date, duration, direction, pos} - A relative date expression (e.g., 3d ago, next 2w)

bare_ast()

@type bare_ast() ::
  {:literal, value()}
  | {:string_literal, binary(), :double | :single}
  | {:identifier, binary()}
  | {:comparison, comparison_op(), bare_ast(), bare_ast()}
  | {:arithmetic, arithmetic_op(), bare_ast(), bare_ast()}
  | {:unary, unary_op(), bare_ast()}
  | {:membership, membership_op(), bare_ast(), bare_ast()}
  | {:logical_and, bare_ast(), bare_ast()}
  | {:logical_or, bare_ast(), bare_ast()}
  | {:logical_not, bare_ast()}
  | {:list, [bare_ast()]}
  | {:object, [{bare_object_key(), bare_ast()}]}
  | {:function_call, binary(), [bare_ast()]}
  | {:bracket_access, bare_ast(), bare_ast()}
  | {:property_access, bare_ast(), binary()}
  | {:duration, [{integer(), binary()}]}
  | {:relative_date, bare_ast(), relative_direction()}

The position-free AST shape Predicator 3.6 produced, as returned by strip_positions/1. The visitors still consume this shape; they move to ast/0 when they learn to carry positions.

bare_object_key()

@type bare_object_key() :: {:identifier, binary()} | {:string_literal, binary()}

A position-free object key, as returned by strip_positions/1.

comparison_op()

@type comparison_op() ::
  :gt | :lt | :gte | :lte | :eq | :equal_equal | :ne | :strict_eq | :strict_ne

Comparison operators in the AST.

membership_op()

@type membership_op() :: :in | :contains

Membership operators in the AST.

object_entry()

@type object_entry() :: {object_key(), ast()}

An object entry (key-value pair) in an object literal.

The key can be either an identifier or a string literal.

object_key()

@type object_key() ::
  {:identifier, binary(), position()} | {:string_literal, binary(), position()}

A key in an object literal - either an identifier or string literal.

parser_state()

@type parser_state() :: %{
  tokens: [Predicator.Lexer.token()],
  position: non_neg_integer()
}

Internal parser state for tracking position and tokens.

position()

@type position() :: Predicator.Types.position() | nil

A node's source position, or nil when the node was not produced by the parser.

relative_direction()

@type relative_direction() :: :ago | :future | :next | :last

Relative date directions in the AST.

result()

@type result() :: {:ok, ast()} | {:error, binary(), pos_integer(), pos_integer()}

Parser result - either success with AST or error with details.

unary_op()

@type unary_op() :: :minus | :bang

Unary operators in the AST.

value()

@type value() ::
  boolean()
  | integer()
  | binary()
  | [value()]
  | Date.t()
  | DateTime.t()
  | Predicator.Types.duration()

A value that can appear in literals.

Functions

ensure_positions(node)

@spec ensure_positions(term()) :: term()

Appends a nil position to any node that lacks one, producing the shape visitors expect.

This is what lets Predicator.decompile/2 and Predicator.Compiler.to_instructions/2 keep accepting a caller-supplied 3.6-shaped AST. A nil position produces no entry in the side table.

Examples

iex> Predicator.Parser.ensure_positions({:literal, 42})
{:literal, 42, nil}

iex> Predicator.Parser.ensure_positions({:literal, 42, {1, 1}})
{:literal, 42, {1, 1}}

parse(tokens)

@spec parse([Predicator.Lexer.token()]) :: result()

Parses a list of tokens into an Abstract Syntax Tree.

Parameters

tokens - List of tokens from the lexer

Returns

{:ok, ast} - Successfully parsed expression
{:error, message, line, column} - Parse error with position

Examples

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("name = \"John\"")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :eq, {:identifier, "name", {1, 1}}, {:string_literal, "John", :double, {1, 8}}, {1, 6}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("active = true")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :eq, {:identifier, "active", {1, 1}}, {:literal, true, {1, 10}}, {1, 8}}}

strip_positions(node)

@spec strip_positions(term()) :: term()

Removes source positions from an AST, producing the position-free shape Predicator 3.6 used.

Total and idempotent: an AST that already carries no positions is returned unchanged, and an unrecognized node is passed through rather than raising.

Examples

iex> Predicator.Parser.strip_positions({:literal, 42, {1, 1}})
{:literal, 42}

iex> Predicator.Parser.strip_positions({:literal, 42})
{:literal, 42}

Predicator.Parser (predicator v3.7.0)

Grammar

Deprecated: = as equality

Source positions

Examples

Summary

Types

Functions

Types

arithmetic_op()

ast()

bare_ast()

bare_object_key()

comparison_op()

membership_op()

object_entry()

object_key()

parser_state()

position()

relative_direction()

result()

unary_op()

value()

Functions

ensure_positions(node)

Examples

parse(tokens)

Parameters

Returns

Examples

strip_positions(node)

Examples

Deprecated: `=` as equality