Predicator.Parser (predicator v3.8.0)

Copy Markdown View Source

Recursive descent parser for predicator expressions.

The parser converts a stream of tokens from the lexer into an Abstract Syntax Tree (AST) with comprehensive error reporting including exact position information.

Grammar

The parser implements this grammar with proper operator precedence:

expression    logical_or
logical_or    logical_and ( "OR" | "||" logical_and )*
logical_and   logical_not ( "AND" | "&&" logical_not )*
logical_not   "NOT" | "!" logical_not | comparison
comparison    addition ( ( ">" | "<" | ">=" | "<=" | "=" (deprecated) | "==" | "!=" | "===" | "!==" | "in" | "contains" ) addition )?
addition      multiplication ( ( "+" | "-" ) multiplication )*
multiplication  unary ( ( "*" | "/" | "%" ) unary )*
unary         ( "-" | "!" ) unary | postfix
postfix       primary ( "[" expression "]" | "." IDENTIFIER )*
primary       NUMBER | FLOAT | STRING | BOOLEAN | DATE | DATETIME | IDENTIFIER | duration | relative_date | function_call | list | object | "(" expression ")"
function_call  FUNCTION_NAME "(" ( expression ( "," expression )* )? ")"
list          "[" ( expression ( "," expression )* )? "]"
object        "{" ( object_entry ( "," object_entry )* )? "}"
object_entry  object_key ":" expression
object_key    IDENTIFIER | STRING
duration      NUMBER UNIT+
relative_date  duration "ago" | duration "from" "now" | "next" duration | "last" duration

Deprecated: = as equality

Using = as an equality operator is deprecated. It still parses and still compiles to ["compare", "EQ"], but parsing one emits a deprecation warning, and Predicator 4.0 makes expression-position = a parse error. Use == instead.

Source positions

Every AST node carries a trailing {line, column} giving the 1-based position of the token that defines it. Leaves point at their own token; operators point at the operator token, so the arithmetic node in a * true reports column 3.

Use strip_positions/1 to recover the position-free shape Predicator 3.6 produced, and ensure_positions/1 to bring a position-free AST up to the shape the visitors expect.

Examples

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("(age >= 18)")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gte, {:identifier, "age", {1, 2}}, {:literal, 18, {1, 9}}, {1, 6}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85 AND age >= 18")
iex> Predicator.Parser.parse(tokens)
{:ok, {:logical_and, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}, {:comparison, :gte, {:identifier, "age", {1, 16}}, {:literal, 18, {1, 23}}, {1, 20}}, {1, 12}}}

Summary

Types

Arithmetic operators in the AST.

Abstract Syntax Tree node types.

The position-free AST shape Predicator 3.6 produced, as returned by strip_positions/1. The visitors still consume this shape; they move to ast/0 when they learn to carry positions.

A position-free object key, as returned by strip_positions/1.

Comparison operators in the AST.

Membership operators in the AST.

An object entry (key-value pair) in an object literal.

A key in an object literal.

How an object key was written: bare ({name: 1}), double-quoted ({"name": 1}), or single-quoted ({'name': 1}).

Internal parser state for tracking position and tokens.

A node's source position, or nil when the node was not produced by the parser.

Relative date directions in the AST.

Parser result - either success with AST or error with details.

Unary operators in the AST.

A value that can appear in literals.

Functions

Appends a nil position to any node that lacks one, producing the shape visitors expect.

Parses a list of tokens into an Abstract Syntax Tree.

Removes source positions from an AST, producing the position-free shape Predicator 3.6 used.

Types

arithmetic_op()

@type arithmetic_op() :: :add | :subtract | :multiply | :divide | :modulo

Arithmetic operators in the AST.

ast()

@type ast() ::
  {:literal, value(), position()}
  | {:string_literal, binary(), :double | :single, position()}
  | {:identifier, binary(), position()}
  | {:comparison, comparison_op(), ast(), ast(), position()}
  | {:arithmetic, arithmetic_op(), ast(), ast(), position()}
  | {:unary, unary_op(), ast(), position()}
  | {:membership, membership_op(), ast(), ast(), position()}
  | {:logical_and, ast(), ast(), position()}
  | {:logical_or, ast(), ast(), position()}
  | {:logical_not, ast(), position()}
  | {:list, [ast()], position()}
  | {:object, [object_entry()], position()}
  | {:function_call, binary(), [ast()], position()}
  | {:bracket_access, ast(), ast(), position()}
  | {:property_access, ast(), binary(), position()}
  | {:duration, [{integer(), binary()}], position()}
  | {:relative_date, ast(), relative_direction(), position()}

Abstract Syntax Tree node types.

Every node carries a trailing source position - the {line, column} of the token that defines it, or nil for a node built by a caller rather than parsed.

  • {:literal, value, pos} - A literal value (number, boolean, list, date, datetime, duration)
  • {:string_literal, value, quote_type, pos} - A string literal with quote type information
  • {:identifier, name, pos} - A variable reference
  • {:comparison, operator, left, right, pos} - A comparison expression (including equality)
  • {:arithmetic, operator, left, right, pos} - An arithmetic expression (+, -, *, /, %)
  • {:unary, operator, operand, pos} - A unary expression (-, !)
  • {:logical_and, left, right, pos} - A logical AND expression
  • {:logical_or, left, right, pos} - A logical OR expression
  • {:logical_not, operand, pos} - A logical NOT expression
  • {:list, elements, pos} - A list literal
  • {:object, entries, pos} - An object literal whose keys are object_key/0
  • {:membership, operator, left, right, pos} - A membership operation (in/contains)
  • {:function_call, name, arguments, pos} - A function call with arguments
  • {:bracket_access, object, key, pos} - A bracket access expression (obj[key])
  • {:property_access, object, property, pos} - A property access expression (obj.prop)
  • {:duration, units, pos} - A duration literal (e.g., 3d8h)
  • {:relative_date, duration, direction, pos} - A relative date expression (e.g., 3d ago, next 2w)

bare_ast()

@type bare_ast() ::
  {:literal, value()}
  | {:string_literal, binary(), :double | :single}
  | {:identifier, binary()}
  | {:comparison, comparison_op(), bare_ast(), bare_ast()}
  | {:arithmetic, arithmetic_op(), bare_ast(), bare_ast()}
  | {:unary, unary_op(), bare_ast()}
  | {:membership, membership_op(), bare_ast(), bare_ast()}
  | {:logical_and, bare_ast(), bare_ast()}
  | {:logical_or, bare_ast(), bare_ast()}
  | {:logical_not, bare_ast()}
  | {:list, [bare_ast()]}
  | {:object, [{bare_object_key(), bare_ast()}]}
  | {:function_call, binary(), [bare_ast()]}
  | {:bracket_access, bare_ast(), bare_ast()}
  | {:property_access, bare_ast(), binary()}
  | {:duration, [{integer(), binary()}]}
  | {:relative_date, bare_ast(), relative_direction()}

The position-free AST shape Predicator 3.6 produced, as returned by strip_positions/1. The visitors still consume this shape; they move to ast/0 when they learn to carry positions.

bare_object_key()

@type bare_object_key() :: {:identifier, binary()} | {:string_literal, binary()}

A position-free object key, as returned by strip_positions/1.

comparison_op()

@type comparison_op() ::
  :gt | :lt | :gte | :lte | :eq | :equal_equal | :ne | :strict_eq | :strict_ne

Comparison operators in the AST.

membership_op()

@type membership_op() :: :in | :contains

Membership operators in the AST.

object_entry()

@type object_entry() :: {object_key(), ast()}

An object entry (key-value pair) in an object literal.

The key can be either an identifier or a string literal.

object_key()

@type object_key() :: {:object_key, binary(), object_key_style(), position()}

A key in an object literal.

Object keys have their own tag rather than reusing the expression node tags, so no consumer has to tell a key from an expression by tuple arity. style records how the key was written - bare, or quoted with which character - so Predicator.Visitors.StringVisitor can render it back as the author wrote it.

object_key_style()

@type object_key_style() :: :identifier | :double | :single

How an object key was written: bare ({name: 1}), double-quoted ({"name": 1}), or single-quoted ({'name': 1}).

parser_state()

@type parser_state() :: %{
  tokens: [Predicator.Lexer.token()],
  position: non_neg_integer()
}

Internal parser state for tracking position and tokens.

position()

@type position() :: Predicator.Types.position() | nil

A node's source position, or nil when the node was not produced by the parser.

relative_direction()

@type relative_direction() :: :ago | :future | :next | :last

Relative date directions in the AST.

result()

@type result() :: {:ok, ast()} | {:error, binary(), pos_integer(), pos_integer()}

Parser result - either success with AST or error with details.

unary_op()

@type unary_op() :: :minus | :bang

Unary operators in the AST.

value()

@type value() ::
  boolean()
  | integer()
  | binary()
  | [value()]
  | Date.t()
  | DateTime.t()
  | Predicator.Types.duration()

A value that can appear in literals.

Functions

ensure_positions(node)

@spec ensure_positions(term()) :: term()

Appends a nil position to any node that lacks one, producing the shape visitors expect.

This is what lets Predicator.decompile/2 and Predicator.Compiler.to_instructions/2 keep accepting a caller-supplied 3.6-shaped AST. A nil position produces no entry in the side table.

Examples

iex> Predicator.Parser.ensure_positions({:literal, 42})
{:literal, 42, nil}

iex> Predicator.Parser.ensure_positions({:literal, 42, {1, 1}})
{:literal, 42, {1, 1}}

parse(tokens)

@spec parse([Predicator.Lexer.token()]) :: result()

Parses a list of tokens into an Abstract Syntax Tree.

Parameters

  • tokens - List of tokens from the lexer

Returns

  • {:ok, ast} - Successfully parsed expression
  • {:error, message, line, column} - Parse error with position

Examples

iex> {:ok, tokens} = Predicator.Lexer.tokenize("score > 85")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :gt, {:identifier, "score", {1, 1}}, {:literal, 85, {1, 9}}, {1, 7}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("name = \"John\"")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :eq, {:identifier, "name", {1, 1}}, {:string_literal, "John", :double, {1, 8}}, {1, 6}}}

iex> {:ok, tokens} = Predicator.Lexer.tokenize("active = true")
iex> Predicator.Parser.parse(tokens)
{:ok, {:comparison, :eq, {:identifier, "active", {1, 1}}, {:literal, true, {1, 10}}, {1, 8}}}

strip_positions(node)

@spec strip_positions(term()) :: term()

Removes source positions from an AST, producing the position-free shape Predicator 3.6 used.

Total and idempotent: an AST that already carries no positions is returned unchanged, and an unrecognized node is passed through rather than raising.

Examples

iex> Predicator.Parser.strip_positions({:literal, 42, {1, 1}})
{:literal, 42}

iex> Predicator.Parser.strip_positions({:literal, 42})
{:literal, 42}