Lexer for WPL-AI language.
Handles tokenization including Python-style significant indentation, producing INDENT and DEDENT tokens for the parser.
Features
- Significant indentation (2 or 4 spaces, must be consistent)
- Keywords, identifiers, strings, numbers
- Dates (YYYY-MM-DD) and times (HH:MM)
- Comments starting with #
- Operators and punctuation
Summary
Functions
Tokenize WPL-AI source text.
Types
@type token() :: {token_type(), any(), WplAi.Errors.Location.t()}
@type token_type() ::
:indent
| :dedent
| :newline
| :eof
| :keyword
| :string
| :number
| :date
| :time
| :datetime
| :ident
| :bare_word
| :arrow
| :range
| :colon
| :comma
| :lparen
| :rparen
| :eq
| :neq
| :gte
| :lte
| :gt
| :lt
| :plus
| :minus
| :percent
| :slash
| :times
Functions
@spec tokenize(String.t()) :: {:ok, [token()]} | {:error, [WplAi.Errors.LexerError.t()]}
Tokenize WPL-AI source text.
Returns {:ok, tokens} or {:error, [LexerError.t()]}.