View Source Grammar.Tokenizer.TokenExtractor protocol (Grammar v0.4.0)

This protocol exposes the functions needed to extract tokens from the input string.

Summary

Types

t()

All the types that implement this protocol.

Functions

Returns true if the token matches the token prototype.

Try to read a token from the input string.

Types

@type t() :: term()

All the types that implement this protocol.

Functions

Link to this function

match?(prototype, token)

View Source
@spec match?(t(), term()) :: boolean()

Returns true if the token matches the token prototype.

This function is used orient the parser in the right clause of a rule, by comparing the current token to the clause list of first tokens.

Example

iex> Grammar.Tokenizer.TokenExtractor.match?("hello", "hello")
true

iex> Grammar.Tokenizer.TokenExtractor.match?("hello", "Horld")
false

iex> Grammar.Tokenizer.TokenExtractor.match?(~r/[0-9]+/, "013456")
true

iex> Grammar.Tokenizer.TokenExtractor.match?(~r/[0-9]+/, "a013456")
false
Link to this function

try_read(token_prototype, input_string)

View Source
@spec try_read(t(), String.t()) :: nil | {String.t(), integer()}

Try to read a token from the input string.

If the token is found, returns the token and its length in the input string. Returns nil otherwise.

Example

iex> Grammar.Tokenizer.TokenExtractor.try_read("hello", "hello world")
{"hello", 5}

iex> Grammar.Tokenizer.TokenExtractor.try_read("Hello", "hello world")
nil

iex> Grammar.Tokenizer.TokenExtractor.try_read(~r/[0-9]+/, "013456toto")
{"013456", 6}

iex> Grammar.Tokenizer.TokenExtractor.try_read(~r/[a-z]+/, "013456toto")
nil