View Source Grammar.Tokenizer (Grammar v0.3.0)

This module extracts the tokens from the input string.

It is driven by the parser to search for specific tokens only, when required.

Summary

Functions

Returns the current token found in the input, if any. Expected tokens are passed as a list of token prototypes.

Creates a new tokenizer for a given input.

Returns the current token found in the input, and consumes it. The expected token prototype is passed as second parameter.

Types

@type t() :: %Grammar.Tokenizer{
  current_column: integer(),
  current_line: integer(),
  drop_spaces?: boolean(),
  input: binary()
}

Functions

Link to this function

current_token(tokenizer, token_prototypes)

View Source
@spec current_token(t(), [any()]) :: {any(), t()}

Returns the current token found in the input, if any. Expected tokens are passed as a list of token prototypes.

The token is not consumed, so two succesive calls to current_token/2 will return the same token.

Examples

iex> tokenizer = Grammar.Tokenizer.new("hello world")
%Grammar.Tokenizer{
  input: "hello world",
  current_line: 1,
  current_column: 1,
  drop_spaces?: true
}
iex> {{"hello", {1, 1}}, _} = Grammar.Tokenizer.current_token(tokenizer, ["hello"])
iex> {{"hello", {1, 1}}, _} = Grammar.Tokenizer.current_token(tokenizer, ["hello"])
iex> {{nil, {1, 1}}, _} = Grammar.Tokenizer.current_token(tokenizer, ["world"])
Link to this function

new(input, drop_spaces? \\ true)

View Source
@spec new(binary(), boolean()) :: t()

Creates a new tokenizer for a given input.

Parameters

  • input: bitstring from which token will be extracted.
  • drop_spaces?: when set to false, the tokenizer will not drop spaces and newlines.

Examples

iex> Grammar.Tokenizer.new("this is the input")
%Grammar.Tokenizer{
  input: "this is the input",
  current_line: 1,
  current_column: 1,
  drop_spaces?: true
}

iex> Grammar.Tokenizer.new("this is the input", false)
%Grammar.Tokenizer{
  input: "this is the input",
  current_line: 1,
  current_column: 1,
  drop_spaces?: false
}
Link to this function

next_token(tokenizer, token_prototype)

View Source
@spec next_token(t(), any()) :: {any(), t()}

Returns the current token found in the input, and consumes it. The expected token prototype is passed as second parameter.

Examples

iex> tokenizer = Grammar.Tokenizer.new("hello world")
iex> {{"hello", {1, 1}}, tokenizer} = Grammar.Tokenizer.next_token(tokenizer, "hello")
iex> {{"world", {1, 7}}, _} = Grammar.Tokenizer.next_token(tokenizer, "world")