Lua.Lexer (Lua v1.0.0-rc.2)

Hand-written lexer for Lua 5.3 using Elixir binary pattern matching.

Tokenizes Lua source code into a list of tokens with position tracking.

Summary

Types

position()

token()

Functions

tokenize(code)

Tokenizes Lua source code into a list of tokens.

Types

position()

@type position() :: %{
  line: pos_integer(),
  column: pos_integer(),
  byte_offset: non_neg_integer()
}

token()

@type token() ::
  {:keyword, atom(), position()}
  | {:identifier, String.t(), position()}
  | {:number, number(), position()}
  | {:string, String.t(), position()}
  | {:operator, atom(), position()}
  | {:delimiter, atom(), position()}
  | {:comment, :single | :multi, String.t(), position()}
  | {:eof, position()}

Functions

tokenize(code)

@spec tokenize(String.t()) :: {:ok, [token()]} | {:error, term()}

Tokenizes Lua source code into a list of tokens.

Examples

iex> Lua.Lexer.tokenize("local x = 42")
{:ok, [
  {:keyword, :local, %{line: 1, column: 1, byte_offset: 0}},
  {:identifier, "x", %{line: 1, column: 7, byte_offset: 6}},
  {:operator, :assign, %{line: 1, column: 9, byte_offset: 8}},
  {:number, 42, %{line: 1, column: 11, byte_offset: 10}},
  {:eof, %{line: 1, column: 13, byte_offset: 12}}
]}