GenAI.Approval.Lexer (GenAI Approval v0.1.0)

Copy Markdown View Source

Splits script source into segments and tokenizes them.

Segments:

  • {:mustache, tokens, {line, col}} — one {{ ... }} construct
  • {:text, string, {line, col}} — raw text between mustaches (only meaningful inside endpoint/vars/outputs blocks; must be whitespace everywhere else)

Comments ({{!-- ... --}} and {{! ... }}) are stripped here.

Tokens are {kind, value, {line, col}} with kind :ident | :string | :number | :punct. No atoms are ever created from input (S2): idents and strings stay binaries.

Summary

Types

segment()
token()

Functions

segments(source)
tokenize(text, arg)

Tokenize a raw-text segment (endpoint/vars/outputs bodies) starting at the given position.

Types

segment()

@type segment() ::
  {:mustache, [token()], {pos_integer(), pos_integer()}}
  | {:text, String.t(), {pos_integer(), pos_integer()}}

token()

@type token() ::
  {:ident | :string | :number | :punct, term(), {pos_integer(), pos_integer()}}

Functions

segments(source)

@spec segments(String.t()) :: {:ok, [segment()]} | {:error, GenAI.Approval.Error.t()}

tokenize(text, arg)

@spec tokenize(
  String.t(),
  {pos_integer(), pos_integer()}
) :: {:ok, [token()]} | {:error, GenAI.Approval.Error.t()}

Tokenize a raw-text segment (endpoint/vars/outputs bodies) starting at the given position.