Alaja.Syntax.Engine (Alaja v2.2.0)

Copy Markdown View Source

Generic single-pass character scanner driven by Alaja.Syntax.Language.

Each scan position tries patterns in order; the first match wins. Multi-line constructs (block comments, multi-line strings) set a context that the next iteration resolves before trying new matches.

Summary

Functions

tokenize(content, lang)

Tokenizes source code into {type, text} tuples.

Functions

tokenize(content, lang)

@spec tokenize(String.t(), Alaja.Syntax.Language.t()) :: [{atom(), String.t()}]

Tokenizes source code into {type, text} tuples.