KiwiCodec.Schema.Tokenizer (kiwi_codec v0.5.0)

Copy Markdown View Source

Tokenizes .kiwi schema text with source positions.

Tokens are identifiers, integers with an optional leading -, and the symbols =, ;, {, }, [], and [deprecated]. Whitespace and // comments separate them. The last token is an empty :eof token marking where the text ends.

Summary

Functions

tokenize(text)

@spec tokenize(String.t()) :: [KiwiCodec.Schema.Token.t()]