Kuddle.V1.Tokenizer (Kuddle v1.1.1)

Copy Markdown View Source

Intermediate process of converting a KDL1 document into some basic tokens that can be parsed.

Summary

Types

token()
tokens()

Functions

tokenize(blob)

Types

token()

@type token() ::
  Kuddle.Tokens.open_block_token()
  | Kuddle.Tokens.close_block_token()
  | Kuddle.Tokens.slashdash_token()
  | Kuddle.Tokens.comment_token()
  | Kuddle.Tokens.dquote_string_token()
  | Kuddle.Tokens.raw_string_token()
  | Kuddle.Tokens.space_token()
  | Kuddle.Tokens.newline_token()
  | Kuddle.Tokens.equal_token()
  | Kuddle.Tokens.semicolon_token()
  | Kuddle.Tokens.fold_token()
  | Kuddle.Tokens.term_token()

tokens()

@type tokens() :: [token()]

Functions

tokenize(blob)

@spec tokenize(String.t()) :: {:ok, tokens(), rest :: String.t()} | {:error, term()}