sqlode/lexer

Types

A SQL token produced by the lexer.

pub type Token {
  Keyword(String)
  Ident(String)
  QuotedIdent(String)
  StringLit(String)
  NumberLit(String)
  Placeholder(String)
  Operator(String)
  LParen
  RParen
  Comma
  Semicolon
  Dot
  Star
}

Constructors

  • Keyword(String)

    SQL keyword (lowercased): SELECT, FROM, CREATE, etc.

  • Ident(String)

    Unquoted identifier: table_name, column_name

  • QuotedIdent(String)

    Quoted identifier: “name”, name, [name] (quotes stripped)

  • StringLit(String)

    String literal (quotes stripped): ‘value’, dollar

  • NumberLit(String)

    Numeric literal: 42, 3.14

  • Placeholder(String)

    Parameter placeholder: $1, ?, :name, @name

  • Operator(String)

    Operator: =, <>, ::, ->, ||, etc.

  • LParen
  • RParen
  • Comma
  • Semicolon
  • Dot
  • Star

Options for controlling how tokens are rendered back to text.

pub type TokenRenderOptions {
  TokenRenderOptions(
    uppercase_keywords: Bool,
    preserve_quotes: Bool,
    engine: option.Option(model.Engine),
  )
}

Constructors

  • TokenRenderOptions(
      uppercase_keywords: Bool,
      preserve_quotes: Bool,
      engine: option.Option(model.Engine),
    )

    Arguments

    uppercase_keywords

    When True, SQL keywords are rendered in UPPERCASE.

    preserve_quotes

    When True, quoted identifiers keep their quotes and string literals escape embedded single-quotes.

    engine

    When set, use engine-specific quote style for identifiers: MySQL → backticks, PostgreSQL/SQLite → double quotes.

Values

pub fn tokenize(sql: String, engine: model.Engine) -> List(Token)

Tokenize a SQL string into a list of tokens. Comments are stripped. String literals and quoted identifiers are preserved as single tokens with their content.

pub fn tokens_to_string(
  tokens: List(Token),
  options: TokenRenderOptions,
) -> String

Render a list of tokens back to a SQL string with smart spacing.

Search Document