PureHTML.Tokenizer (PureHTML v0.5.0)

View Source

HTML5 tokenizer that produces a stream of tokens.

The tokenizer is implemented as a state machine. Each call to next_token/1 advances the machine until it produces a token, then returns the token and the updated state. The tree builder switches the state for raw text, RCDATA, script data, and plaintext elements with set_state/2, and tells the tokenizer whether the adjusted current node is foreign with set_foreign_content/2 (for CDATA sections).

Usage

iex> PureHTML.Tokenizer.tokenize("<p>Hello</p>") |> Enum.to_list()
[{:start_tag, "p", [], false}, {:character, "Hello"}, {:end_tag, "p"}]

Token Types

  • {:doctype, name, public_id, system_id, force_quirks?}
  • {:start_tag, name, attrs, self_closing?}
  • {:end_tag, name}
  • {:comment, data}
  • {:pi, target, data}
  • {:character, data}

Summary

Types

t()
token()

Functions

new(input, opts \\ [])

Creates a new tokenizer state from input.

next_token(state)

Gets the next token from the tokenizer.

set_foreign_content(tokenizer, in_foreign_content)

Updates the foreign content context flag.

set_state(tokenizer, state)

Switches the tokenizer state. The tree builder does this for the generic raw text and RCDATA element parsing algorithms, script, and plaintext.

tokenize(input, opts \\ [])

Tokenizes an HTML string, returning a Stream of tokens.

tokenize_with_errors(input, opts \\ [])

Tokenizes an HTML string and returns {tokens, error_count}, where the count follows the parse errors of the tokenizer and the input stream.

Types

t()

@type t() :: %PureHTML.Tokenizer{
  adjusted_current_node_not_in_html_namespace: term(),
  attr_name: term(),
  attr_names: term(),
  attr_value: term(),
  buffer: term(),
  deferred_token: term(),
  duplicate_attr: term(),
  eof_emitted: term(),
  error_count: term(),
  input: term(),
  last_start_tag: term(),
  pending_chars: term(),
  return_state: term(),
  state: term(),
  token: term(),
  xml_violation_mode: term()
}

token()

@type token() ::
  {:doctype, String.t() | nil, String.t() | nil, String.t() | nil, boolean()}
  | {:start_tag, String.t(), [{String.t(), String.t()}], boolean()}
  | {:end_tag, String.t()}
  | {:comment, String.t()}
  | {:pi, String.t(), String.t()}
  | {:character, String.t()}
  | :eof

Functions

new(input, opts \\ [])

@spec new(
  String.t(),
  keyword()
) :: t()

Creates a new tokenizer state from input.

The input is decoded as UTF-8 with U+FFFD replacement, newlines are normalized, and the input stream parse errors (controls, noncharacters) are counted into error_count.

Options

  • :initial_state - Starting tokenizer state (default: :data)
  • :last_start_tag - Last start tag name for appropriate end tag checks

next_token(state)

@spec next_token(t()) :: {token(), t()} | nil

Gets the next token from the tokenizer.

Returns {token, updated_tokenizer} or nil when done.

set_foreign_content(tokenizer, in_foreign_content)

@spec set_foreign_content(t(), boolean()) :: t()

Updates the foreign content context flag.

When in_foreign_content is true, <![CDATA[ will be parsed as a CDATA section. When false (default), it's treated as a bogus comment per HTML5 spec.

set_state(tokenizer, state)

@spec set_state(t(), atom()) :: t()

Switches the tokenizer state. The tree builder does this for the generic raw text and RCDATA element parsing algorithms, script, and plaintext.

tokenize(input, opts \\ [])

@spec tokenize(
  String.t(),
  keyword()
) :: Enumerable.t()

Tokenizes an HTML string, returning a Stream of tokens.

The stream is lazy - tokens are produced on demand as the stream is consumed.

tokenize_with_errors(input, opts \\ [])

@spec tokenize_with_errors(
  String.t(),
  keyword()
) :: {[term()], non_neg_integer()}

Tokenizes an HTML string and returns {tokens, error_count}, where the count follows the parse errors of the tokenizer and the input stream.