StylesheetParser.Syntax.Tokenizer (StylesheetParser v0.1.0)

Copy Markdown View Source

The tokenizer of CSS Syntax Level 3, over preprocessed input (StylesheetParser.tokenize/1 turns \r\n, \r and \f into \n and NUL into U+FFFD).

Each token is {type, value, [line: line, column: column]}, where it starts, columns counting bytes from 1. Its value is its source text: a string keeps its quotes, and escapes stay escaped (StylesheetParser.unescape/1 decodes them). Comments are dropped, an unclosed one running to the end.

TypeValueSuch as
:identthe namecolor, --gap, \31 0
:functionthe name and its (; a StylesheetParser.Function among component valuesvar(
:at_keywordthe name and its @@media
:hashthe name and its ##main, #f00
:stringthe string, with its quotes"a.png"
:bad_stringa string with a newline in it, up to the newline"a
:urlthe whole url() without quotes, or with escapes in urlurl(a.png), u\72l(a.png)
:bad_urla url() with whitespace, a quote or a ( insideurl(a b)
:numberthe number12, -0.5, 1e3
:percentagethe number, without its %50 for 50%
:dimension[number, unit]["1", "px"]
:whitespacethe whitespace" ", "\n "
:delimany other code point>, ., !
:colon, :semicolon, :comma:, ;, ,
:lparen, :rparen, :lbracket, :rbracket, :lcurly, :rcurlythe bracket; a StylesheetParser.SimpleBlock among component values(
:cdo, :cdc<!--, -->
:urangea unicode range, only in a unicode-range declaration's valueU+0-7F
iex> StylesheetParser.tokenize("a { margin: 1px }")
{:ok,
 [
   {:ident, "a", [line: 1, column: 1]},
   {:whitespace, " ", [line: 1, column: 2]},
   {:lcurly, "{", [line: 1, column: 3]},
   {:whitespace, " ", [line: 1, column: 4]},
   {:ident, "margin", [line: 1, column: 5]},
   {:colon, ":", [line: 1, column: 11]},
   {:whitespace, " ", [line: 1, column: 12]},
   {:dimension, ["1", "px"], [line: 1, column: 13]},
   {:whitespace, " ", [line: 1, column: 16]},
   {:rcurly, "}", [line: 1, column: 17]}
 ]}

Summary

Functions

Parses the given binary as consume.

Functions

consume(binary, opts \\ [])

@spec consume(binary(), keyword()) ::
  {:ok, [term()], rest, context, line, byte_offset}
  | {:error, reason, rest, context, line, byte_offset}
when line: {pos_integer(), byte_offset},
     byte_offset: non_neg_integer(),
     rest: binary(),
     reason: String.t(),
     context: map()

Parses the given binary as consume.

Returns {:ok, [token], rest, context, position, byte_offset} or {:error, reason, rest, context, line, byte_offset} where position describes the location of the consume (start position) as {line, offset_to_start_of_line}.

To column where the error occurred can be inferred from byte_offset - offset_to_start_of_line.

Options

  • :byte_offset - the byte offset for the whole binary, defaults to 0
  • :line - the line and the byte offset into that line, defaults to {1, byte_offset}
  • :context - the initial context value. It will be converted to a map