The tokenizer of CSS Syntax Level 3, over
preprocessed input (StylesheetParser.tokenize/1 turns \r\n, \r and \f into \n and NUL into
U+FFFD).
Each token is {type, value, [line: line, column: column]}, where it starts, columns counting
bytes from 1. Its value is its source text: a string keeps its quotes, and escapes stay escaped
(StylesheetParser.unescape/1 decodes them). Comments are dropped, an unclosed one running to
the end.
| Type | Value | Such as |
|---|---|---|
:ident | the name | color, --gap, \31 0 |
:function | the name and its (; a StylesheetParser.Function among component values | var( |
:at_keyword | the name and its @ | @media |
:hash | the name and its # | #main, #f00 |
:string | the string, with its quotes | "a.png" |
:bad_string | a string with a newline in it, up to the newline | "a |
:url | the whole url() without quotes, or with escapes in url | url(a.png), u\72l(a.png) |
:bad_url | a url() with whitespace, a quote or a ( inside | url(a b) |
:number | the number | 12, -0.5, 1e3 |
:percentage | the number, without its % | 50 for 50% |
:dimension | [number, unit] | ["1", "px"] |
:whitespace | the whitespace | " ", "\n " |
:delim | any other code point | >, ., ! |
:colon, :semicolon, :comma | :, ;, , | |
:lparen, :rparen, :lbracket, :rbracket, :lcurly, :rcurly | the bracket; a StylesheetParser.SimpleBlock among component values | ( |
:cdo, :cdc | <!--, --> | |
:urange | a unicode range, only in a unicode-range declaration's value | U+0-7F |
iex> StylesheetParser.tokenize("a { margin: 1px }")
{:ok,
[
{:ident, "a", [line: 1, column: 1]},
{:whitespace, " ", [line: 1, column: 2]},
{:lcurly, "{", [line: 1, column: 3]},
{:whitespace, " ", [line: 1, column: 4]},
{:ident, "margin", [line: 1, column: 5]},
{:colon, ":", [line: 1, column: 11]},
{:whitespace, " ", [line: 1, column: 12]},
{:dimension, ["1", "px"], [line: 1, column: 13]},
{:whitespace, " ", [line: 1, column: 16]},
{:rcurly, "}", [line: 1, column: 17]}
]}
Summary
Functions
Parses the given binary as consume.
Functions
@spec consume(binary(), keyword()) :: {:ok, [term()], rest, context, line, byte_offset} | {:error, reason, rest, context, line, byte_offset} when line: {pos_integer(), byte_offset}, byte_offset: non_neg_integer(), rest: binary(), reason: String.t(), context: map()
Parses the given binary as consume.
Returns {:ok, [token], rest, context, position, byte_offset} or
{:error, reason, rest, context, line, byte_offset} where position
describes the location of the consume (start position) as {line, offset_to_start_of_line}.
To column where the error occurred can be inferred from byte_offset - offset_to_start_of_line.
Options
:byte_offset- the byte offset for the whole binary, defaults to 0:line- the line and the byte offset into that line, defaults to{1, byte_offset}:context- the initial context value. It will be converted to a map