wasm_wat_lex (wasm v0.1.0)

View Source

Tokens of the WebAssembly text format. Start here when the text parser rejects something it should accept.

The grammar has six kinds of token and two kinds of comment, and almost all of the difficulty is in the last two:

tokenexample
parenthesis( )
keywordmodule, i32.add, offset=4
identifier$fac
string"a\\6Ab"
number-1_000, 0x1.8p3, nan:0x4000, inf
reservedanything else, which is a syntax error where it appears

A keyword may contain =. offset=4 is one token, not three, so the lexer cannot treat = as punctuation and the parser splits it later.

Numbers carry underscores. 1_000_000 is a million and 0x_ff is malformed, because a separator has to sit between two digits.

Block comments nest. (; a (; b ;) c ;) is one comment. A scanner that stopped at the first ;) would leave c ;) as tokens, which is how a comment becomes a syntax error a long way from its cause.

Tokens have to be separated. (data"a") is not data followed by a string: with nothing between them the two are one token, and a malformed one. Only parentheses, whitespace and comments separate.

Quoted identifiers ($"...") and annotations ((@name ...)) are not lexed. Both belong to proposals this runtime does not implement, and both are refused by name here rather than half-accepted: they are exactly the id and annotations suites the specification manifest already lists as out of scope.

Positions are byte offsets from the start of the input, carried on every token so a parse error can say where rather than what.

Summary

Types

token()

A token and the byte offset it starts at.

Functions

scan_all(Bin)

As tokens/1, but raising rather than answering.

tokens(Bin)

Tokenise, or fail with a structured error.

Types

token()

-nominal token() ::
             {lparen, pos()} |
             {rparen, pos()} |
             {keyword, pos(), binary()} |
             {id, pos(), binary()} |
             {string, pos(), binary()} |
             {reserved, pos(), binary()}.

A token and the byte offset it starts at.

Functions

scan_all(Bin)

-spec scan_all(binary()) -> [token()].

As tokens/1, but raising rather than answering.

For a caller that is already inside wasm_error:capture/1 and would otherwise have to match {ok, _} and re-raise. Matching is what a first version of the reader did, and a malformed file then arrived as an internal error with a badmatch in it instead of the lexer's own diagnosis.

tokens(Bin)

-spec tokens(binary()) -> {ok, [token()]} | {error, wasm_error:error()}.

Tokenise, or fail with a structured error.

Answers {ok, Tokens} or {error, Error} in the shape everything else here uses, so a malformed text module is a value like a malformed binary one.