Lexer for Bash syntax.

Produces a stream of tokens with position information for the parser to consume. Handles quoting, escapes, and nested structures (command substitution, etc.) correctly.

Token Types

Tokens are tuples of {type, value, line, column} or {type, line, column} for valueless tokens like operators.

Word tokens

• {:word, parts, line, col} - A word with parts (literals, variables, etc.)
• {:assignment_word, name, value_parts, line, col} - VAR=value

Operators

• {:pipe, line, col} - |
• {:and_if, line, col} - &&
• {:or_if, line, col} - ||
• {:background, line, col} - &
• {:semi, line, col} - ;
• {:dsemi, line, col} - ;;
• {:semi_and, line, col} - ;&
• {:dsemi_and, line, col} - ;;&
• {:newline, line, col} - \n
• {:lparen, line, col} - (
• {:rparen, line, col} - )
• {:lbrace, line, col} - {
• {:rbrace, line, col} - }

Redirections

• {:less, fd, line, col} - <
• {:greater, fd, line, col} - >
• {:dgreater, fd, line, col} - >>
• {:lessand, fd, line, col} - <&
• {:greaterand, fd, line, col} - >&
• {:lessgreat, fd, line, col} - <>
• {:dless, fd, line, col} - << (heredoc)
• {:dlessdash, fd, line, col} - <<- (heredoc with tab stripping)
• {:tless, fd, line, col} - <<< (herestring)
• {:andgreat, line, col} - &>
• {:anddgreat, line, col} - &>>
• {:brace_fd, varname, line, col} - {VAR} before redirect operator

Reserved words (context-dependent)

• {:if, line, col}
• {:then, line, col}
• {:else, line, col}
• {:elif, line, col}
• {:fi, line, col}
• {:case, line, col}
• {:esac, line, col}
• {:for, line, col}
• {:while, line, col}
• {:until, line, col}
• {:do, line, col}
• {:done, line, col}
• {:in, line, col}
• {:function, line, col}
• {:bang, line, col} - !

Test constructs

• {:lbracket, line, col} - [
• {:rbracket, line, col} - ]
• {:dlbracket, line, col} - [[
• {:drbracket, line, col} - ]]

Arithmetic

• {:arith_command, content, line, col} - ((...)) arithmetic command with raw content

Special

• {:eof, line, col} - End of input
• {:comment, text, line, col} - # comment