Regex-based tokenizer that splits text into tokens with byte offsets.

Mirrors upstream langextract's RegexTokenizer: letter runs, digit runs, and same-symbol runs (... is one token, ?! is two) — so Tooke’s tokenizes as Tooke · ’ · s and the aligner sees the bare name exactly as upstream does. Whitespace tokens are additionally preserved (upstream tracks newlines as a token flag instead) for continuous offset mapping and the chunker's newline rule. No text normalization is applied.

Internal — no stability guarantees; see the README's "Stability" section. Documented because it explains how the library works, not because it is API.