Regex-based tokenizer that splits text into tokens with byte offsets.
Whitespace tokens are preserved for continuous offset mapping. No text normalization is applied.
Summary
Functions
@spec tokenize(String.t()) :: [LangExtract.Alignment.Token.t()]
Regex-based tokenizer that splits text into tokens with byte offsets.
Whitespace tokens are preserved for continuous offset mapping. No text normalization is applied.
@spec tokenize(String.t()) :: [LangExtract.Alignment.Token.t()]