LangExtract.Alignment.Tokenizer (LangExtract v0.7.0)

Copy Markdown View Source

Regex-based tokenizer that splits text into tokens with byte offsets.

Whitespace tokens are preserved for continuous offset mapping. No text normalization is applied.

Summary

Functions

tokenize(text)

@spec tokenize(String.t()) :: [LangExtract.Alignment.Token.t()]