LangExtract.Alignment.Token (LangExtract v0.9.0)

Copy Markdown View Source

A token with its byte position in the source text.

Offsets are byte positions in the UTF-8 binary, matching Regex.scan/3 with return: :index and consumable by binary_part/3.

Internal — no stability guarantees; see the README's "Stability" section. Documented because it explains how the library works, not because it is API.

Summary

Types

t()
token_type()

Types

t()

@type t() :: %LangExtract.Alignment.Token{
  byte_end: non_neg_integer(),
  byte_start: non_neg_integer(),
  text: String.t(),
  type: token_type()
}

token_type()

@type token_type() :: :word | :number | :punctuation | :whitespace