LangExtract.Alignment.Token (LangExtract v0.5.0)

Copy Markdown View Source

A token with its byte position in the source text.

Offsets are byte positions in the UTF-8 binary, matching Regex.scan/3 with return: :index and consumable by binary_part/3.

Summary

Types

t()
token_type()

Types

t()

@type t() :: %LangExtract.Alignment.Token{
  byte_end: non_neg_integer(),
  byte_start: non_neg_integer(),
  text: String.t(),
  type: token_type()
}

token_type()

@type token_type() :: :word | :number | :punctuation | :whitespace