LangExtract.Chunker (LangExtract v0.9.0)

Splits text into sentence-level chunks using the Alignment.Tokenizer.

Sentence boundary rules (mirroring upstream's find_sentence_range):

A :punctuation token ending in a sentence terminator (., !, ?, CJK equivalents — ... counts, since symbol runs are one token) ends a sentence, unless the previous token plus the terminator form a known abbreviation ("Dr" <> "." == "Dr.").
After sentence-ending punctuation, trailing closing punctuation (", ', ), ], }, », ”, ’) is consumed into the same sentence.
A :whitespace token containing \n starts a new sentence unless the next token begins lowercase — lines opening with quotes, digits, or capitals all break (upstream: "assume break unless lowercase").

Summary

Splits text into chunks respecting sentence boundaries.

@spec chunk(
  String.t(),
  keyword()
) :: [LangExtract.Chunker.Chunk.t()]

Splits text into chunks respecting sentence boundaries.