LangExtract.Chunker (LangExtract v0.11.0)

Copy Markdown View Source

Splits text into sentence-aware chunks, mirroring upstream's ChunkIterator.

A chunk is a token interval: its text runs from its first token's start to its last token's end. Whitespace between chunks belongs to no chunk, so chunks do not tile the source — matching upstream, whose token stream has no whitespace tokens. Byte offsets always slice the chunk's text back out of the source verbatim.

Sentence boundary rules (upstream's find_sentence_range semantics):

  1. A :punctuation token ending in a sentence terminator (., !, ?, CJK equivalents — ... counts, since symbol runs are one token) ends a sentence, unless the previous token plus the terminator form a known abbreviation ("Dr" <> "." == "Dr."). Whitespace is invisible here, so "Dr ." still reads as the abbreviation.
  2. After sentence-ending punctuation, trailing closing punctuation (", ', ), ], }, », ”, ’) is consumed into the same sentence — across any whitespace, so the quote opening the next line's dialogue attaches to the sentence before it, exactly as upstream.
  3. A token first on its line (its gap from the previous token contains \n or \r) starts a new sentence unless it begins lowercase — lines opening with quotes, digits, or capitals all break (upstream: "assume break unless lowercase").

Chunk assembly (mirroring upstream's ChunkIterator.__next__):

  1. A single token longer than the budget forms a chunk by itself.
  2. An oversized sentence is cut at the most recent newline within budget when one exists, else at the last token that fits; the remainder restarts sentence discovery mid-sentence.
  3. A chunk that completes a broken sentence never absorbs following sentences.
  4. Otherwise whole sentences pack into the chunk while they fit.

Budgets count characters (upstream's max_char_buffer unit — see chunk/2); offsets are bytes.

Departure from upstream's shape: every boundary rule is position-local, so sentence ends are precomputed for all positions in one backward pass, where upstream rescans forward from each chunk start — quadratic on boundary-free text (minified JSON, logs). Behavior is pinned two ways: the parity fixtures tie the line-comparable port to upstream, and the differential test ties this rewrite to that port, frozen as LangExtract.Test.ChunkerBaseline in test support.

Summary

Functions

chunk(text, opts)

Splits text into chunks respecting sentence boundaries.

Functions

chunk(text, opts)

@spec chunk(
  String.t(),
  keyword()
) :: [LangExtract.Chunker.Chunk.t()]

Splits text into chunks respecting sentence boundaries.

Options

  • :max_chunk_chars — maximum characters per chunk (required). Char-denominated to mirror upstream's max_char_buffer — chars are code points, Python's len unit — so chunk boundaries land identically across the two libraries; the cross-library benchmarks depend on that. Output offsets are bytes.