LangExtract.Chunker (LangExtract v0.4.0)

Copy Markdown View Source

Splits text into sentence-level chunks using the Alignment.Tokenizer.

Sentence boundary rules:

  1. A :punctuation token of ., !, or ? ends a sentence, unless it forms a known abbreviation with the preceding word token.
  2. After sentence-ending punctuation, trailing closing punctuation (", ', ), ], }, », ”, ’) is consumed into the same sentence.
  3. A :whitespace token containing \n followed by an uppercase-starting :word token starts a new sentence.

Summary

Functions

chunk(text, opts)

Splits text into chunks respecting sentence boundaries.

find_sentences(text)

Functions

chunk(text, opts)

@spec chunk(
  String.t(),
  keyword()
) :: [LangExtract.Chunker.Chunk.t()]

Splits text into chunks respecting sentence boundaries.

Options

  • :max_chunk_chars — maximum characters per chunk (required)

find_sentences(text)

@spec find_sentences(String.t()) :: [String.t()]