Arcana.Chunker.Default (Arcana v3.0.1)

Copy Markdown View Source

Default text chunker using the text_chunker library.

Supports multiple formats (plaintext, markdown, etc.) and can size chunks by characters or tokens.

Options

  • :chunk_size - Maximum chunk size (default: 450)
  • :chunk_overlap - Overlap between chunks (default: 50)
  • :format - Text format: :plaintext, :markdown, :elixir, etc. (default: :plaintext)
  • :size_unit - How to measure size: :characters or :tokens (default: :tokens)
  • :chars_per_token - Characters assumed per token when size_unit is :tokens (default: 4). A positive integer; anything else raises an ArgumentError
  • :max_chunk_chars - Ceiling on a chunk's length in bytes. A chunk over it is split rather than emitted (default: none). Splits land on grapheme boundaries, so a single grapheme longer than the budget - a ZWJ emoji sequence runs to 25 bytes - is emitted whole rather than cut into invalid UTF-8. Any budget worth setting is far above that. A positive integer or nil; anything else raises an ArgumentError

Sizing by tokens is an estimate, not a guarantee

With size_unit: :tokens the size is converted to characters by multiplying by :chars_per_token, which defaults to 4. That is a fair average for flowing English prose and not an upper bound. Dense text runs closer to 3 characters per token, and tables of numbers can approach 2, so the default 450 tokens can produce chunks an embedder with a 512 token window rejects:

413 Input validation error: 'inputs' must have less than 512 tokens. Given: 573

Nothing here can tell you that has happened, because the chunker does not run the model's tokenizer. Two ways to stay inside the budget:

  • lower :chars_per_token for a corpus you know is dense — product data sheets, code, chemical names, digit-heavy tables
  • set :max_chunk_chars as a backstop, so a pathological chunk is split instead of being handed to the embedder oversized

Remember any prefix your embedder adds (E5-style passage: / query:) comes out of the same budget.

Examples

Arcana.Chunker.Default.chunk("Hello world", chunk_size: 100)
Arcana.Chunker.Default.chunk(markdown_text, format: :markdown, chunk_size: 512)
Arcana.Chunker.Default.chunk(text, size_unit: :tokens, chunk_size: 256)