Arcana.Chunker.Custom (Arcana v2.0.1)

Copy Markdown View Source

Custom chunking provider using user-provided functions.

This module wraps a user-provided function to implement the Arcana.Chunker behaviour. It's used internally when configuring a function as the chunker.

Configuration

# Function that returns list of chunk maps
config :arcana, chunker: fn text, opts ->
  [%{text: text, chunk_index: 0, token_count: div(String.length(text), 4)}]
end

The function must:

  • Accept text (string) and opts (keyword list)
  • Return a list of maps, each with :text, :chunk_index, and :token_count