Behaviour for text chunking providers used by Arcana.

Arcana accepts any module that implements this behaviour. Built-in implementations are provided for:

• Arcana.Chunker.Default - Default chunking using text_chunker library

Configuration

Configure your chunking provider in config.exs:

# Default: text_chunker-based chunking
config :arcana, chunker: :default

# Default chunker with custom options
config :arcana, chunker: {:default, chunk_size: 512, chunk_overlap: 100}

# Custom function
config :arcana, chunker: fn text, opts -> [%{text: text, chunk_index: 0, token_count: 10}] end

# Custom module implementing this behaviour
config :arcana, chunker: MyApp.SemanticChunker
config :arcana, chunker: {MyApp.SemanticChunker, model: "..."}

Implementing a Custom Chunker

Create a module that implements this behaviour:

defmodule MyApp.SemanticChunker do
  @behaviour Arcana.Chunker

  @impl true
  def chunk(text, opts) do
    # Custom chunking logic...
    # Return list of chunk maps
    [
      %{text: "chunk 1", chunk_index: 0, token_count: 50},
      %{text: "chunk 2", chunk_index: 1, token_count: 45}
    ]
  end
end

Then configure:

config :arcana, chunker: {MyApp.SemanticChunker, model: "..."}

Chunk Format

Each chunk returned must be a map with at minimum:

• :text - The chunk text content (required)
• :chunk_index - Zero-based index of this chunk (required)
• :token_count - Estimated token count (required)
• :metadata - Optional map stored on the chunk as-is

Any other key is folded into the stored metadata too. Metadata round-trips through JSONB, so keys are stringified on the way in: a chunk of %{text: ..., chunk_index: 0, token_count: 8, page: 3} reads back as metadata["page"] == 3.

Arcana.Chunker.Default uses this to report each chunk's "start_byte"/"end_byte" range in the source text.