Chunx.Chunker behaviour (chunx v0.2.1)

Defines the shared callback types for chunkers.

Chunker input must be valid UTF-8. Invalid text is returned as {:error, {:invalid_text, :invalid_utf8}} before tokenization begins.

Summary

chunk_result()

embedding_fun()

Splits text using a tokenizer.

@type chunk_result() ::
  {:ok, [Chunx.Chunk.t()] | [Chunx.SentenceChunk.t()]} | {:error, term()}

@type embedding_fun() :: ([String.t()] -> [Nx.Tensor.t()])

@callback chunk(
  text :: String.t(),
  tokenizer :: Chunx.Tokenizer.t(),
  opts_or_embedding_fun :: keyword() | embedding_fun()
) :: chunk_result()

Splits text using a tokenizer.

Semantic chunkers take an embedding function instead of options as the third argument and accept options as a fourth argument.

text - The text to chunk
tokenizer - The tokenizer to use
opts_or_embedding_fun - Options specific to the chunking strategy, or the embedding function used by a semantic chunker

(optional)

@callback chunk(
  text :: String.t(),
  tokenizer :: Chunx.Tokenizer.t(),
  embedding_fun(),
  opts :: keyword()
) :: chunk_result()