AI.Splitter (fnord v0.8.24)

View Source

This module is used to split a string into chunks by the number of tokens, while accounting for other data that might be going with it to the API endpoint with the limited token count.

For example, the search entry agent may be processing a large file, one that must be split into 3 slices just to fit it into the payload of an API call. In order to retain context between chunks, the agent essentially reduces over the file, keeping track of information in the previous chunks to generate a final summary. Doing that means that we need to not only split the file by the number of tokens in each slice, but also keep some space for the bespoke data that will be added to the payload as the agent's "accumulator".

Summary

Functions

Returns the next chunk and updated splitter state, accounting for the bespoke input tokens.

Types

t()

@type t() :: %AI.Splitter{done: boolean(), input: binary(), model: AI.Model.t()}

Functions

new(input, model)

next_chunk(tok, bespoke_input)

next_chunk(tok, bespoke_input, max_chunk_tokens)

Returns the next chunk and updated splitter state, accounting for the bespoke input tokens.

Optionally, a max_chunk_tokens can be provided to limit the chunk size explicitly.