ExLLM.Infrastructure.Streaming.ChunkBatcher (ex_llm v0.8.1)

View Source

Intelligent chunk batching for optimized streaming output.

This GenServer batches small chunks together to reduce I/O operations and provide smoother output, especially for terminals and slow consumers.

Features

  • Configurable batch size and timeout
  • Adaptive batching based on chunk characteristics
  • Min/max batch size constraints
  • Performance metrics tracking

Example

# Start a batcher
{:ok, batcher} = ChunkBatcher.start_link(
  batch_size: 5,
  batch_timeout_ms: 25,
  min_batch_size: 3
)

# Add chunks
:ok = ChunkBatcher.add_chunk(batcher, chunk1)
{:batch_ready, chunks} = ChunkBatcher.add_chunk(batcher, chunk2)

# Force flush
chunks = ChunkBatcher.flush(batcher)

Summary

Functions

Adds a chunk to the batcher.

Returns a specification to start this module under a supervisor.

Forces a flush of the current batch.

Gets current metrics from the batcher.

Starts a chunk batcher.

Stops the batcher, returning any remaining chunks.

Functions

add_chunk(batcher, chunk)

@spec add_chunk(GenServer.server(), ExLLM.Types.StreamChunk.t()) ::
  :ok | {:batch_ready, [ExLLM.Types.StreamChunk.t()]}

Adds a chunk to the batcher.

Returns:

  • :ok - Chunk added, batch not ready
  • {:batch_ready, chunks} - Batch is ready for processing

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

flush(batcher)

Forces a flush of the current batch.

Returns the list of chunks in the batch (may be empty).

get_metrics(batcher)

@spec get_metrics(GenServer.server()) :: map()

Gets current metrics from the batcher.

start_link(opts \\ [])

@spec start_link(keyword()) :: GenServer.on_start()

Starts a chunk batcher.

Options

  • :batch_size - Target batch size (default: 5)
  • :batch_timeout_ms - Max time to wait for batch (default: 25ms)
  • :min_batch_size - Minimum chunks before batching (default: 1)
  • :max_batch_size - Maximum chunks per batch (default: 20)
  • :adaptive - Enable adaptive batching (default: true)
  • :on_batch_ready - Optional callback when batch is ready

stop(batcher)

Stops the batcher, returning any remaining chunks.