ExLLM.Infrastructure.Streaming.ChunkBatcher (ex_llm v0.8.1)
View SourceIntelligent chunk batching for optimized streaming output.
This GenServer batches small chunks together to reduce I/O operations and provide smoother output, especially for terminals and slow consumers.
Features
- Configurable batch size and timeout
- Adaptive batching based on chunk characteristics
- Min/max batch size constraints
- Performance metrics tracking
Example
# Start a batcher
{:ok, batcher} = ChunkBatcher.start_link(
batch_size: 5,
batch_timeout_ms: 25,
min_batch_size: 3
)
# Add chunks
:ok = ChunkBatcher.add_chunk(batcher, chunk1)
{:batch_ready, chunks} = ChunkBatcher.add_chunk(batcher, chunk2)
# Force flush
chunks = ChunkBatcher.flush(batcher)
Summary
Functions
Adds a chunk to the batcher.
Returns a specification to start this module under a supervisor.
Forces a flush of the current batch.
Gets current metrics from the batcher.
Starts a chunk batcher.
Stops the batcher, returning any remaining chunks.
Functions
@spec add_chunk(GenServer.server(), ExLLM.Types.StreamChunk.t()) :: :ok | {:batch_ready, [ExLLM.Types.StreamChunk.t()]}
Adds a chunk to the batcher.
Returns:
:ok
- Chunk added, batch not ready{:batch_ready, chunks}
- Batch is ready for processing
Returns a specification to start this module under a supervisor.
See Supervisor
.
@spec flush(GenServer.server()) :: [ExLLM.Types.StreamChunk.t()]
Forces a flush of the current batch.
Returns the list of chunks in the batch (may be empty).
@spec get_metrics(GenServer.server()) :: map()
Gets current metrics from the batcher.
@spec start_link(keyword()) :: GenServer.on_start()
Starts a chunk batcher.
Options
:batch_size
- Target batch size (default: 5):batch_timeout_ms
- Max time to wait for batch (default: 25ms):min_batch_size
- Minimum chunks before batching (default: 1):max_batch_size
- Maximum chunks per batch (default: 20):adaptive
- Enable adaptive batching (default: true):on_batch_ready
- Optional callback when batch is ready
@spec stop(GenServer.server()) :: [ExLLM.Types.StreamChunk.t()]
Stops the batcher, returning any remaining chunks.