ExLLM.Infrastructure.Streaming.StreamBuffer (ex_llm v0.8.1)
View SourceEfficient circular buffer implementation for stream chunk management.
This module provides a memory-efficient circular buffer specifically designed for handling streaming LLM responses. It prevents unbounded memory growth while maintaining high performance for chunk processing.
Features
- Fixed-size circular buffer to prevent memory issues
- O(1) push and pop operations
- Configurable overflow strategies
- Buffer state monitoring and metrics
- Batch operations for efficiency
Example
# Create a buffer with capacity for 100 chunks
buffer = StreamBuffer.new(100)
# Push chunks
{:ok, buffer} = StreamBuffer.push(buffer, chunk1)
{:ok, buffer} = StreamBuffer.push(buffer, chunk2)
# Pop chunks
{:ok, chunk, buffer} = StreamBuffer.pop(buffer)
# Pop multiple chunks at once
{chunks, buffer} = StreamBuffer.pop_many(buffer, 5)
# Check buffer status
StreamBuffer.size(buffer) # => 42
StreamBuffer.fill_percentage(buffer) # => 42.0
Summary
Functions
Clears all chunks from the buffer.
Returns true if the buffer is empty.
Returns the fill percentage of the buffer (0.0-100.0).
Returns true if the buffer is full.
Creates a new stream buffer with the given capacity and options.
Pops a chunk from the buffer.
Pops up to n chunks from the buffer.
Pushes a chunk into the buffer.
Pushes a chunk, automatically handling overflow based on strategy.
Returns the current size of the buffer.
Returns buffer statistics.
Returns all chunks currently in the buffer as a list.
Types
@type overflow_strategy() :: :drop | :overwrite | :block
@type t() :: %ExLLM.Infrastructure.Streaming.StreamBuffer{ capacity: pos_integer(), data: :array.array(), head: non_neg_integer(), overflow_count: non_neg_integer(), overflow_strategy: overflow_strategy(), size: non_neg_integer(), tail: non_neg_integer(), total_popped: non_neg_integer(), total_pushed: non_neg_integer() }
Functions
Clears all chunks from the buffer.
Returns a new empty buffer with the same configuration.
Returns true if the buffer is empty.
Returns the fill percentage of the buffer (0.0-100.0).
Returns true if the buffer is full.
@spec new( pos_integer(), keyword() ) :: t()
Creates a new stream buffer with the given capacity and options.
Options
:overflow_strategy
- How to handle buffer overflow (default::drop
):drop
- Drop new chunks when buffer is full:overwrite
- Overwrite oldest chunks when buffer is full:block
- Return error when buffer is full (caller must handle)
Examples
# Basic buffer
buffer = StreamBuffer.new(100)
# With custom overflow strategy
buffer = StreamBuffer.new(50, overflow_strategy: :overwrite)
@spec pop(t()) :: {:ok, ExLLM.Types.StreamChunk.t(), t()} | {:empty, t()}
Pops a chunk from the buffer.
Returns {:ok, chunk, buffer}
or {:empty, buffer}
.
@spec pop_many(t(), pos_integer()) :: {[ExLLM.Types.StreamChunk.t()], t()}
Pops up to n chunks from the buffer.
Returns {chunks, buffer}
where chunks is a list of at most n chunks.
The chunks are returned in FIFO order.
@spec push(t(), ExLLM.Types.StreamChunk.t()) :: {:ok, t()} | {:overflow, t()}
Pushes a chunk into the buffer.
Returns {:ok, buffer}
if successful, or {:overflow, buffer}
if buffer
is full and overflow_strategy is :block
.
For :drop
strategy, the chunk is silently dropped.
For :overwrite
strategy, the oldest chunk is overwritten.
@spec push!(t(), ExLLM.Types.StreamChunk.t()) :: t()
Pushes a chunk, automatically handling overflow based on strategy.
This is a convenience function that always returns the buffer, making it suitable for use in pipelines.
@spec size(t()) :: non_neg_integer()
Returns the current size of the buffer.
Returns buffer statistics.
Includes current state metrics and lifetime counters.
@spec to_list(t()) :: [ExLLM.Types.StreamChunk.t()]
Returns all chunks currently in the buffer as a list.
Chunks are returned in FIFO order (oldest first). This is primarily for debugging and testing.