ExLLM.Infrastructure.Streaming.StreamBuffer (ex_llm v0.8.1)

View Source

Efficient circular buffer implementation for stream chunk management.

This module provides a memory-efficient circular buffer specifically designed for handling streaming LLM responses. It prevents unbounded memory growth while maintaining high performance for chunk processing.

Features

  • Fixed-size circular buffer to prevent memory issues
  • O(1) push and pop operations
  • Configurable overflow strategies
  • Buffer state monitoring and metrics
  • Batch operations for efficiency

Example

# Create a buffer with capacity for 100 chunks
buffer = StreamBuffer.new(100)

# Push chunks
{:ok, buffer} = StreamBuffer.push(buffer, chunk1)
{:ok, buffer} = StreamBuffer.push(buffer, chunk2)

# Pop chunks
{:ok, chunk, buffer} = StreamBuffer.pop(buffer)

# Pop multiple chunks at once
{chunks, buffer} = StreamBuffer.pop_many(buffer, 5)

# Check buffer status
StreamBuffer.size(buffer)         # => 42
StreamBuffer.fill_percentage(buffer) # => 42.0

Summary

Functions

Clears all chunks from the buffer.

Returns true if the buffer is empty.

Returns the fill percentage of the buffer (0.0-100.0).

Returns true if the buffer is full.

Creates a new stream buffer with the given capacity and options.

Pops a chunk from the buffer.

Pops up to n chunks from the buffer.

Pushes a chunk into the buffer.

Pushes a chunk, automatically handling overflow based on strategy.

Returns the current size of the buffer.

Returns buffer statistics.

Returns all chunks currently in the buffer as a list.

Types

overflow_strategy()

@type overflow_strategy() :: :drop | :overwrite | :block

t()

@type t() :: %ExLLM.Infrastructure.Streaming.StreamBuffer{
  capacity: pos_integer(),
  data: :array.array(),
  head: non_neg_integer(),
  overflow_count: non_neg_integer(),
  overflow_strategy: overflow_strategy(),
  size: non_neg_integer(),
  tail: non_neg_integer(),
  total_popped: non_neg_integer(),
  total_pushed: non_neg_integer()
}

Functions

clear(buffer)

@spec clear(t()) :: t()

Clears all chunks from the buffer.

Returns a new empty buffer with the same configuration.

empty?(arg1)

@spec empty?(t()) :: boolean()

Returns true if the buffer is empty.

fill_percentage(stream_buffer)

@spec fill_percentage(t()) :: float()

Returns the fill percentage of the buffer (0.0-100.0).

full?(stream_buffer)

@spec full?(t()) :: boolean()

Returns true if the buffer is full.

new(capacity, opts \\ [])

@spec new(
  pos_integer(),
  keyword()
) :: t()

Creates a new stream buffer with the given capacity and options.

Options

  • :overflow_strategy - How to handle buffer overflow (default: :drop)
    • :drop - Drop new chunks when buffer is full
    • :overwrite - Overwrite oldest chunks when buffer is full
    • :block - Return error when buffer is full (caller must handle)

Examples

# Basic buffer
buffer = StreamBuffer.new(100)

# With custom overflow strategy
buffer = StreamBuffer.new(50, overflow_strategy: :overwrite)

pop(buffer)

@spec pop(t()) :: {:ok, ExLLM.Types.StreamChunk.t(), t()} | {:empty, t()}

Pops a chunk from the buffer.

Returns {:ok, chunk, buffer} or {:empty, buffer}.

pop_many(buffer, n)

@spec pop_many(t(), pos_integer()) :: {[ExLLM.Types.StreamChunk.t()], t()}

Pops up to n chunks from the buffer.

Returns {chunks, buffer} where chunks is a list of at most n chunks. The chunks are returned in FIFO order.

push(buffer, chunk)

@spec push(t(), ExLLM.Types.StreamChunk.t()) :: {:ok, t()} | {:overflow, t()}

Pushes a chunk into the buffer.

Returns {:ok, buffer} if successful, or {:overflow, buffer} if buffer is full and overflow_strategy is :block.

For :drop strategy, the chunk is silently dropped. For :overwrite strategy, the oldest chunk is overwritten.

push!(buffer, chunk)

@spec push!(t(), ExLLM.Types.StreamChunk.t()) :: t()

Pushes a chunk, automatically handling overflow based on strategy.

This is a convenience function that always returns the buffer, making it suitable for use in pipelines.

size(stream_buffer)

@spec size(t()) :: non_neg_integer()

Returns the current size of the buffer.

stats(buffer)

@spec stats(t()) :: map()

Returns buffer statistics.

Includes current state metrics and lifetime counters.

to_list(buffer)

@spec to_list(t()) :: [ExLLM.Types.StreamChunk.t()]

Returns all chunks currently in the buffer as a list.

Chunks are returned in FIFO order (oldest first). This is primarily for debugging and testing.