ExLLM.Infrastructure.Streaming.FlowController (ex_llm v0.8.1)
View SourceAdvanced flow control for streaming LLM responses.
This GenServer manages the flow of chunks between fast producers (LLM APIs) and potentially slow consumers (terminals, network connections). It provides backpressure, rate limiting, and graceful degradation under load.
Features
- Producer/consumer pattern with async processing
- Automatic backpressure when consumers fall behind
- Configurable buffer thresholds
- Comprehensive metrics tracking
- Graceful error handling
Architecture
The FlowController sits between the streaming source and the consumer:
LLM API -> FlowController -> Consumer Callback
|
+-> Buffer (with backpressure)
+-> Metrics
+-> Rate Limiting
Example
# Start a flow controller
{:ok, controller} = FlowController.start_link(
consumer: callback_fn,
buffer_capacity: 100,
backpressure_threshold: 0.8
)
# Feed chunks
FlowController.push_chunk(controller, chunk)
# Get metrics
metrics = FlowController.get_metrics(controller)
Summary
Functions
Returns a specification to start this module under a supervisor.
Signals that the stream is complete.
Gets current metrics from the flow controller.
Gets the current status of the flow controller.
Pushes a chunk to the flow controller.
Starts a flow controller.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor
.
@spec complete_stream(GenServer.server()) :: :ok
Signals that the stream is complete.
@spec get_metrics(GenServer.server()) :: ExLLM.Infrastructure.Streaming.FlowController.Metrics.t()
Gets current metrics from the flow controller.
@spec get_status(GenServer.server()) :: map()
Gets the current status of the flow controller.
@spec push_chunk(GenServer.server(), ExLLM.Types.StreamChunk.t()) :: :ok | {:error, :backpressure}
Pushes a chunk to the flow controller.
Returns :ok
or {:error, :backpressure}
if backpressure is active.
@spec start_link(keyword()) :: GenServer.on_start()
Starts a flow controller.
Options
:consumer
- Consumer callback function (required):buffer_capacity
- Maximum buffer size (default: 100):backpressure_threshold
- Buffer fill ratio to trigger backpressure (default: 0.8):rate_limit_ms
- Minimum time between chunks in ms (default: 1):overflow_strategy
- Buffer overflow strategy (default: :drop):batch_config
- Chunk batching configuration (optional):on_metrics
- Callback for metrics reports (optional)