ExLLM.Providers.Shared.EnhancedStreamingCoordinator (ex_llm v0.8.1)
View SourceEnhanced streaming coordinator with advanced flow control, intelligent batching, and sophisticated buffering strategies.
This module extends the basic StreamingCoordinator with our advanced streaming infrastructure while maintaining backward compatibility. It provides:
- Advanced flow control with backpressure (FlowController)
- Intelligent chunk batching with adaptive sizing (ChunkBatcher)
- Circular buffering with overflow strategies (StreamBuffer)
- Real-time metrics and performance monitoring
- Error recovery and graceful degradation
Enhanced Features
- Backpressure Control: Automatically handles slow consumers by applying backpressure when buffers fill up, preventing memory exhaustion
- Adaptive Batching: Intelligent chunk batching that adapts to chunk size and arrival rate for optimal performance
- Multiple Overflow Strategies: Choose between drop, overwrite, or block strategies when buffers are full
- Real-time Metrics: Comprehensive metrics tracking including throughput, latency, buffer utilization, and error rates
- Advanced Error Recovery: Graceful handling of consumer errors with automatic retry and recovery mechanisms
Usage
# Basic enhanced streaming (same as original)
{:ok, stream_id} = EnhancedStreamingCoordinator.start_stream(
url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1
)
# Advanced streaming with flow control
{:ok, stream_id} = EnhancedStreamingCoordinator.start_stream(
url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1,
enable_flow_control: true,
buffer_capacity: 100,
backpressure_threshold: 0.8,
overflow_strategy: :drop
)
# Intelligent batching
{:ok, stream_id} = EnhancedStreamingCoordinator.start_stream(
url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1,
enable_batching: true,
batch_size: 5,
batch_timeout_ms: 25,
adaptive_batching: true
)
# Combined advanced features
{:ok, stream_id} = EnhancedStreamingCoordinator.start_stream(
url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1,
enable_flow_control: true,
enable_batching: true,
buffer_capacity: 50,
backpressure_threshold: 0.9,
batch_size: 3,
adaptive_batching: true,
on_metrics: &handle_metrics/1
)
Configuration Options
Flow Control Options
:enable_flow_control
- Enable advanced flow control (default: false):buffer_capacity
- Buffer capacity in chunks (default: 100):backpressure_threshold
- Buffer fill ratio to trigger backpressure (default: 0.8):overflow_strategy
- Strategy when buffer overflows::drop
,:overwrite
,:block
(default::drop
):rate_limit_ms
- Minimum time between chunks in ms (default: 1)
Batching Options
:enable_batching
- Enable intelligent chunk batching (default: false):batch_size
- Target batch size (default: 5):batch_timeout_ms
- Max time to wait for batch (default: 25):adaptive_batching
- Enable adaptive batch sizing (default: true):min_batch_size
- Minimum chunks before batching (default: 1):max_batch_size
- Maximum chunks per batch (default: 20)
Monitoring Options
:track_detailed_metrics
- Enable detailed metrics tracking (default: false):on_metrics
- Callback for real-time metrics reports:metrics_interval_ms
- Metrics reporting interval (default: 1000)
Backward Compatibility
This module is fully backward compatible with the original StreamingCoordinator. When advanced features are not explicitly enabled, it behaves identically to the original implementation.
Summary
Functions
Create an enhanced stream collector with optional flow control and batching.
Execute enhanced streaming with advanced infrastructure.
Create a simple enhanced streaming implementation.
Start an enhanced streaming request with optional advanced features.
Functions
Create an enhanced stream collector with optional flow control and batching.
Execute enhanced streaming with advanced infrastructure.
Create a simple enhanced streaming implementation.
This provides the same interface as StreamingCoordinator.simple_stream/1 but with optional enhanced features.
Example
EnhancedStreamingCoordinator.simple_stream(
url: url,
request: request,
headers: headers,
callback: callback,
parse_chunk: &parse_chunk/1,
enable_flow_control: true,
buffer_capacity: 50,
enable_batching: true,
batch_size: 3
)
@spec start_stream(String.t(), map(), list(), function(), keyword()) :: {:ok, String.t()} | {:error, term()}
Start an enhanced streaming request with optional advanced features.
This function provides the same interface as the original StreamingCoordinator but with additional options for enabling advanced streaming features.
Enhanced Options
In addition to all original options, supports:
- Flow control options (see module documentation)
- Batching options (see module documentation)
- Monitoring options (see module documentation)
Examples
# Basic usage (identical to original StreamingCoordinator)
{:ok, stream_id} = start_stream(url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1
)
# With advanced flow control
{:ok, stream_id} = start_stream(url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1,
enable_flow_control: true,
buffer_capacity: 50,
backpressure_threshold: 0.9
)
# With intelligent batching
{:ok, stream_id} = start_stream(url, request, headers, callback,
parse_chunk_fn: &parse_chunk/1,
enable_batching: true,
batch_size: 3,
adaptive_batching: true
)