ExLLM.Providers.Shared.StreamingBehavior behaviour (ex_llm v0.8.1)

View Source

Shared streaming behavior and utilities for ExLLM adapters.

Provides common patterns for handling Server-Sent Events (SSE) streaming responses from LLM APIs. This module can be used as a behavior or just for its utility functions.

Summary

Callbacks

Callback for parsing a streaming chunk into an ExLLM StreamChunk.

Functions

Accumulate streaming chunks into a complete response.

Create a stream chunk for function calls.

Create a stream chunk for text content.

Common streaming response handler that works with HTTPoison async responses.

Parse Server-Sent Events from a data stream.

Callbacks

parse_stream_chunk(t)

@callback parse_stream_chunk(String.t()) ::
  {:ok, ExLLM.Types.StreamChunk.t() | :done} | {:error, term()}

Callback for parsing a streaming chunk into an ExLLM StreamChunk.

Each provider has a different chunk format, so adapters must implement this.

Functions

accumulate_chunks(chunks)

@spec accumulate_chunks([ExLLM.Types.StreamChunk.t()]) :: map()

Accumulate streaming chunks into a complete response.

Useful for collecting all chunks before processing.

create_function_chunk(name, arguments, opts \\ [])

@spec create_function_chunk(String.t(), String.t(), keyword()) ::
  ExLLM.Types.StreamChunk.t()

Create a stream chunk for function calls.

create_text_chunk(text, opts \\ [])

@spec create_text_chunk(
  String.t(),
  keyword()
) :: ExLLM.Types.StreamChunk.t()

Create a stream chunk for text content.

handle_stream(ref, adapter_module, callback, opts \\ [])

@spec handle_stream(reference(), module(), function(), keyword()) ::
  {:ok, any()} | {:error, term()}

Common streaming response handler that works with HTTPoison async responses.

Options

  • :timeout - Stream timeout in milliseconds (default: 5 minutes)
  • :buffer - Initial buffer content (default: "")
  • :on_error - Error callback function

Examples

StreamingBehavior.handle_stream(ref, MyAdapter, fn chunk ->
  # Process each chunk
  send(self(), {:chunk, chunk})
end)

parse_sse_stream(data, buffer)

@spec parse_sse_stream(binary(), binary()) :: {[String.t()], binary()}

Parse Server-Sent Events from a data stream.

Returns a list of parsed events and the remaining buffer.