ReqLLM.Provider.ResponseBuilder behaviour (ReqLLM v1.13.0)

View Source

Behaviour for provider-specific Response assembly from StreamChunks.

Both streaming and non-streaming paths converge here, ensuring identical Response structs regardless of the code path taken.

Why This Exists

Different LLM providers have subtle differences in how they represent responses, tool calls, finish reasons, and metadata. Previously, these differences were handled in multiple places (streaming vs non-streaming, provider-specific decoders), leading to behavioral inconsistencies.

This behaviour centralizes provider-specific Response assembly logic, ensuring that:

  1. Streaming and non-streaming produce identical Response structs
  2. Provider quirks are handled in one place per provider
  3. New providers have a clear extension point

Provider-Specific Handling

  • Anthropic: Ensures content blocks are never empty (API requirement)
  • OpenAI Responses API: Propagates response_id for stateless multi-turn
  • Google: Detects functionCall to set correct finish_reason
  • Default (OpenAI Chat): Standard handling for OpenAI-compatible providers

Usage

Both streaming and non-streaming paths should:

  1. Decode wire format to [StreamChunk.t()]

  2. Collect metadata (usage, finish_reason, provider-specific)

  3. Call the appropriate builder:

    builder = ResponseBuilder.for_model(model) {:ok, response} = builder.build_response(chunks, metadata, opts)

Summary

Types

chunks()
metadata()
opts()

Callbacks

build_response(chunks, metadata, opts)

Build a Response struct from accumulated StreamChunks and metadata.

Functions

for_model(model)

Get the appropriate ResponseBuilder module for a given model.

Types

chunks()

@type chunks() :: [ReqLLM.StreamChunk.t()]

metadata()

@type metadata() :: %{
  optional(:usage) => map(),
  optional(:finish_reason) => atom(),
  optional(:response_id) => String.t(),
  optional(:provider_meta) => map()
}

opts()

@type opts() :: [context: ReqLLM.Context.t(), model: LLMDB.Model.t()]

Callbacks

build_response(chunks, metadata, opts)

@callback build_response(chunks(), metadata(), opts()) ::
  {:ok, ReqLLM.Response.t()} | {:error, term()}

Build a Response struct from accumulated StreamChunks and metadata.

This is the central point where provider-specific logic is applied to produce a consistent Response struct.

Parameters

  • chunks - List of StreamChunk structs from stream processing or converted from non-streaming response
  • metadata - Map containing usage, finish_reason, and provider-specific metadata collected during response processing
  • opts - Keyword list with :context and :model

Returns

  • {:ok, Response.t()} - Successfully built response
  • {:error, term()} - Build failed

Functions

for_model(model)

@spec for_model(LLMDB.Model.t()) :: module()

Get the appropriate ResponseBuilder module for a given model.

Routes to provider-specific builders based on model metadata:

  • Anthropic models → Anthropic.ResponseBuilder
  • Google/Gemini models → Google.ResponseBuilder
  • Vertex Claude models → Anthropic.ResponseBuilder
  • OpenAI Responses API models → OpenAI.ResponsesAPI.ResponseBuilder
  • All others → Provider.Defaults.ResponseBuilder

Examples

iex> model = ReqLLM.model!("anthropic:claude-3-haiku-20240307")
iex> ResponseBuilder.for_model(model)
ReqLLM.Providers.Anthropic.ResponseBuilder

iex> model = ReqLLM.model!("openai:gpt-4o")
iex> ResponseBuilder.for_model(model)
ReqLLM.Provider.Defaults.ResponseBuilder