ALLM.Providers.Support.Transport (allm v0.6.1)

Copy Markdown View Source

Shared HTTP-transport option handling and Finch async plumbing for the streaming provider adapters.

Streaming adapters open their request with Finch.async_request/3 and forward a small set of transport opts verbatim (see ALLM.Adapter.transport_opts/0 for the full neutral list, which also covers the non-streaming Req path).

Why this module exists

Finch's HTTP/1 pool defaults :receive_timeout to 15,000 ms (Finch.HTTP1.Pool), and that timer covers the gap before the first byte of the response body. Reasoning models (gpt-5.x, extended-thinking Claude) routinely spend longer than that thinking before the first SSE chunk is emitted, so the transport killed the request mid-think and the adapter reported %AdapterError{reason: :network_error} — a misleading error for what is really "the model is still working".

Worse, ALLM's own inter-event timer (opts[:stream_timeout], default 60,000 ms) could never fire: the 15 s transport timer always won the race, so raising :stream_timeout alone had no effect.

finch_opts/2 fixes both halves: :stream_timeout becomes the single governing knob, and the transport timer is defaulted above it (by 30,000 ms of headroom) so it is a backstop rather than the primary timer. An explicit opts[:receive_timeout] still wins.

Summary

Functions

The opts forwarded verbatim to Finch.async_request/3.

Build the extra-opts keyword forwarded to Finch.async_request/3.

Functions

finch_forwarded_opts()

@spec finch_forwarded_opts() :: [atom()]

The opts forwarded verbatim to Finch.async_request/3.

finch_opts(opts, stream_timeout)

@spec finch_opts(
  keyword(),
  timeout()
) :: keyword()

Build the extra-opts keyword forwarded to Finch.async_request/3.

Takes the resolved adapter opts and the already-resolved stream_timeout (the value the adapter passes to its receive ... after clause). Returns the subset of opts Finch understands, with :receive_timeout defaulted to stream_timeout + 30000 when the caller did not set it explicitly.

A stream_timeout of :infinity yields receive_timeout: :infinity.

Examples

iex> ALLM.Providers.Support.Transport.finch_opts([], 60_000)
[receive_timeout: 90_000]

iex> ALLM.Providers.Support.Transport.finch_opts([receive_timeout: 5_000], 60_000)
[receive_timeout: 5_000]

iex> ALLM.Providers.Support.Transport.finch_opts([], :infinity)
[receive_timeout: :infinity]