ALLM.Providers.OpenAI.Speech (allm v0.6.1)

Copy Markdown View Source

OpenAI text-to-speech adapter. Implements ALLM.SpeechAdapter and ALLM.SpeechStreamAdapter against POST /v1/audio/speech.

Layer B — runtime. Wire it with ALLM.Engine.new(speech_adapter: ALLM.Providers.OpenAI.Speech) and call it through ALLM.synthesize/3. The key resolves via ALLM.Keys.fetch!(:openai, opts) at request-build time, so no key ever lives on the engine.

req = ALLM.SpeechRequest.new(input: "Hello.", voice: "coral", format: :wav)
{:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, api_key: "sk-...")
{:ok, wav_bytes} = ALLM.Audio.to_binary(resp.audio)

Wire-field map

Observed live on 2026-09-24 by scripts/record_openai_audio_fixtures.exs, whose probe halts the recording pass on any mismatch.

ConcernOpenAI
EndpointPOST https://api.openai.com/v1/audio/speech, JSON body
Authauthorization: Bearer <key>
inputALLM.SpeechRequest.input, at most 4096 code points
model:model, or gpt-4o-mini-tts when nil
voice:voice, or alloy when nil. Forwarded verbatim; OpenAI's valid set differs per model
response_format:format as a string (mp3, opus, aac, flac, wav, pcm); omitted when nil, and OpenAI answers mp3. :ulaw and :alaw have no OpenAI value and are refused (see gates)
instructions, speedforwarded when set, omitted when nil. OpenAI accepts instructions on tts-1 with a 200 even though it documents that model as ignoring them
OptionsALLM.SpeechRequest.options merged under the fields above; stream_format is dropped
200 bodyraw audio bytes; content-type names the format
Streamingthe same request; the body arrives with transfer-encoding: chunked, and its first bytes arrive while the rest is still being synthesized (observed 2026-09-26 on gpt-4o-mini-tts and tts-1)
sample_ratenot sent. pcm and wav are always 24,000 Hz, so the response reports 24000 for them and nil otherwise
Usagenone: the body is raw audio, so :usage is an all-nil %ALLM.Usage{}
Correlationx-request-id response header
Error envelope{"error": {"message", "type", "param", "code"}}; the 401 is sent as text/plain
Unknown fieldsignored (200), so a mistyped :options key does nothing and raises no error

Adapter-injected defaults

  • model defaults to "gpt-4o-mini-tts" when :model is nil.
  • voice defaults to "alloy" when :voice is nil. OpenAI requires a voice, and alloy is accepted by every model probed.
  • The HTTP receive timeout defaults to 60 s when opts[:request_timeout] is absent.

Pre-flight gates

Before any HTTP I/O and before ALLM.Keys.fetch!/2, so a request that is going to be rejected never needs a key:

  1. Input shape. A non-binary, empty, or non-UTF-8 :input → :invalid_request with metadata.field: :input.
  2. Input length. More than 4096 code points → :context_length_exceeded with metadata.count and metadata.max. Code points, not graphemes and not bytes: an accented letter written as a base letter plus a combining mark counts twice, and a precomposed one counts once. The unit was settled by a live probe on 2026-09-24. A provider 400 whose message names string_too_long maps to the same reason, so a caller whose model has another limit sees one reason.
  3. Format. :ulaw and :alaw (8 kHz telephony audio) have no response_format value → :unsupported_feature with metadata.field: :format. This runs before the sample-rate gate, so a telephony request at 8,000 Hz names the format.
  4. Sample rate. OpenAI cannot produce a requested rate. For :pcm and :wav a sample_rate of nil or 24000 passes; for every other format (and for a nil format, which OpenAI answers as mp3) only nil passes. Anything else is :unsupported_feature with metadata.field: :sample_rate.

synthesize/2, prepare_request/2 and stream_synthesize/2 run the same four gates.

Response

:audio is %ALLM.Audio{source: {:binary, bytes}}. :format is derived from the response content-type through ALLM.SpeechResponse.mime_to_format/1, and :audio.mime_type is that format's canonical MIME type (or the raw content type when it is an audio/* type outside the table). A 200 whose content type is not audio/*, or whose body is empty, is :malformed_response. :raw is always nil: the audio bytes live once, in :audio. :model is the model that was sent.

opts[:request_id] is reflected onto response.request_id; when absent, OpenAI's x-request-id header is used. request.metadata round-trips onto response.metadata.

The x-request-id fallback is unreachable through ALLM.synthesize/3

The façade always supplies opts[:request_id] (it generates one when the caller does not), so on that path response.request_id is the façade's id and OpenAI's own correlation id is never observed. It surfaces only on a direct synthesize/2 / decode_response/4 call that omits opts[:request_id]. Identical to ALLM.Providers.OpenAI.Moderation.

Streaming

stream_synthesize/2 sends the same request with Finch.async_request/3 on the ALLM.Finch pool (HTTP/1) instead of Req, and returns a lazy stream of ALLM.SpeechEvent values: :speech_started when the response headers arrive, one :audio_delta per body chunk, then :speech_completed. Usage is an all-nil %ALLM.Usage{}, as on synthesize/2. stream_synthesize_input/3 is not implemented: OpenAI's text-in streaming is a different API.

An HTTP error status is not classified from the status alone. The error body is collected until the response ends and then classified exactly as synthesize/2 classifies it, so the key redactor and the string_too_long rule see the provider's message. A stream is never retried once it has been returned.

Retry

Each attempt runs inside ALLM.Retry.run/3 under opts[:retry] (default :default). The attempt marks :rate_limited (honouring Retry-After), :provider_unavailable, :timeout and :network_error as retryable, and the policy decides. The default policy's retry_on is HTTP codes plus :timeout, so on its own this loop retries :timeout only. Through ALLM.synthesize/3 the façade retries all four reasons, and :timeout is retried by both loops: up to 9 attempts at the default policy, against 3 for the other three reasons.

Error-struct hygiene

%ALLM.Error.SpeechAdapterError{} derives Jason.Encoder, so it is often logged. No raw response body, request header or Authorization value is copied into it, and there is no body preview. Every provider-authored string it carries (:message, metadata.openai_code, metadata.openai_type) passes a redactor that replaces key-shaped tokens with [REDACTED]. OpenAI's real 401 masks the key it echoes (sk-proj-*****…9900, observed 2026-09-24); the redactor is defence in depth.

Not every error is JSON-encodable

The struct derives Jason.Encoder, but a :timeout, :network_error or invalid-JSON :malformed_response error carries an exception struct (Req.TransportError, Jason.DecodeError) on :cause, and Jason.encode!/1 raises Protocol.UndefinedError on it. Drop or Exception.message/1 the :cause before encoding. The image, embedding and moderation adapter errors share this limitation.

Test-injection escape hatch

synthesize/2 honours opts[:adapter_opts][:speech_script]: when the key is present, the call is handed to ALLM.Providers.FakeSpeech.synthesize/2 BEFORE any of this adapter's gates run. This is what lets the ALLM.SpeechAdapter conformance suite drive a real adapter without an HTTP stub. prepare_request/2 returns a stub error under the same key, because a scripted response has no Req.Request analogue.

Summary

Functions

Return an unfired Req.Request configured exactly as synthesize/2 would fire it, for callers who add headers, middleware or their own retry loop.

Stream speech for request.input from OpenAI as it is synthesized.

Synthesize speech from request.input against OpenAI.

Functions

prepare_request(request, opts)

@spec prepare_request(
  ALLM.SpeechRequest.t(),
  keyword()
) :: {:ok, Req.Request.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}

Return an unfired Req.Request configured exactly as synthesize/2 would fire it, for callers who add headers, middleware or their own retry loop.

The pre-flight gates run first, so this is defined only for an input that passes them. Under opts[:adapter_opts][:speech_script] it returns a stub error instead of delegating to ALLM.Providers.FakeSpeech.

Examples

iex> req = ALLM.SpeechRequest.new(input: "Hi.")
iex> {:ok, http} = ALLM.Providers.OpenAI.Speech.prepare_request(req, api_key: "sk-x")
iex> URI.to_string(http.url)
"https://api.openai.com/v1/audio/speech"

stream_synthesize(request, opts)

@spec stream_synthesize(
  ALLM.SpeechRequest.t(),
  keyword()
) ::
  {:ok, Enumerable.t(ALLM.SpeechEvent.t())}
  | {:error, ALLM.Error.SpeechAdapterError.t()}

Stream speech for request.input from OpenAI as it is synthesized.

Returns {:ok, enumerable} of ALLM.SpeechEvent values, or {:error, %ALLM.Error.SpeechAdapterError{}} from a pre-flight gate. The four gates of synthesize/2 run first, then ALLM.Keys.fetch!/2 (which raises %ALLM.Error.EngineError{reason: :missing_key} by design). No HTTP request is made until the enumerable is reduced.

Injected defaults: the same model ("gpt-4o-mini-tts") and voice ("alloy") as synthesize/2, and the same JSON body.

Options:

  • :stream_timeout — milliseconds of silence between two transport messages before the stream ends with {:error, %SpeechAdapterError{reason: :timeout}}. Default 60000. The transport's own receive timeout defaults above it (see ALLM.Providers.Support.Transport).
  • :receive_timeout, :request_timeout, :pool_timeout — forwarded to Finch.async_request/3.
  • :finch_name — the Finch pool (default ALLM.Finch).
  • :finch_module — the module called for async_request/3 and cancel_async_request/1 (default Finch); tests pass ALLM.Test.FinchStub.

Each transport option is read from the top level of opts. ALLM.stream_synthesize/3 hoists an engine's adapter_opts: transport keys (ALLM.Adapter.transport_opts/0) there, as the chat façades do; a direct call passes them at the top level.

Events: :speech_started (with sample_rate 24000 for :pcm and :wav, else nil) once the response headers arrive, one :audio_delta per non-empty body chunk, and :speech_completed with an all-nil usage. A failure ends the stream with one {:error, _}:

  • an HTTP error status: the error body is collected until the response ends, then classified as synthesize/2 classifies it (a 401 is :authentication_failed, a string_too_long 400 is :context_length_exceeded), with provider strings redacted;
  • a 200 whose content-type is not audio/*: :malformed_response;
  • a 200 with no audio bytes: :invalid_request with metadata.cause: :empty_input;
  • no transport message within :stream_timeout: :timeout;
  • a transport failure: :network_error.

Halting the stream early (Enum.take/2) cancels the HTTP request and removes the request's already-queued messages from the calling process's mailbox. This is best-effort: a message the request process sends while it is being shut down can still arrive afterwards.

opts[:adapter_opts][:speech_script] hands the call to ALLM.Providers.FakeSpeech.stream_synthesize/2 before any gate runs.

Examples

iex> req = ALLM.SpeechRequest.new(input: "Hello.", format: :pcm)
iex> opts = [adapter_opts: [speech_script: [{:ok, "PCM-bytes"}]]]
iex> {:ok, events} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, opts)
iex> for {:audio_delta, bytes} <- events, into: "", do: bytes
"PCM-bytes"

iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :mp3, sample_rate: 24_000)
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
iex> err.reason
:unsupported_feature

iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :ulaw)
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
iex> {err.reason, err.metadata.field}
{:unsupported_feature, :format}

synthesize(request, opts)

@spec synthesize(
  ALLM.SpeechRequest.t(),
  keyword()
) ::
  {:ok, ALLM.SpeechResponse.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}

Synthesize speech from request.input against OpenAI.

Returns {:ok, %ALLM.SpeechResponse{}} or {:error, %ALLM.Error.SpeechAdapterError{}}. The one exception is ALLM.Keys.fetch!/2, which raises %ALLM.Error.EngineError{reason: :missing_key} by design; all four pre-flight gates run ahead of it.

Sample rate: response.sample_rate is 24000 for :pcm and :wav and nil for every other format. A request sample_rate other than nil (or 24000 for :pcm / :wav) is refused with :unsupported_feature before any I/O.

Formats: :ulaw and :alaw are refused with :unsupported_feature and metadata.field: :format before any I/O; OpenAI has no telephony output.

Injected defaults: model is "gpt-4o-mini-tts" and voice is "alloy" when the request leaves them nil, and the receive timeout is 60 s when opts[:request_timeout] is absent.

Retry: this adapter's own ALLM.Retry.run/3 loop retries :timeout under the default policy. :rate_limited, :provider_unavailable and :network_error are retryable too, but only under a caller-supplied opts[:retry] that lists them (as ALLM.synthesize/3's own loop does).

See the module documentation for the gates, the wire-field map and the adapter_opts[:speech_script] test-injection short-circuit.

Examples

iex> req = ALLM.SpeechRequest.new(input: "Hello.")
iex> opts = [adapter_opts: [speech_script: [{:ok, "ID3-bytes"}]]]
iex> {:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, opts)
iex> ALLM.Audio.to_binary(resp.audio)
{:ok, "ID3-bytes"}

iex> req = ALLM.SpeechRequest.new(input: String.duplicate("a", 4097))
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.synthesize(req, [])
iex> {err.reason, err.metadata}
{:context_length_exceeded, %{count: 4097, max: 4096}}