OpenAI text-to-speech adapter. Implements ALLM.SpeechAdapter and
ALLM.SpeechStreamAdapter against POST /v1/audio/speech.
Layer B — runtime. Wire it with
ALLM.Engine.new(speech_adapter: ALLM.Providers.OpenAI.Speech) and call it
through ALLM.synthesize/3. The key resolves via
ALLM.Keys.fetch!(:openai, opts) at request-build time, so no key ever
lives on the engine.
req = ALLM.SpeechRequest.new(input: "Hello.", voice: "coral", format: :wav)
{:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, api_key: "sk-...")
{:ok, wav_bytes} = ALLM.Audio.to_binary(resp.audio)Wire-field map
Observed live on 2026-09-24 by scripts/record_openai_audio_fixtures.exs,
whose probe halts the recording pass on any mismatch.
| Concern | OpenAI |
|---|---|
| Endpoint | POST https://api.openai.com/v1/audio/speech, JSON body |
| Auth | authorization: Bearer <key> |
input | ALLM.SpeechRequest.input, at most 4096 code points |
model | :model, or gpt-4o-mini-tts when nil |
voice | :voice, or alloy when nil. Forwarded verbatim; OpenAI's valid set differs per model |
response_format | :format as a string (mp3, opus, aac, flac, wav, pcm); omitted when nil, and OpenAI answers mp3. :ulaw and :alaw have no OpenAI value and are refused (see gates) |
instructions, speed | forwarded when set, omitted when nil. OpenAI accepts instructions on tts-1 with a 200 even though it documents that model as ignoring them |
| Options | ALLM.SpeechRequest.options merged under the fields above; stream_format is dropped |
| 200 body | raw audio bytes; content-type names the format |
| Streaming | the same request; the body arrives with transfer-encoding: chunked, and its first bytes arrive while the rest is still being synthesized (observed 2026-09-26 on gpt-4o-mini-tts and tts-1) |
sample_rate | not sent. pcm and wav are always 24,000 Hz, so the response reports 24000 for them and nil otherwise |
| Usage | none: the body is raw audio, so :usage is an all-nil %ALLM.Usage{} |
| Correlation | x-request-id response header |
| Error envelope | {"error": {"message", "type", "param", "code"}}; the 401 is sent as text/plain |
| Unknown fields | ignored (200), so a mistyped :options key does nothing and raises no error |
Adapter-injected defaults
modeldefaults to"gpt-4o-mini-tts"when:modelisnil.voicedefaults to"alloy"when:voiceisnil. OpenAI requires a voice, andalloyis accepted by every model probed.- The HTTP receive timeout defaults to 60 s
when
opts[:request_timeout]is absent.
Pre-flight gates
Before any HTTP I/O and before ALLM.Keys.fetch!/2, so a request that is
going to be rejected never needs a key:
- Input shape. A non-binary, empty, or non-UTF-8
:input→:invalid_requestwithmetadata.field: :input. - Input length. More than 4096 code points →
:context_length_exceededwithmetadata.countandmetadata.max. Code points, not graphemes and not bytes: an accented letter written as a base letter plus a combining mark counts twice, and a precomposed one counts once. The unit was settled by a live probe on 2026-09-24. A provider 400 whose message namesstring_too_longmaps to the same reason, so a caller whose model has another limit sees one reason. - Format.
:ulawand:alaw(8 kHz telephony audio) have noresponse_formatvalue →:unsupported_featurewithmetadata.field: :format. This runs before the sample-rate gate, so a telephony request at 8,000 Hz names the format. - Sample rate. OpenAI cannot produce a requested rate. For
:pcmand:wavasample_rateofnilor24000passes; for every other format (and for anilformat, which OpenAI answers asmp3) onlynilpasses. Anything else is:unsupported_featurewithmetadata.field: :sample_rate.
synthesize/2, prepare_request/2 and stream_synthesize/2 run the same
four gates.
Response
:audio is %ALLM.Audio{source: {:binary, bytes}}. :format is derived
from the response content-type through ALLM.SpeechResponse.mime_to_format/1,
and :audio.mime_type is that format's canonical MIME type (or the raw
content type when it is an audio/* type outside the table). A 200 whose
content type is not audio/*, or whose body is empty, is
:malformed_response. :raw is always nil: the audio bytes live once,
in :audio. :model is the model that was sent.
opts[:request_id] is reflected onto response.request_id; when absent,
OpenAI's x-request-id header is used. request.metadata round-trips onto
response.metadata.
The x-request-id fallback is unreachable through ALLM.synthesize/3
The façade always supplies opts[:request_id] (it generates one when the
caller does not), so on that path response.request_id is the façade's
id and OpenAI's own correlation id is never observed. It surfaces only on
a direct synthesize/2 / decode_response/4 call that omits
opts[:request_id]. Identical to ALLM.Providers.OpenAI.Moderation.
Streaming
stream_synthesize/2 sends the same request with Finch.async_request/3
on the ALLM.Finch pool (HTTP/1) instead of Req, and returns a lazy
stream of ALLM.SpeechEvent values: :speech_started when the response
headers arrive, one :audio_delta per body chunk, then
:speech_completed. Usage is an all-nil %ALLM.Usage{}, as on
synthesize/2. stream_synthesize_input/3 is not implemented: OpenAI's
text-in streaming is a different API.
An HTTP error status is not classified from the status alone. The error
body is collected until the response ends and then classified exactly as
synthesize/2 classifies it, so the key redactor and the
string_too_long rule see the provider's message. A stream is never
retried once it has been returned.
Retry
Each attempt runs inside ALLM.Retry.run/3 under opts[:retry]
(default :default). The attempt marks :rate_limited (honouring
Retry-After), :provider_unavailable, :timeout and :network_error
as retryable, and the policy decides. The default policy's retry_on is
HTTP codes plus :timeout, so on its own this loop retries :timeout
only. Through ALLM.synthesize/3 the façade retries all four reasons,
and :timeout is retried by both loops: up to 9 attempts at the default
policy, against 3 for the other three reasons.
Error-struct hygiene
%ALLM.Error.SpeechAdapterError{} derives Jason.Encoder, so it is
often logged. No raw response body, request header or
Authorization value is copied into it, and there is no body preview.
Every provider-authored string it carries (:message,
metadata.openai_code, metadata.openai_type) passes a redactor that
replaces key-shaped tokens with [REDACTED]. OpenAI's real 401 masks the
key it echoes (sk-proj-*****…9900, observed 2026-09-24); the redactor is
defence in depth.
Not every error is JSON-encodable
The struct derives Jason.Encoder, but a :timeout, :network_error
or invalid-JSON :malformed_response error carries an exception struct
(Req.TransportError, Jason.DecodeError) on :cause, and
Jason.encode!/1 raises Protocol.UndefinedError on it. Drop or
Exception.message/1 the :cause before encoding. The image, embedding
and moderation adapter errors share this limitation.
Test-injection escape hatch
synthesize/2 honours opts[:adapter_opts][:speech_script]: when the key
is present, the call is handed to ALLM.Providers.FakeSpeech.synthesize/2
BEFORE any of this adapter's gates run. This is what lets the
ALLM.SpeechAdapter conformance suite drive a real adapter without an
HTTP stub. prepare_request/2 returns a stub error under the same key,
because a scripted response has no Req.Request analogue.
Summary
Functions
Return an unfired Req.Request configured exactly as synthesize/2 would
fire it, for callers who add headers, middleware or their own retry loop.
Stream speech for request.input from OpenAI as it is synthesized.
Synthesize speech from request.input against OpenAI.
Functions
@spec prepare_request( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, Req.Request.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}
Return an unfired Req.Request configured exactly as synthesize/2 would
fire it, for callers who add headers, middleware or their own retry loop.
The pre-flight gates run first, so this is defined only for an input that
passes them. Under opts[:adapter_opts][:speech_script] it returns a stub
error instead of delegating to ALLM.Providers.FakeSpeech.
Examples
iex> req = ALLM.SpeechRequest.new(input: "Hi.")
iex> {:ok, http} = ALLM.Providers.OpenAI.Speech.prepare_request(req, api_key: "sk-x")
iex> URI.to_string(http.url)
"https://api.openai.com/v1/audio/speech"
@spec stream_synthesize( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, Enumerable.t(ALLM.SpeechEvent.t())} | {:error, ALLM.Error.SpeechAdapterError.t()}
Stream speech for request.input from OpenAI as it is synthesized.
Returns {:ok, enumerable} of ALLM.SpeechEvent values, or
{:error, %ALLM.Error.SpeechAdapterError{}} from a pre-flight gate. The
four gates of synthesize/2 run first, then ALLM.Keys.fetch!/2 (which
raises %ALLM.Error.EngineError{reason: :missing_key} by design). No
HTTP request is made until the enumerable is reduced.
Injected defaults: the same model ("gpt-4o-mini-tts") and
voice ("alloy") as synthesize/2, and the same JSON body.
Options:
:stream_timeout— milliseconds of silence between two transport messages before the stream ends with{:error, %SpeechAdapterError{reason: :timeout}}. Default 60000. The transport's own receive timeout defaults above it (seeALLM.Providers.Support.Transport).:receive_timeout,:request_timeout,:pool_timeout— forwarded toFinch.async_request/3.:finch_name— the Finch pool (defaultALLM.Finch).:finch_module— the module called forasync_request/3andcancel_async_request/1(defaultFinch); tests passALLM.Test.FinchStub.
Each transport option is read from the top level of opts.
ALLM.stream_synthesize/3 hoists an engine's adapter_opts: transport
keys (ALLM.Adapter.transport_opts/0) there, as the chat façades do; a
direct call passes them at the top level.
Events: :speech_started (with sample_rate 24000 for
:pcm and :wav, else nil) once the response headers arrive, one
:audio_delta per non-empty body chunk, and :speech_completed with an
all-nil usage. A failure ends the stream with one {:error, _}:
- an HTTP error status: the error body is collected until the response
ends, then classified as
synthesize/2classifies it (a 401 is:authentication_failed, astring_too_long400 is:context_length_exceeded), with provider strings redacted; - a 200 whose
content-typeis notaudio/*::malformed_response; - a 200 with no audio bytes:
:invalid_requestwithmetadata.cause: :empty_input; - no transport message within
:stream_timeout::timeout; - a transport failure:
:network_error.
Halting the stream early (Enum.take/2) cancels the HTTP request and
removes the request's already-queued messages from the calling process's
mailbox. This is best-effort: a message the request process sends while
it is being shut down can still arrive afterwards.
opts[:adapter_opts][:speech_script] hands the call to
ALLM.Providers.FakeSpeech.stream_synthesize/2 before any gate runs.
Examples
iex> req = ALLM.SpeechRequest.new(input: "Hello.", format: :pcm)
iex> opts = [adapter_opts: [speech_script: [{:ok, "PCM-bytes"}]]]
iex> {:ok, events} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, opts)
iex> for {:audio_delta, bytes} <- events, into: "", do: bytes
"PCM-bytes"
iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :mp3, sample_rate: 24_000)
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
iex> err.reason
:unsupported_feature
iex> req = ALLM.SpeechRequest.new(input: "Hi.", format: :ulaw)
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.stream_synthesize(req, [])
iex> {err.reason, err.metadata.field}
{:unsupported_feature, :format}
@spec synthesize( ALLM.SpeechRequest.t(), keyword() ) :: {:ok, ALLM.SpeechResponse.t()} | {:error, ALLM.Error.SpeechAdapterError.t()}
Synthesize speech from request.input against OpenAI.
Returns {:ok, %ALLM.SpeechResponse{}} or
{:error, %ALLM.Error.SpeechAdapterError{}}. The one exception is
ALLM.Keys.fetch!/2, which raises %ALLM.Error.EngineError{reason: :missing_key}
by design; all four pre-flight gates run ahead of it.
Sample rate: response.sample_rate is 24000 for :pcm
and :wav and nil for every other format. A request sample_rate other
than nil (or 24000 for :pcm / :wav) is refused with
:unsupported_feature before any I/O.
Formats: :ulaw and :alaw are refused with :unsupported_feature
and metadata.field: :format before any I/O; OpenAI has no telephony
output.
Injected defaults: model is "gpt-4o-mini-tts" and voice is
"alloy" when the request leaves them nil, and the receive
timeout is 60 s when opts[:request_timeout]
is absent.
Retry: this adapter's own ALLM.Retry.run/3 loop retries :timeout
under the default policy. :rate_limited, :provider_unavailable and
:network_error are retryable too, but only under a caller-supplied
opts[:retry] that lists them (as ALLM.synthesize/3's own loop does).
See the module documentation for the gates, the wire-field map and the
adapter_opts[:speech_script] test-injection short-circuit.
Examples
iex> req = ALLM.SpeechRequest.new(input: "Hello.")
iex> opts = [adapter_opts: [speech_script: [{:ok, "ID3-bytes"}]]]
iex> {:ok, resp} = ALLM.Providers.OpenAI.Speech.synthesize(req, opts)
iex> ALLM.Audio.to_binary(resp.audio)
{:ok, "ID3-bytes"}
iex> req = ALLM.SpeechRequest.new(input: String.duplicate("a", 4097))
iex> {:error, err} = ALLM.Providers.OpenAI.Speech.synthesize(req, [])
iex> {err.reason, err.metadata}
{:context_length_exceeded, %{count: 4097, max: 4096}}