ALLM.Providers.Gemini.Transcription (allm v0.6.1)

Google Gemini speech-to-text adapter. Implements ALLM.TranscriptionAdapter against generateContent.

Layer B — runtime. Wire it with ALLM.Engine.new(transcription_adapter: ALLM.Providers.Gemini.Transcription) and call it through ALLM.transcribe/3. The key resolves via ALLM.Keys.fetch!(:gemini, opts) after the pre-flight gates, so no key ever lives on the engine.

req = ALLM.TranscriptionRequest.new(audio: ALLM.Audio.from_file("clip.mp3"))
{:ok, resp} = ALLM.Providers.Gemini.Transcription.transcribe(req, api_key: "AIza...")
resp.text

Transcription is prompted chat

Gemini has no transcription endpoint. This adapter sends the audio inline to a chat model together with a fixed instruction:

Generate a verbatim transcript of this audio. Output only the transcript.

The transcript is therefore a language model's answer, not the output of a dedicated speech model. It is usually verbatim, but it can paraphrase, tidy disfluencies, or refuse, where a Whisper-family model cannot. When exact wording matters, prefer ALLM.Providers.OpenAI.Transcription.

Silence is not reliably an empty transcript

Given eight minutes of digital silence, the model returned a fluent, invented transcript rather than "" (observed 2026-09-24 on gemini-flash-latest, which then resolved to gemini-3.8-flash). Do not treat non-empty :text as proof that the clip contained speech.

:language is added to the instruction as a one-sentence hint. :prompt is appended as a context block (names, spellings, subject matter); it is context, not a replacement for the instruction.

Wire-field map

Concern	Gemini
Endpoint	`POST {base}/models/<model>:generateContent`
Base URL	`https://generativelanguage.googleapis.com/v1beta`, overridable via `adapter_opts[:endpoint]`
Auth	`x-goog-api-key` header
Body	`{"contents": [{"role": "user", "parts": [{"text": <instruction + hints>}, {"inlineData": {"mimeType", "data"}}]}]}`
`model`	`:model`, or `gemini-flash-latest` when `nil`; a `models/` prefix is accepted
Options	`ALLM.TranscriptionRequest.options` becomes `generationConfig` (e.g. `%{"temperature" => 0.0}`)
Text	the `text` parts of the first candidate, concatenated; parts marked `"thought": true` are dropped
Usage	`usageMetadata`: `promptTokenCount` → `:input_tokens`, `candidatesTokenCount` → `:output_tokens`, `thoughtsTokenCount` → `:reasoning_tokens`, `totalTokenCount` → `:total_tokens`
Id	`responseId` → `:id`
Correlation	none read; neither `x-request-id` nor `x-goog-request-id` came back on the recorded responses
Size limit	20 MB per request (documented), base64 included; see `max_audio_bytes/0`
Unknown fields	rejected (400 `Unknown name`), so a mistyped `:options` key surfaces as `:invalid_request`

Accepted MIME types: audio/wav, audio/mpeg, audio/aiff, audio/aac, audio/ogg, audio/opus, audio/flac.

Adapter-injected defaults

model defaults to "gemini-flash-latest" when :model is nil.
The HTTP receive timeout defaults to 120 s when opts[:request_timeout] is absent.

Pre-flight gates

In this order, before any HTTP I/O and before ALLM.Keys.fetch!/2:

Resolvable. Audio whose bytes cannot be resolved (a missing file, a directory, invalid base64, an off-shape source, or an :audio that is not an %ALLM.Audio{}) → :invalid_request with metadata.cause and metadata.field: :audio.
Size. More than max_audio_bytes/0 bytes → :invalid_request with metadata.count and metadata.max. A {:file, path} source is measured with a stat, never read.
MIME. A :mime_type that is nil or not an accepted type → :invalid_request with metadata.mime_type. Parameters and case are ignored ("audio/ogg; codecs=vorbis" is sent as audio/ogg).
Span flags. timestamps: true or logprobs: true → :unsupported_feature with metadata.field naming the flag (:timestamps first when both are set).

Word timings and log-probabilities are refused

This adapter honours neither span flag, so response.spans is always nil. timestamps: true is refused because a generateContent answer carries no timings. logprobs: true is refused for every Gemini model on the basis of one live probe on 2026-09-28: gemini-flash-latest answered generationConfig.responseLogprobs: true with 400 "Logprobs is not enabled for this model". That is a statement about the one model probed, not about Gemini as a provider; other models were not tried. Use ALLM.Providers.OpenAI.Transcription for token log-probabilities, or ALLM.Providers.ElevenLabs.Transcription for word timings and log-probabilities.

Response

:text is the transcript ("" when the model returns no text). :raw is the provider body. :model is the model that was sent. :language and :duration_seconds are always nil: Gemini reports neither. opts[:request_id] is reflected onto response.request_id, and request.metadata round-trips onto response.metadata.

Truncation is not an error. When the model stops at its output-token limit (finishReason: "MAX_TOKENS"), the partial transcript is returned as {:ok, response} with response.metadata.finish_reason == :length. Check for it on long clips.

Safety blocks are errors. A candidate stopped for SAFETY, RECITATION or another content-policy reason, or a prompt blocked by promptFeedback.blockReason, returns :content_filter. Transcribing well-known recited material (song lyrics, famous speeches) can trip RECITATION.

There is no provider-side request id

Neither x-request-id nor x-goog-request-id came back on the recorded responses, and this adapter reads no response header for a request id, so response.request_id is opts[:request_id] or nil. Through ALLM.transcribe/3 it is always the façade's id (generated when the caller does not pass one). The provider's responseId is on response.id.

No retries

This adapter makes one HTTP attempt per call and returns every classified error, retryable or not, as {:error, _}. Each attempt re-uploads the whole clip, so the only retry loop is ALLM.transcribe/3's (3 attempts at the default policy). A direct transcribe/2 caller who wants retries wraps the call.

Errors

Wire	Reason
400 with `details[].reason == "API_KEY_INVALID"`	`:authentication_failed`
400 containing `exceeds the maximum number of tokens` or `input token count`	`:context_length_exceeded`
other 400, 404	`:invalid_request`
401, 403	`:authentication_failed`
429	`:rate_limited` (`retry_after_ms` from `Retry-After`)
500, 502, 503, 504	`:provider_unavailable`

Gemini answers a bad key with a 400, not a 401. This adapter reads the API_KEY_INVALID marker so a bad key is :authentication_failed.

Error-struct hygiene

No raw response body, request header or API-key value is copied into %ALLM.Error.TranscriptionAdapterError{}, and there is no body preview. The provider's message and status pass a redactor that replaces Google-shaped credentials (AIza… keys and ya29.… OAuth tokens) with [REDACTED].

Not every error is JSON-encodable

The struct derives Jason.Encoder, but a :timeout, :network_error or invalid-JSON :malformed_response error carries an exception struct (Req.TransportError, Jason.DecodeError) on :cause, and Jason.encode!/1 raises Protocol.UndefinedError on it. Drop or Exception.message/1 the :cause before encoding. The image, embedding and moderation adapter errors share this limitation.

Test-injection escape hatch

transcribe/2 honours opts[:adapter_opts][:transcription_script]: when the key is present, the call is handed to ALLM.Providers.FakeTranscription.transcribe/2 BEFORE any of this adapter's gates run, with adapter_opts[:max_audio_bytes] set to this adapter's own max_audio_bytes/0 so a real clip is not rejected by the Fake's small default cap, and adapter_opts[:span_flags] set to [] so the Fake refuses both span flags exactly as this adapter does. prepare_request/2 returns a stub error under the same key.

Summary

Functions

max_audio_bytes()

Return the largest audio clip, in bytes, this adapter will send.

prepare_request(request, opts)

Return an unfired Req.Request configured exactly as transcribe/2 would fire it.

transcribe(request, opts)

Transcribe request.audio with a Gemini model.

Functions

max_audio_bytes()

@spec max_audio_bytes() :: pos_integer()

Return the largest audio clip, in bytes, this adapter will send.

Google documents a 20 MB limit on an inline request, and the clip travels base64-encoded, which adds a third. This cap is the largest raw size whose encoding fits 20 MiB with 64 KiB to spare for the instruction and JSON framing. It is conservative: a live probe on 2026-09-24 saw a clip at this cap accepted, and also a 15,831,040-byte clip (about 21.1 MB once encoded). Larger clips need Gemini's Files API, which this adapter does not use.

Examples

iex> ALLM.Providers.Gemini.Transcription.max_audio_bytes()
15_679_488

prepare_request(request, opts)

@spec prepare_request(
  ALLM.TranscriptionRequest.t(),
  keyword()
) :: {:ok, Req.Request.t()} | {:error, ALLM.Error.TranscriptionAdapterError.t()}

Return an unfired Req.Request configured exactly as transcribe/2 would fire it.

The pre-flight gates run first, so this is defined only for audio that passes them. Under opts[:adapter_opts][:transcription_script] it returns a stub error instead of delegating.

Examples

iex> req = ALLM.TranscriptionRequest.new(audio: ALLM.Audio.from_binary("ID3", "audio/mpeg"))
iex> {:ok, http} = ALLM.Providers.Gemini.Transcription.prepare_request(req, api_key: "AIza-x")
iex> URI.to_string(http.url)
"https://generativelanguage.googleapis.com/v1beta/models/gemini-flash-latest:generateContent"

transcribe(request, opts)

@spec transcribe(
  ALLM.TranscriptionRequest.t(),
  keyword()
) ::
  {:ok, ALLM.TranscriptionResponse.t()}
  | {:error, ALLM.Error.TranscriptionAdapterError.t()}

Transcribe request.audio with a Gemini model.

Returns {:ok, %ALLM.TranscriptionResponse{}} or {:error, %ALLM.Error.TranscriptionAdapterError{}}. The one exception is ALLM.Keys.fetch!/2, which raises %ALLM.Error.EngineError{reason: :missing_key} by design; every pre-flight gate runs ahead of it.

Injected defaults: model is "gemini-flash-latest" when the request leaves it nil, and the receive timeout is 120 s when opts[:request_timeout] is absent.

No retries: one HTTP attempt per call, whatever the error. Wrap the call yourself if you want retries, or go through ALLM.transcribe/3, which retries up to 3 times at the default policy.

The transcript comes from a chat model. See the module documentation for what that means for fidelity, truncation (metadata.finish_reason) and :content_filter.

Examples

iex> audio = ALLM.Audio.from_binary("ID3", "audio/mpeg")
iex> req = ALLM.TranscriptionRequest.new(audio: audio)
iex> opts = [adapter_opts: [transcription_script: [{:ok, "hello"}]]]
iex> {:ok, resp} = ALLM.Providers.Gemini.Transcription.transcribe(req, opts)
iex> resp.text
"hello"

iex> audio = ALLM.Audio.from_binary("webm", "audio/webm")
iex> req = ALLM.TranscriptionRequest.new(audio: audio)
iex> {:error, err} = ALLM.Providers.Gemini.Transcription.transcribe(req, [])
iex> {err.reason, err.metadata.mime_type}
{:invalid_request, "audio/webm"}

Both span flags are refused before any key is needed (see the module documentation):

iex> audio = ALLM.Audio.from_binary("ID3", "audio/mpeg")
iex> req = ALLM.TranscriptionRequest.new(audio: audio, logprobs: true)
iex> {:error, err} = ALLM.Providers.Gemini.Transcription.transcribe(req, [])
iex> {err.reason, err.metadata.field}
{:unsupported_feature, :logprobs}