Synaptic Voice Guide

View Source

This guide documents Synaptic's voice subsystem and principles for building real-time voice applications: how to enable voice, configure it, and use the framework to get the desired behavior. Synaptic does not ship a UI or client; your application owns the transport and UX. The following sections outline how to structure your app so that workflows stay in Synaptic while voice I/O behaves the way you want.

Mode-specific setup playbooks

For implementation-grade, step-by-step setup instructions by mode:

Frontend setup playbooks

For frontend-only integration details (event wiring, UI states, recorder/playback controls):

For reusable personas, concrete tools, typed session context, and deterministic authorization, see docs/voice-profiles.md.

What the framework provides

Synaptic.Voice is the only public voice API.

  • Headless voice: mode: :turn_based or mode: :duplex. Synaptic owns the session process; you push audio or text, call end_turn, and consume normalized events.
  • Realtime voice: mode: :realtime with provider-specific transport semantics:
    • OpenAI: client-direct WebRTC + server sideband orchestration
    • Gemini: server-relay WebSocket (client talks to your app, app talks to Gemini Live)

Provider selection is per session with provider: :openai | :gemini | :eleven_labs for headless voice and provider: :openai | :gemini for realtime. Public mixed stacks are rejected.

Choose :realtime when you want low-latency conversational transport. Choose :turn_based or :duplex when you want explicit turn control and direct STT/TTS orchestration.

Enabling and configuring voice

Dependencies and application

Add synaptic to your project. Synaptic’s application starts its own PubSub, Finch, and voice/realtime supervisors; you do not need to start these yourself.

Configuration

Compile-time (e.g. config/config.exs):

# Tools (for workflow LLM calls)
config :synaptic, Synaptic.Tools.OpenAI,
  model: "gpt-4o-mini"

config :synaptic, Synaptic.Voice,
  default_mode: :duplex,
  default_provider: :openai,
  audio_format_default: %{encoding: :pcm16le, sample_rate_hz: 24_000, channels: 1}

config :synaptic, Synaptic.Voice.Providers.OpenAI,
  default_experience: :legacy,
  stt_model: "gpt-4o-mini-transcribe",
  tts_model: "gpt-4o-mini-tts",
  tts_audio_format: "pcm16",
  realtime_model: "gpt-4o-realtime-preview",
  realtime_response_mode: :orchestrated,
  voice: "alloy",
  realtime_2_1_model: "gpt-realtime-2.1",
  realtime_2_1_transcription_model: "gpt-realtime-whisper",
  realtime_2_1_reasoning_effort: "low",
  realtime_2_1_response_mode: :native,
  realtime_2_1_turn_detection: "semantic_vad",
  realtime_2_1_turn_eagerness: "low",
  realtime_2_1_noise_reduction: "near_field",
  realtime_2_1_voice: "marin"

config :synaptic, Synaptic.Voice.Providers.Gemini,
  stt_model: "gemini-2.5-flash",
  tts_model: "gemini-2.5-flash-preview-tts",
  live_model: "gemini-2.5-flash-native-audio-preview",
  voice: "Kore",
  live_voice: "Kore"

config :synaptic, Synaptic.Voice.Providers.ElevenLabs,
  tts_model_id: "eleven_multilingual_v2",
  stt_model_id: "scribe_v2",
  tts_output_format: "pcm_24000",
  include_timestamps: true

Runtime / secrets (e.g. config/runtime.exs):

if api_key = System.get_env("OPENAI_API_KEY") do
  config :synaptic, Synaptic.Tools.OpenAI, api_key: api_key
  config :synaptic, Synaptic.Voice.Providers.OpenAI,
    api_key: api_key,
    realtime_2_1_model: System.get_env("OPENAI_REALTIME_2_1_MODEL", "gpt-realtime-2.1")
end

if gemini_api_key = System.get_env("GEMINI_API_KEY") do
  config :synaptic, Synaptic.Voice.Providers.Gemini, api_key: gemini_api_key
end

if elevenlabs_api_key = System.get_env("ELEVENLABS_API_KEY") do
  config :synaptic, Synaptic.Voice.Providers.ElevenLabs, api_key: elevenlabs_api_key
end

Principles for realtime voice

Realtime has two provider patterns:

  • OpenAI realtime: client talks directly to provider over WebRTC; server orchestrates through sideband.
  • Gemini realtime: server holds the Live WebSocket; client audio/text is relayed through your app.

Create sessions with:

Synaptic.Voice.start_session(MyWorkflow, %{},
  provider: :openai,
  mode: :realtime,
  experience: :realtime_2_1,
  profile: MyApp.Voice.AssistantProfile
)

Synaptic.Voice.start_session(MyWorkflow, %{}, provider: :gemini, mode: :realtime)

Use transport from the return payload to bootstrap client behavior:

  • OpenAI transport includes provider bootstrap fields like client_secret, expires_at, model, voice.
  • Gemini transport describes relay expectations (audio_config) and does not expose provider credentials to clients.

New OpenAI applications should use experience: :realtime_2_1. It selects gpt-realtime-2.1, Marin, native responses, semantic VAD at low eagerness, near-field noise reduction, gpt-realtime-whisper, and low reasoning. Override the model globally with OPENAI_REALTIME_2_1_MODEL, or per session:

Synaptic.Voice.start_session(MyWorkflow, %{},
  provider: :openai,
  mode: :realtime,
  experience: :realtime_2_1,
  provider_opts: [
    realtime: [model: "gpt-realtime-2.1-mini", reasoning_effort: "low"]
  ]
)

Supported reasoning-effort experiments are minimal, low, medium, high, and xhigh. If omitted, OpenAI's model default is used.

In the Realtime 2.1 experience, native response mode is the default. Realtime owns normal conversation and calls only the concrete capabilities compiled from the session's voice profile. Configure gpt-5.6-luna explicitly for workflows that should use Luna for routing, reasoning, or additional tool calls. Results are sent back to the same Realtime session so the audio model can phrase and speak them naturally. Set response_mode: :orchestrated (or per-session configuration) to keep the deterministic transcript -> workflow -> spoken-readout path for A/B comparisons.

Compatibility behavior

Existing OpenAI realtime calls that omit both experience: and profile: use :legacy. This preserves orchestrated workflow-on-every-turn behavior, the configured legacy model and voice, server_vad, and the original ephemeral session contract. Explicit experience: wins; otherwise a profile implies :realtime_2_1. Applications may set default_experience: :realtime_2_1 only after reviewing all unversioned callers.

SessionBootstrap.create_ephemeral_session/1 retains its original endpoint, flat request schema, and returned session shape. The new create_client_secret/1 function uses the GA client-secret endpoint and nested session schema. create_browser_bootstrap/1 chooses the correct path from the resolved experience and exposes a stable transport map to application code.

Realtime operation matrix

  • OpenAI realtime supports: client_connected/2, client_disconnected/2, ingest_provider_event/2.
  • Gemini realtime supports: push_audio/3, push_text/3, end_turn/2, cancel_output/1.
  • Unsupported operations return {:error, :unsupported_for_mode}.

Workflow design for realtime

In native mode, treat the workflow as an on-demand tool: accept the delegated query, use the application-configured reasoning model and tools, and return factual results for Realtime 2.1 to express naturally.

The following every-turn workflow rules apply to orchestrated mode:

  • The realtime sideband resumes the workflow with the final user transcript. The default mapping sends %{human_input_text: transcript} (or %{answer: transcript} if the suspended step’s resume_schema has :answer).
  • Design your workflow so that the first step that needs user speech has suspend: true and resume_schema: %{human_input_text: :string} (or answer: :string). When the step runs after resume, read from context[:human_input][:human_input_text] or context[:human_input][:answer] (or fallback to context.human_input_text / context.answer) and put the result into context (e.g. query).
  • After tools and LLM, produce a step result that includes the spoken reply (e.g. assistant_answer). The framework will send the appropriate provider instructions so the provider speaks that content.
  • To support multiple turns, add a later step that again suspends with resume_schema: %{human_input_text: :string}; on resume, route back to your router or first logic step with the new query so the same pipeline runs again.

Realtime API summary (server)

FunctionPurpose

| Synaptic.Voice.start_session(workflow_module, input, provider: :openai | :gemini, mode: :realtime, ...) | Start a run and realtime session; return payload for the client. | | Synaptic.Voice.subscribe_session(session_id) | Subscribe to session events ({:synaptic_voice_event, envelope}). | | Synaptic.Voice.unsubscribe_session(session_id) | Unsubscribe. | | Synaptic.Voice.client_connected(session_id) | OpenAI realtime only. | | Synaptic.Voice.client_disconnected(session_id) | OpenAI realtime only. | | Synaptic.Voice.ingest_provider_event(session_id, payload) | OpenAI realtime only. | | Synaptic.Voice.approve_capability(session_id, capability_name) | Grant the next invocation of a pending consequential capability; OpenAI realtime only. | | Synaptic.Voice.push_audio(session_id, chunk) | Headless and Gemini realtime. | | Synaptic.Voice.push_text(session_id, text) | Headless and Gemini realtime. | | Synaptic.Voice.stop_session(session_id, reason) | Stop the session. |

Realtime session events (PubSub)

Subscribe with Synaptic.Voice.subscribe_session(session_id). All voice modes publish on topic "synaptic:voice:session:" <> session_id as:

{:synaptic_voice_event, envelope}

Envelope fields include: :event, :data, :session_id, :run_id, :seq, :ts_ms.

Useful events:

  • :session_started, :session_stopped, :session_error
  • :input_partial_text, :input_final_text
  • :assistant_response_started, :assistant_text_chunk, :assistant_response_done
  • :duplex_state_changed
  • :provider_outbound — the payload to send to the provider on the data channel (e.g. response.create); push this to your client so it can forward it.

Use these to drive UI (status, transcripts) and to relay :provider_outbound to the client.

Principles for headless voice

When you own the transport (e.g. your own WebSocket) and want Synaptic to run STT/TTS and workflows:

  • Start or attach: Synaptic.Voice.start_session(workflow_module, input, opts) starts a run and session; attach_run(run_id, opts) attaches a session to an existing run. Both return {:ok, %{session_id, run_id, mode, stack, transport}}.

  • Provider bundle: pass provider: :openai | :gemini | :eleven_labs. Router derives pure stacks internally; public stack is rejected unless _allow_custom_stack: true (internal/tests).

  • Input: Stream audio with push_audio(session_id, chunk, opts) and/or send text with push_text(session_id, text, opts). When the user turn is complete, call end_turn(session_id, opts) so STT finalizes (if applicable) and the run resumes with the transcript.
  • Output: Subscribe with Synaptic.Voice.subscribe_session(session_id); events arrive on topic "synaptic:voice:session:" <> session_id as {:synaptic_voice_event, envelope}. Use :assistant_text_chunk, :assistant_audio_chunk, etc., to drive your playback and UI. Built-in headless providers now default to one TTS generation per assistant turn for more consistent tone; segmented TTS remains the fallback for adapters without capability metadata.
  • Interruption: In duplex mode, call cancel_output(session_id) when the user interrupts; the framework emits :duplex_interruption and transitions back to listening.
  • Resume mapping: By default, the transcript is sent as %{human_input_text: transcript} or %{answer: transcript} depending on the step’s resume_schema. Override with resume_mapper: fn transcript, state -> payload end in session options.
  • Unsupported operations: functions that do not make sense for provider+mode return {:error, :unsupported_for_mode}.
  • Cleanup: Call Synaptic.Voice.stop_session(session_id, reason) and unsubscribe when the session ends.

Optional turn admission

Headless sessions can defer workflow resume until an application policy decides whether a final transcript is complete. Pass turn_admission as a one- or two-argument function, or as a module implementing Synaptic.Voice.TurnAdmission:

turn_admission = fn input, _opts ->
  if incomplete?(input.accumulated_transcript) do
    {:keep_listening, %{reason: :incomplete}}
  else
    {:commit, %{reason: :complete}}
  end
end

{:ok, session} =
  Synaptic.Voice.attach_run(run_id,
    provider: :eleven_labs,
    mode: :duplex,
    turn_admission: turn_admission,
    turn_admission_opts: [policy: :my_app]
  )

The policy input contains :prompt, :latest_transcript, :accumulated_transcript, :transcript_count, and the original :end_turn_opts. Policies run outside the session GenServer, so model- or network-backed decisions do not block voice control messages. Policy failures emit a :session_error and fall back to :commit so accepted speech is not silently lost.

Providers declare whether final transcripts are segments or cumulative hypotheses. Override this only for custom adapters with stt_final_mode: :segment | :cumulative | :replace.

Headless: modules and event envelope

  • Modules: Synaptic.Voice (API), Synaptic.Voice.Router, Synaptic.Voice.SessionRegistry, Synaptic.Voice.HeadlessSessionSupervisor, Synaptic.Voice.RealtimeSessionSupervisor, Synaptic.Voice.Event, Synaptic.Voice.TextSegmenter; engines: Synaptic.Voice.Sessions.Headless, Synaptic.Voice.Sessions.Realtime.OpenAI, Synaptic.Voice.Sessions.Realtime.Gemini; providers: Synaptic.Voice.Providers.OpenAI.*, Synaptic.Voice.Providers.Gemini.*, Synaptic.Voice.Providers.ElevenLabs.*.
  • Headless output strategies: headless sessions select an internal TTS strategy from provider capability metadata. Current strategies are :segmented_batch, :single_shot, and reserved :streaming.
  • Envelope: guaranteed fields :v, :session_id, :run_id, :seq, :ts_ms, :event, :data.
  • Event names: :turn_started, :input_partial_text, :input_final_text, :turn_admission_started, :turn_admission_decided, :assistant_text_chunk, :assistant_text_done, :assistant_audio_chunk, :assistant_audio_done, :duplex_interruption, :duplex_state_changed, :session_error, :session_stopped.

Configuration reference

Defaults are:

  • default_mode: :duplex
  • default_provider: :openai
  • audio_format_default: %{encoding: :pcm16le, sample_rate_hz: 24_000, channels: 1}

Telemetry

  • Headless: [:synaptic, :voice, :session, :start|:stop], [:synaptic, :voice, :stt, :partial|:final|:error], [:synaptic, :voice, :tts, :chunk|:done|:error], [:synaptic, :voice, :duplex, :interruption], [:synaptic, :voice, :latency] (e.g. stt_first_partial_ms, llm_first_chunk_ms, tts_first_chunk_ms, turn_total_ms).
  • Realtime: [:synaptic, :voice, :realtime, :session, :start] and related.

Testing

  • Voice tests: test/synaptic/voice/event_test.exs, text_segmenter_test.exs, session_test.exs, openai_test.exs.
  • Synthetic latency: mix run scripts/voice_latency_harness.exs.

Limitations

  • No built-in UI or client transport; your app provides both.
  • Headless OpenAI STT batches audio and transcribes on end_turn; partials can be simulated via push_audio(..., partial_text: ...).
  • TTS playback is your responsibility. Depending on provider capabilities, headless voice may synthesize per segment or once per assistant turn.
  • Realtime behavior is provider-specific: OpenAI uses client-direct WebRTC + sideband; Gemini uses server-relay Live WebSocket.

Extending to another provider

Implement the headless behaviours:

Realtime is not provider-neutral: OpenAI and Gemini each use dedicated engines, and provider: :eleven_labs is headless-only.