Vox RTC Server for Elixir

Copy Markdown View Source

vox_rtc_server lets a trusted Elixir service create and control Vox-hosted WebRTC conversations over Vox's native gRPC API.

Vox remains responsible for WebRTC media, VAD, end-of-utterance detection, turn state, interruption handling, STT, TTS, and audio playout. The SDK owns only the gRPC connection and one ordered bidirectional control stream per RTC session.

This package does not use PondSocket and does not implement application-user authentication. Your application keeps its existing Plug, Phoenix, session, cookie, or JWT boundary. The api_key option is only the trusted credential used by the application server when calling Vox.

Installation

Add the package to mix.exs:

def deps do
  [
    {:vox_rtc_server, "~> 0.2.5"}
  ]
end

Supervision

Start one client for each Vox deployment:

children = [
  {VoxRtcServer.Client,
   name: MyApp.Vox,
   target: "dns://vox-service.vox.svc.cluster.local:9090",
   api_key: System.fetch_env!("VOX_API_KEY")}
]

Supervisor.start_link(children, strategy: :one_for_one)

Browser WebSocket gateway

Mount the provided Plug at the same-origin path used by @eleven-am/vox-rtc-client:

plug VoxRtcServer.Gateway,
  client: MyApp.Vox,
  path: "/api/vox/rtc",
  on_session_created: fn %{request: request, session: session} ->
    MyApp.Calls.register(request, session)
    :ok
  end,
  on_session_closed: fn %{session: session, reason: reason} ->
    MyApp.Calls.unregister(session, reason)
    :ok
  end

The Plug uses WebSockAdapter, so it works with Plug-compatible servers that support WebSocket upgrades, including Bandit and Cowboy. Each upgraded browser socket creates and owns one controlled gRPC session. The socket carries SDP, full-trickle ICE, and lifecycle events for the whole call; WebRTC media flows directly between the browser and Vox.

The gateway preserves offer and candidate generations, null candidate completion, and stale generations. Legacy generation-less negotiation remains compatible until a generated offer is received. It exposes request metadata to the lifecycle hooks but never returns the Vox credential or gRPC target to the browser.

The target is the gRPC endpoint, not Vox's HTTP port. Omit api_key only when the Vox deployment does not require API authentication. Supply gRPC connection options through connect_options, including a GRPC.Credential for TLS. The SDK uses gRPC's Mint adapter by default; pass an explicit :adapter only when your application deliberately uses another supported transport.

Controlled RTC session

Create a session from trusted server code:

alias VoxRtcServer.{Client, Session, SessionConfig}

{:ok, bootstrap, session} =
  Client.create_controlled_session(MyApp.Vox)

:ok =
  Session.configure(session, %SessionConfig{
    stt_model: "parakeet-stt:tdt-0.6b-v3",
    tts_model: "kokoro-tts:v1.0",
    voice: "af_heart",
    turn_profile: "browser_default",
    speech_context: true
  })

bootstrap contains the session id, expiry, attach TTL, and public ICE server configuration. Return only the fields your browser signaling route needs. Do not return the Vox API key or gRPC endpoint to the browser.

The caller is subscribed automatically. Incoming events arrive as ordinary Elixir messages:

receive do
  {:vox_rtc, ^session, %VoxRtcServer.Event{type: :transcript_completed} = event} ->
    transcript = event.payload.transcript
    speech_context = event.payload.speech_context
end

Speech context is opt-in and final-only. The value is Vox's versioned prosody and audio-event result decoded from google.protobuf.Struct into VoxRtcServer.SpeechContext.

Schema v2 exposes timestamped emotions and vocal speaker spans plus environmental sounds. Sound spans also contain a model score from 0 to 1:

case event.payload.speech_context do
  %VoxRtcServer.SpeechContext{} = context ->
    Enum.each(context.emotions || [], fn span ->
      IO.inspect({:emotion, span.label, span.start_ms, span.end_ms})
    end)

    Enum.each(context.sounds || [], fn sound ->
      IO.inspect({:sound, sound.label, sound.score})
    end)

  nil ->
    :ok
end

A :partial result identifies the unavailable :speaker or :sounds track; a :failed result identifies both. Unsupported or malformed context is decoded as nil without dropping the transcript event.

An application can explicitly manage subscriptions:

:ok = Session.subscribe(session, self())
:ok = Session.unsubscribe(session, self())

Full-trickle signaling

Forward the browser's offer and ICE candidates as they arrive. Candidate completion is an explicit :complete message; the SDK does not implement half-trickle or wait for ICE gathering to finish.

alias VoxRtcServer.{IceCandidate, Session, SessionDescription}

:ok =
  Session.send_offer(
    session,
    %SessionDescription{type: "offer", sdp: browser_offer_sdp},
    false,
    1
  )

:ok =
  Session.send_ice_candidate(
    session,
    %IceCandidate{
      candidate: candidate,
      sdp_mid: sdp_mid,
      sdp_m_line_index: sdp_m_line_index,
      username_fragment: username_fragment
    },
    1
  )

:ok = Session.send_ice_candidate(session, :complete, 1)

The optional generation is preserved in both directions. Omit it only when talking to a legacy, generation-less negotiation.

Vox answers through :answer, :ice_candidate, and :ice_candidates_complete events. Your application forwards those signaling messages to its browser connection. Media then flows directly between the browser and Vox; it does not pass through the Elixir application.

Streaming a response

Text can be appended incrementally as an LLM generates it:

alias VoxRtcServer.{ResponseOptions, ResponseOutputOptions, Session}

options = %ResponseOptions{
  allow_interruptions: true,
  output: %ResponseOutputOptions{
    model: "qwen3-tts:0.6b-clone",
    voice: "samantha",
    language: "fr",
    speed: 0.9,
    params: %{"temperature" => 0.7}
  }
}

case Session.start_response_and_wait(session, options, 5_000) do
  {:ok, ack} ->
    Logger.info("effective output", output: inspect(ack.output))
    response = %ResponseOptions{
      allow_interruptions: true,
      generation_id: ack.generation_id
    }

    :ok = Session.append_response_text(session, "The first generated phrase", response)
    :ok = Session.append_response_text(session, " and the next phrase.", response)
    :ok = Session.commit_response(session, response)

  {:error, %VoxRtcServer.ErrorEvent{} = error} ->
    Logger.warning("Vox rejected response start", code: error.code)
end

start_response_and_wait/3 generates a generation_id when one is not provided, sends response.start, and waits for the matching response.created acknowledgement. A correlated rejection, acknowledgement timeout, or control-stream failure returns {:error, %VoxRtcServer.ErrorEvent{}}. An event from a stale generation cannot resolve the wait.

The session automatically carries the active generation through subsequent delta, commit, and cancel commands, so passing the returned id explicitly is optional. Passing it explicitly is useful when application work is concurrent because Vox can reject stale work without affecting the current response. The response-scoped output is optional. Vox fills omitted fields from the session configuration and returns the immutable effective selection in the acknowledgement.

Session.start_response/2 remains available for fire-and-forget control. Use cancel_response/2 to cancel an active response and replace_response_text/3 when the complete text must replace the current buffer.

Typed errors

Signaling failures arrive as :signaling_error; conversation failures arrive as :error. Both carry %VoxRtcServer.ErrorEvent{} payloads containing message, stable code, recoverable, and an optional generation_id. Known server error codes are available through VoxRtcServer.ErrorEvent.known_codes/0 and known_code?/1.

Only a non-recoverable error or closed control stream should end the call. Recoverable errors belong to the command or response generation named by the event and do not invalidate the RTC session.

Lifecycle

The session process:

  • sends attach as the first control-stream message;
  • serializes all signaling and response writes on that stream;
  • monitors its owning client and subscribers;
  • cancels failed streams;
  • closes its stream when the owner or caller closes;
  • never restarts a completed session.

Call Session.close/2 when an application call ends and Client.close/1 during an intentional client shutdown. Supervision shutdown also closes owned sessions and the gRPC channel.

Contract source

Generated protobuf and gRPC modules live under lib/generated. They are generated from priv/proto/vox.proto, copied from Vox's canonical proto/vox.proto. The public SDK wraps those generated modules so application code does not have to construct control-stream envelopes directly.