An SSH tunnel from this node's loopback to one sandbox VM's loopback agent port, with a private key that never touches disk.
Everything here was established empirically against OTP 29's :ssh and
against a real OpenSSH 10.3p1 sshd — which is what a GCE VM runs — rather
than from the API docs. The load-bearing results:
:loopbackis mandatory.tcpip_tunnel_to_server/6withListenHost = :any(or{0,0,0,0}) binds the local listener on every interface, publishing the sandbox agent to the LAN, and still returns a perfectly normal{:ok, port}. There is no error to catch, which is why it is a constant here and a test there.{:ok, port}is not evidence of reachability. The call only does a localgen_tcp:listen; thedirect-tcpipchannel is opened lazily, per accepted connection. A tunnel to a closed remote port succeeds identically.GET /v1/healthis the only readiness proof, which is what makesCrowdControl.Provider.acquire/1's contract mandatory rather than stylistic.- The
Timeoutargument is ignored by the handler, so this module never relies on it. falsefromis_host_key/5does not reject a host key whensilently_accept_hosts: true— OTP overrides it. Only{:error, _}is honoured on both settings. Both are set the safe way here.save_accepted_hostdefaults totrue, so it is switched off explicitly: no code path may write aknown_hostsfile.:user_diris validated to exist even when a customkey_cbnever reads it, so an empty directory has to exist somewhere.- A
key_cbfailure reason never reaches the caller.:ssh.connect/4collapses everything into"Key exchange failed"or"Unable to connect using the available authentication methods". The only way out is for the callback to report to the calling process, which is whatKeyCbdoes.
The keypair is derived, not random
A random per-session keypair cannot survive CrowdControl.Provider.reconnect/1:
the key lives in RAM, gcp_compute has no instances.setMetadata, so the
VM's authorized_keys is fixed at create time and a node restart would leave
a live sandbox permanently unreachable.
So the ed25519 seed is sha256("cc-gce-ssh/v1" <> agent_token), where the
agent token is CrowdControl.Provider.token/1 — itself an HMAC of
:sandboxd_secret over the session key. reconnect/1 therefore
re-derives the identical keypair from the persisted session_key alone, and
the private half still never exists anywhere but memory. It is the same trade
the agent token already makes, with the same failure mode: rotating
:sandboxd_secret fails reattach closed.
The seed is one-way from the token (sha256 of it, with a domain tag), so neither derived secret discloses the other, and both are per-session.
Host keys: what is and is not checked
:host_key_fp pins the VM's host key by SHA-256 fingerprint when the caller
has one. Nothing supplies it for an ordinary per-session VM: a fresh GCE
guest generates its host key on first boot, and reading it back needs
instances.getSerialPortOutput, which gcp_compute does not wrap.
So by default the first host key presented is accepted. Two alternatives were
considered and rejected: silently_accept_hosts: true (strictly worse — it
also disables pinning when a fingerprint is known), and generating the host
key ourselves and shipping its private half in instance metadata (which would
make it readable by every project viewer, i.e. would hand the MITM key to
more parties than it protects against). This is a documented regression
against CrowdControl.Provider.Docker, where the transport is a loopback
socket and there is nothing to authenticate.
Teardown
:ssh.close/1 is the only way to remove a tunnel: :ssh.stop_listener/2
returns :ok and does nothing to a forward listener, and repeated
tcpip_tunnel_to_server/6 calls stack up additional listeners. close/1 is
idempotent, and dropping the connection closes the local port promptly — an
in-flight request fails within milliseconds rather than hanging.
Summary
Types
An :ssh connection reference — a local pid, never persisted.
An OTP #'ECPrivateKey'{} record holding an ed25519 keypair.
Functions
Whether the SSH connection is still up.
Close a tunnel and every forward listener under it.
The session's ed25519 keypair, derived from session_key.
The ssh-keys metadata value that authorizes this session's key.
Connect to host and forward a fresh loopback port to the VM's agent port.
The Linux user the tunnel authenticates as.
Types
Functions
Whether the SSH connection is still up.
The disambiguator for an agent HTTP failure: the transport error is
:socket_closed_remotely whether the agent is not listening, forwarding was
denied, the SSH connection dropped, or the VM is gone. Only the connection
ref can tell those apart.
@spec close(conn() | nil) :: :ok
Close a tunnel and every forward listener under it.
Idempotent, and a no-op for a handle that never had one.
The session's ed25519 keypair, derived from session_key.
In memory only, and identical on every call for a given session key and
:sandboxd_secret — see the moduledoc for why that is a requirement rather
than a shortcut.
The ssh-keys metadata value that authorizes this session's key.
USERNAME:KEY_VALUE, the format the guest agent's getUserKeys parses. The
non-expiring form is deliberate: expiry lives in a google-ssh {json} key
comment, a malformed expireOn makes the guest agent drop the key
entirely — total silent failure — and a custom comment before the marker
silently disables expiry anyway. Exposure is bounded by VM deletion and
scheduling.maxRunDuration instead, both of which are enforced server-side.
Connect to host and forward a fresh loopback port to the VM's agent port.
Returns {:ok, local_port, conn}. conn belongs in
CrowdControl.Provider.Endpoint's :transport, never in a persisted handle:
it is a local pid, and the local port is OS-assigned and different on every
reconnect.
Options
:ssh_port— default22:agent_port— the portsandboxdbinds on the VM's loopback, default8080:deadline— aSystem.monotonic_time(:millisecond)deadline for the whole retry loop; defaults to one:connect_timeoutfrom now:connect_timeout— per attempt, default10_000:host_key_fp— pin the VM's host key by"SHA256:…"fingerprint
Three connect failures are retried until the deadline, because all three are
the normal state of a booting VM: :econnrefused (sshd not listening yet),
:timeout (TCP accepts, no SSH banner yet), and authentication failure (the
guest agent has not yet turned the metadata key into an authorized_keys
line). Surfacing any of them immediately would fail every acquire.
@spec ssh_user() :: String.t()
The Linux user the tunnel authenticates as.