CrowdControl.Provider.Gce.Tunnel (crowd_control v0.1.1)

Copy Markdown View Source

An SSH tunnel from this node's loopback to one sandbox VM's loopback agent port, with a private key that never touches disk.

Everything here was established empirically against OTP 29's :ssh and against a real OpenSSH 10.3p1 sshd — which is what a GCE VM runs — rather than from the API docs. The load-bearing results:

  • :loopback is mandatory. tcpip_tunnel_to_server/6 with ListenHost = :any (or {0,0,0,0}) binds the local listener on every interface, publishing the sandbox agent to the LAN, and still returns a perfectly normal {:ok, port}. There is no error to catch, which is why it is a constant here and a test there.
  • {:ok, port} is not evidence of reachability. The call only does a local gen_tcp:listen; the direct-tcpip channel is opened lazily, per accepted connection. A tunnel to a closed remote port succeeds identically. GET /v1/health is the only readiness proof, which is what makes CrowdControl.Provider.acquire/1's contract mandatory rather than stylistic.
  • The Timeout argument is ignored by the handler, so this module never relies on it.
  • false from is_host_key/5 does not reject a host key when silently_accept_hosts: true — OTP overrides it. Only {:error, _} is honoured on both settings. Both are set the safe way here.
  • save_accepted_host defaults to true, so it is switched off explicitly: no code path may write a known_hosts file.
  • :user_dir is validated to exist even when a custom key_cb never reads it, so an empty directory has to exist somewhere.
  • A key_cb failure reason never reaches the caller. :ssh.connect/4 collapses everything into "Key exchange failed" or "Unable to connect using the available authentication methods". The only way out is for the callback to report to the calling process, which is what KeyCb does.

The keypair is derived, not random

A random per-session keypair cannot survive CrowdControl.Provider.reconnect/1: the key lives in RAM, gcp_compute has no instances.setMetadata, so the VM's authorized_keys is fixed at create time and a node restart would leave a live sandbox permanently unreachable.

So the ed25519 seed is sha256("cc-gce-ssh/v1" <> agent_token), where the agent token is CrowdControl.Provider.token/1 — itself an HMAC of :sandboxd_secret over the session key. reconnect/1 therefore re-derives the identical keypair from the persisted session_key alone, and the private half still never exists anywhere but memory. It is the same trade the agent token already makes, with the same failure mode: rotating :sandboxd_secret fails reattach closed.

The seed is one-way from the token (sha256 of it, with a domain tag), so neither derived secret discloses the other, and both are per-session.

Host keys: what is and is not checked

:host_key_fp pins the VM's host key by SHA-256 fingerprint when the caller has one. Nothing supplies it for an ordinary per-session VM: a fresh GCE guest generates its host key on first boot, and reading it back needs instances.getSerialPortOutput, which gcp_compute does not wrap.

So by default the first host key presented is accepted. Two alternatives were considered and rejected: silently_accept_hosts: true (strictly worse — it also disables pinning when a fingerprint is known), and generating the host key ourselves and shipping its private half in instance metadata (which would make it readable by every project viewer, i.e. would hand the MITM key to more parties than it protects against). This is a documented regression against CrowdControl.Provider.Docker, where the transport is a loopback socket and there is nothing to authenticate.

Teardown

:ssh.close/1 is the only way to remove a tunnel: :ssh.stop_listener/2 returns :ok and does nothing to a forward listener, and repeated tcpip_tunnel_to_server/6 calls stack up additional listeners. close/1 is idempotent, and dropping the connection closes the local port promptly — an in-flight request fails within milliseconds rather than hanging.

Summary

Types

An :ssh connection reference — a local pid, never persisted.

An OTP #'ECPrivateKey'{} record holding an ed25519 keypair.

Functions

Whether the SSH connection is still up.

Close a tunnel and every forward listener under it.

The session's ed25519 keypair, derived from session_key.

The ssh-keys metadata value that authorizes this session's key.

Connect to host and forward a fresh loopback port to the VM's agent port.

The Linux user the tunnel authenticates as.

Types

conn()

@type conn() :: pid()

An :ssh connection reference — a local pid, never persisted.

keypair()

@type keypair() :: tuple()

An OTP #'ECPrivateKey'{} record holding an ed25519 keypair.

Functions

alive?(conn)

@spec alive?(conn() | nil) :: boolean()

Whether the SSH connection is still up.

The disambiguator for an agent HTTP failure: the transport error is :socket_closed_remotely whether the agent is not listening, forwarding was denied, the SSH connection dropped, or the VM is gone. Only the connection ref can tell those apart.

close(conn)

@spec close(conn() | nil) :: :ok

Close a tunnel and every forward listener under it.

Idempotent, and a no-op for a handle that never had one.

keypair(session_key)

@spec keypair(String.t()) :: keypair()

The session's ed25519 keypair, derived from session_key.

In memory only, and identical on every call for a given session key and :sandboxd_secret — see the moduledoc for why that is a requirement rather than a shortcut.

metadata_ssh_keys(session_key)

@spec metadata_ssh_keys(String.t()) :: String.t()

The ssh-keys metadata value that authorizes this session's key.

USERNAME:KEY_VALUE, the format the guest agent's getUserKeys parses. The non-expiring form is deliberate: expiry lives in a google-ssh {json} key comment, a malformed expireOn makes the guest agent drop the key entirely — total silent failure — and a custom comment before the marker silently disables expiry anyway. Exposure is bounded by VM deletion and scheduling.maxRunDuration instead, both of which are enforced server-side.

open(host, session_key, opts \\ [])

@spec open(String.t(), String.t(), keyword()) ::
  {:ok, pos_integer(), conn()} | {:error, term()}

Connect to host and forward a fresh loopback port to the VM's agent port.

Returns {:ok, local_port, conn}. conn belongs in CrowdControl.Provider.Endpoint's :transport, never in a persisted handle: it is a local pid, and the local port is OS-assigned and different on every reconnect.

Options

  • :ssh_port — default 22
  • :agent_port — the port sandboxd binds on the VM's loopback, default 8080
  • :deadline — a System.monotonic_time(:millisecond) deadline for the whole retry loop; defaults to one :connect_timeout from now
  • :connect_timeout — per attempt, default 10_000
  • :host_key_fp — pin the VM's host key by "SHA256:…" fingerprint

Three connect failures are retried until the deadline, because all three are the normal state of a booting VM: :econnrefused (sshd not listening yet), :timeout (TCP accepts, no SSH banner yet), and authentication failure (the guest agent has not yet turned the metadata key into an authorized_keys line). Surfacing any of them immediately would fail every acquire.

ssh_user()

@spec ssh_user() :: String.t()

The Linux user the tunnel authenticates as.