One Google Compute Engine VM per sandbox, running sandboxd, reached through an SSH tunnel to the VM's loopback.

Requires the optional :gcp_compute dependency and the OTP :ssh application. Nothing else: no gcloud, no IAP client, no agent on the host.

CrowdControl.run("hello",
  backend:
    {CrowdControl.Backend.Sandboxd,
     provider:
       {CrowdControl.Provider.Gce,
        project: "my-project",
        zone: "us-central1-a",
        sandboxd_url: "https://.../sandboxd-linux-amd64.tar.gz",
        sandboxd_sha256: "…64 hex…"}}
)

Network posture, chosen rather than inherited

external_ip defaults to true, and that is the safe default here only because of what surrounds it:

• sandboxd binds 127.0.0.1 on the VM, so the agent is not on the network at all. It is reachable exclusively through the caller's SSH tunnel, whose local listener is bound to loopback on this node.
• TCP 22 is therefore the only reachable port, authenticated by a per-session ed25519 key with PasswordAuthentication never in play.

Two consequences to decide about deliberately rather than discover:

• The default VPC allows 0.0.0.0/0 on port 22. On such a network the VM's sshd is internet-facing (publickey-only). Pass :tags and attach a firewall rule scoped to your egress addresses if that is not acceptable; the provider cannot do it for you, since it creates no firewall rules.
• external_ip: false is the hardened mode: no public address at all. It requires same-VPC connectivity from the node that calls acquire/1 — there is no pure-Elixir IAP tunnel client — and Cloud NAT (or a private artifact mirror), because the startup script fetches apt packages and the sandboxd release over the network.

Note that gcp_compute's own :external_ip default is also true, so this is stated rather than relied upon: a provider that forgot it would ship public sandbox VMs.

A leaked spot VM bills forever

This is the highest-stakes failure in the provider, so it is defended twice:

• Every failure on the acquire path destroys the instance before returning, including a failed instances.insert — whose failure modes include an operation poll that timed out after the VM was created. The instance name is derived from the session key before the insert, so the rollback can always name the VM.
• scheduling.maxRunDuration plus spot's instanceTerminationAction: DELETE is a server-side backstop that needs no BEAM. If this node dies mid-acquire/1, nothing local knows the VM exists and CrowdControl.Reaper never sees it; GCE deletes it anyway.

:max_run_duration therefore has no "off": a caller who wants a long-lived sandbox passes a large number.

:ready_timeout, measured

Measured on a real spot e2-small in us-central1-a, no :bootstrap_script, release tarball in a same-region bucket:

:ready_timeout bounds the last three — it starts once the insert operation is DONE — so the measured requirement is 31.1s. The default is 180_000, about six times that, because the number this has to survive is not the one above: it is the same boot with a :bootstrap_script that installs a CLI. :running costing nothing is worth noticing — by the time the operation reports DONE the guest is already RUNNING with an address, so nearly all of the wait is the guest finishing its own boot, apt-get, and the release download.

Raise it for a heavy bootstrap; lower it for a prebuilt image, where 60s is ample. Attach to [:crowd_control, :gce, :phase] and measure your own image rather than guessing — that is what the events are for. No behaviour in this module depends on the specific value, but :max_run_duration's floor is derived from it, so an inflated :ready_timeout inflates the orphan backstop too.

Telemetry

acquire/1 emits one event per phase, on success and on failure:

[:crowd_control, :gce, :phase]
measurements: %{duration_ms: non_neg_integer()}
metadata:     %{phase: :insert | :running | :ssh | :health,
                result: :ok | :error,
                instance_name: String.t(),
                zone: String.t()}

A failing phase is emitted with result: :error, which is the one a caller most needs: "it timed out" is not actionable, ":ssh timed out after 180s" names the firewall rule.

What is persisted, and what reattach needs

The Store record keeps five fields — project, zone, instance name, owner, session key. CrowdControl.Provider.scrub/1 drops everything else, because everything else is either a credential (%GcpCompute.Config{} holds a live token-provider argument; :api_key/:env may hold real keys) or a local pid.

So a different node reattaching to a sandbox needs the client config from configuration rather than from the record:

config :crowd_control,
  gce: [project: "my-project", zone: "us-central1-a"]

Anything else reconnect/1 reads from options — :agent_port, :ready_timeout, :ssh_port — belongs in the same place if it is not the default. CrowdControl.Reaper's own reattach path is unaffected: it passes the handles list_live/1 returned, which carry the options it was called with.

The tunnel's keypair is not persisted either, and does not need to be: it is derived from the session key on every connect. See CrowdControl.Provider.Gce.Tunnel.

Labels reject ., so the Docker keys cannot be reused

GCE label keys and values allow only lowercase letters, digits, - and _. crowd_control.session is therefore illegal, and so is a raw owner like nonode@nohost. This provider uses crowd_control-session, crowd_control-owner-hash (a sha256 prefix, exactly the trick CrowdControl.Backend.Kubernetes uses for the same reason) and crowd_control-agent, and puts the raw owner in instance metadata, where values are unconstrained. CrowdControl.Reaper re-checks the raw owner exactly before destroying anything, so both gates stay honest.

Options

Client:

• :project, :zone, :token_provider — see GcpCompute.Config.new/1; or pass a ready %GcpCompute.Config{} as :gce_config. Application env under :gce fills in the rest.

Agent image (required — see CrowdControl.Provider.Gce.Startup):

• :sandboxd_url — release tarball URL
• :sandboxd_sha256 — its SHA-256; mandatory, never skipped
• :bootstrap_script — shell run as root before the agent is installed

Instance shape, all passed through to GcpCompute.Instance.spec/1:

• :machine_type, :source_image, :disk_size_gb, :network, :subnetwork, :tags, :service_account, :scopes
• :spot — default true
• :external_ip — default true; see above
• :max_run_duration — seconds, and there is no "off": a caller who wants a long-lived sandbox passes a large number. Defaults to :ready_timeout + the session's own :timeout + 5 minutes, and an explicit value below :ready_timeout + 5 minutes is refused — a deadline that can expire while acquire/1 is still waiting deletes live work and reports it as a failed bootstrap.
• :metadata — extra instance metadata. This provider's own keys are merged over it: a caller-supplied ssh-keys would lock the tunnel out of its own sandbox.

Timing and transport:

• :ready_timeout — operation DONE → healthy agent, default 180_000; measured requirement is 31s for a sandbox with no bootstrap script
• :insert_timeout, :delete_timeout — operation polls, default 300_000
• :agent_port — default 8080, :capture_path — default /var/log/cc/out.jsonl
• :ssh_port — default 22
• :host_key_fp — pin the VM's host key; see CrowdControl.Provider.Gce.Tunnel for why nothing supplies it by default
• :req_adapter — test seam, threaded into the agent endpoint's Req options. The GCP client's own seam is :req_options inside the config.

No service account is attached unless :service_account is set, and that is deliberate: with one, the sandboxed CLI can mint project credentials from the metadata server with the granted scopes.