# GcpCompute [![Hex.pm](https://img.shields.io/hexpm/v/gcp_compute.svg)](https://hex.pm/packages/gcp_compute) [![Docs](https://img.shields.io/badge/hex-docs-blue.svg)](https://hexdocs.pm/gcp_compute) Spawn and manage **Google Compute Engine** instances from Elixir, over the Compute REST API. Pluggable auth, telemetry on every call, and an ergonomic builder for the (verbose) instance body. > [!WARNING] > **`spot: true` is the default.** A bare > `GcpCompute.launch(config, "worker-1")` provisions a **Spot VM**: cheap and > preemptible, and when GCP preempts it the instance **deletes itself along > with everything on its disk** (`provisioningModel: "SPOT"`, > `automaticRestart: false`, `instanceTerminationAction: "DELETE"`). That is > deliberate — this library is built for disposable batch workers — but pass > `spot: false` for anything whose disk you care about. > **Why REST and not gRPC?** The Compute Engine API is **REST/JSON only** — it > has no gRPC endpoint (it's the one notable exception among GCP APIs). So unlike > [`pubsub_grpc`](https://hex.pm/packages/pubsub_grpc), there is no connection > pool to run: a request is a handful of calls dominated by the 20–40s the VM > takes to boot, where pooling buys nothing. `GcpCompute` talks to it with > [`Req`](https://hexdocs.pm/req); a `GcpCompute.Config` is the only handle you > pass around. The right place for gRPC + [`grpc_connection_pool`](https://hex.pm/packages/grpc_connection_pool) > is an agent running *inside* the machine you spawn — see [Roadmap](#roadmap). ## Installation ```elixir def deps do [ {:gcp_compute, "~> 0.3.0"}, # Recommended token provider (optional dependency): {:goth, "~> 1.4"} ] end ``` ## Quickstart ```elixir # 1. Start Goth to mint OAuth tokens (in your application supervision tree) children = [{Goth, name: MyApp.Goth}] # 2. Build a config once and reuse it {:ok, config} = GcpCompute.Config.production( project: "my-project", zone: "us-central1-a", goth: MyApp.Goth ) # 3. Spawn a cheap, self-deleting Spot VM and wait until it exists {:ok, instance} = GcpCompute.insert_instance_and_wait(config, name: "worker-1", machine_type: "e2-micro", spot: true, max_run_duration: 3600, # hard server-side TTL (seconds) startup_script: "#!/bin/bash\necho ready > /tmp/ready", labels: %{"owner" => "platform"} ) GcpCompute.Instance.external_ip(instance) #=> "34.x.x.x" # 4. Tear it down (and wait for the delete operation) {:ok, _op} = GcpCompute.delete_instance_and_wait(config, "worker-1") ``` Prefer to drive the lifecycle yourself? Every mutating call returns an `GcpCompute.Operation` you can poll: ```elixir {:ok, op} = GcpCompute.insert_instance(config, name: "worker-1", machine_type: "e2-micro") {:ok, done} = GcpCompute.wait_for_operation(config, op, timeout: :timer.minutes(3)) ``` ## Finding instances ```elixir # Exact name. Returns a tagged 404 when it does not exist — match, don't rescue. case GcpCompute.get_instance(config, "worker-1") do {:ok, vm} -> vm {:error, %GcpCompute.Error{status: 404}} -> :not_found end # Everything in the config's zone (single page of items, token discarded). {:ok, instances} = GcpCompute.list_instances(config) # Server-side search. Note: instances.list takes GLOBS, not regexes. {:ok, workers} = GcpCompute.list_instances(config, filter: ~s(name = "worker-*")) {:ok, running} = GcpCompute.list_instances(config, filter: ~s(status = "RUNNING")) {:ok, batch} = GcpCompute.list_instances(config, filter: ~s(labels.role = "batch")) {:ok, both} = GcpCompute.list_instances(config, filter: ~s(labels.env = "prod" AND status = "RUNNING")) # Explicit pagination when you need the token. {:ok, %{items: page1, next_page_token: token}} = GcpCompute.list_instances_page(config, max_results: 50) {:ok, %{items: page2}} = GcpCompute.list_instances_page(config, max_results: 50, page_token: token) ``` Three sharp edges, all verified against the live API: - **Globs, not regexes.** `name = "worker-*"` works; `name ~ "worker"` is rejected with a 400 `Invalid list filter expression`. - **`order_by` is narrow.** `"name"` and `"creationTimestamp desc"` are accepted; `"name desc"` is a 400. - **Listing is per-zone.** The `aggregatedList` endpoint is not wrapped, so if you do not know an instance's zone you must iterate zones yourself. There is also no `stream`/auto-pagination helper — `list_instances/2` returns one page and discards the token, so loop on `list_instances_page/2` for large zones. ## Logs, tags and SSH ```elixir # "The VM's logs" = serial console output. This is where a :startup_script's # output lands, and it is the only log surface the Compute API itself offers. {:ok, %{contents: log, next: next}} = GcpCompute.instance_logs(config, "worker-1") # Poll incrementally — the buffer is ~145 KB on a booted Debian image, so pass # the previous :next back as :start rather than refetching the whole thing. {:ok, %{contents: delta}} = GcpCompute.instance_logs(config, "worker-1", start: next) # Ports 1..4; 1 carries boot and startup-script output. Anything else is # rejected locally, before a request goes out. {:ok, _} = GcpCompute.instance_logs(config, "worker-1", port: 2) ``` **Network tags vs labels** are different things and both round-trip: ```elixir {:ok, vm} = GcpCompute.launch(config, "web-1", tags: ["http-server", "ssh"], # firewall/route targets -> vm.tags labels: %{"env" => "prod"} # key/value metadata -> vm.labels ) vm.tags #=> ["http-server", "ssh"] vm.labels #=> %{"env" => "prod"} ``` Mutating either *after* creation (`instances.setTags` / `setLabels`) is not wrapped: both take a fingerprint for optimistic concurrency, and hiding a read-modify-write behind a helper would hide a lost-update race. The fingerprints are in `vm.raw` if you want to call those endpoints yourself. **SSH.** This library authorises keys but does not open connections: ```elixir {:ok, vm} = GcpCompute.launch(config, "worker-1", ssh_keys: %{"deploy" => "ssh-ed25519 AAAAC3Nz… deploy@laptop"} ) GcpCompute.Instance.external_ip(vm) #=> "34.x.x.x" — then use your own client ``` Deliberately no in-library SSH client: it would mean owning host-key verification policy, key parsing and `known_hosts` — a larger security surface than the rest of this package — and GCE SSH also involves OS Login, a separate API. For "run a command and see the output", prefer `:startup_script` plus `instance_logs/3`: no inbound access, no keys, no extra API. ## Boot disks: image, snapshot, or an existing disk A boot disk has exactly one source. Passing two is rejected before the request — GCP's own answer is a 400 naming a JSON field you never typed (`Cannot specify both 'source' and 'initializeParams'`). ```elixir # 1. From an image (the default: debian-12 if you say nothing). GcpCompute.launch(config, "worker-1", disk_size_gb: 50, disk_type: "pd-ssd") # 2. From a snapshot — a NEW disk is created, so the snapshot is never mutated # and any number of instances can boot the same one concurrently. GcpCompute.launch(config, "worker-2", source_snapshot: "nightly-backup") # 3. Attaching an EXISTING disk — nothing is created; it must be in the same # zone and not already attached. GcpCompute.launch(config, "worker-3", source_disk: "stateful-1") ``` Bare names are qualified for you (`nightly-backup` → `global/snapshots/nightly-backup`); pass a full path to cross projects. **`autoDelete` defaults differ, deliberately.** A disk *created* from an image or snapshot belongs to the instance and is deleted with it. A disk you *attached* already existed, so deleting the instance leaves it alone. Override either way with `boot_disk_auto_delete:`. Snapshot and disk *management* is not wrapped: those are different resources (`disks`, `snapshots`), and this package is the instance client. Booting *from* one is an instance concern, which is why it lives here. ## Spot VMs and preemption `:spot` defaults to **`true`**, so the quickstart VM is preemptible and, with the default `instanceTerminationAction: "DELETE"`, **deletes itself** when preempted. That is the right default for disposable batch workers and the wrong one if the disk matters — pass `spot: false` then. ```elixir GcpCompute.Instance.spot?(vm) #=> true # A preempted VM is DELETED, so `get/3` answers 404 and cannot tell you why it # went away. The operation outlives the instance, and can: case GcpCompute.instance_preemption(config, "worker-1") do {:ok, nil} -> :not_preempted # also the answer for a name that never existed {:ok, op} -> {:preempted_at, op.end_time} end # Or the whole audit trail — "what happened to my VM?" {:ok, ops} = GcpCompute.instance_operations(config, "worker-1") Enum.map(ops, &{&1.operation_type, &1.status_message}) #=> [{"compute.instances.preempted", "Instance was preempted."}, {"insert", nil}] # Test your own preemption handling on purpose. This is a REAL preemption: GcpCompute.simulate_maintenance_event(config, "worker-1") ``` Measured against a live Spot VM: `RUNNING` → `STOPPING` at ~60 s → gone (404) at ~73 s. Budget minutes, not seconds. There is no retry-on-preemption loop here: deciding whether to relaunch, where, and how often is orchestration, and it belongs in the supervised layer described under [Roadmap](#roadmap) rather than in a REST client. ## Examples & guides - **Runnable examples** in [`examples/`](https://github.com/nyo16/gcp_compute/tree/master/examples): - [`stubbed_demo.exs`](https://github.com/nyo16/gcp_compute/blob/master/examples/stubbed_demo.exs) — the full insert → poll → get flow offline, **no GCP or credentials** (`elixir examples/stubbed_demo.exs`). - [`spawn_spot_vm.exs`](https://github.com/nyo16/gcp_compute/blob/master/examples/spawn_spot_vm.exs) — a real (billable) Spot VM. - **Live smoke tests** (real project, real cents) — see [Testing without GCP](#testing-without-gcp) for what each one is for: [`smoke_test.exs`](https://github.com/nyo16/gcp_compute/blob/master/examples/smoke_test.exs), [`smoke_coverage.exs`](https://github.com/nyo16/gcp_compute/blob/master/examples/smoke_coverage.exs), [`smoke_advanced.exs`](https://github.com/nyo16/gcp_compute/blob/master/examples/smoke_advanced.exs). - **Guides:** [Getting Started](guides/getting-started.md) · [Configuring Machines](guides/configuring-machines.md) · [Testing](guides/testing.md) ## Configuration `GcpCompute.Config` is validated with `NimbleOptions`. Three builders cover the common cases: ```elixir # Production — tokens minted by Goth {:ok, config} = GcpCompute.Config.production(project: "p", goth: MyApp.Goth) # From application env # config :my_app, :gcp_compute, # project: "p", zone: "europe-west4-a", # token_provider: {GcpCompute.TokenProvider.Goth, MyApp.Goth} {:ok, config} = GcpCompute.Config.from_env(:my_app, :gcp_compute) # Local / emulator / tests — a static token, no Goth required {:ok, config} = GcpCompute.Config.local(project: "p", base_url: "http://localhost:8080/compute/v1") ``` | Option | Default | Notes | | ---------------- | ---------------------------------------------- | -------------------------------------------------- | | `:project` | — (required) | GCP project id. | | `:zone` | `"us-central1-a"` | Default zone; override per call with `zone:`. | | `:token_provider`| `{TokenProvider.Goth, GcpCompute.Goth}` | `{module, arg}` implementing `GcpCompute.TokenProvider`. | | `:base_url` | `"https://compute.googleapis.com/compute/v1"` | Point at the emulator or a proxy. **Trusted app config — never derive from user input.** | | `:req_options` | `[]` | Merged into every `Req` request (`:retry`, `:adapter`, …). | | `:allow_insecure`| `false` (`local/1` sets `true`) | Permit a non-`https://` `:base_url`. Off by default so a bearer token is never sent in cleartext. `new/1` warns when it is on and the host is not loopback/RFC1918/internal. | ### Pluggable auth Auth is a behaviour, `GcpCompute.TokenProvider`, so the library never hard-depends on Goth. Built-ins: `TokenProvider.Goth` (production), `TokenProvider.Static` (tests/emulator). Bring your own (workload identity, metadata server, Vault) by implementing one callback: ```elixir defmodule MyApp.MetadataToken do @behaviour GcpCompute.TokenProvider @impl true def fetch_token(_arg), do: {:ok, %{token: fetch_from_metadata_server()}} end ``` ### Configurable machines `GcpCompute.Instance.spec/1` turns friendly options into the Compute insert body (defaults shown): | Option | Default | | ------------------- | --------------------------------------------------- | | `:machine_type` | `"e2-micro"` | | `:source_image` | `debian-cloud/.../debian-12` | | `:disk_size_gb` | `10` | | `:spot` | `true` (SPOT, no auto-restart, DELETE on terminate) | | `:max_run_duration` | `nil` (set seconds for a hard server-side TTL) | | `:external_ip` | `true` | | `:startup_script`, `:metadata`, `:labels`, `:tags`, `:network`, `:subnetwork`, `:service_account`, `:scopes` | — | Need a field it doesn't cover? Pass a raw Compute body map to `GcpCompute.Instances.insert/3` instead. ## Telemetry Every API call is a `:telemetry.span/3`: | Event | Metadata | | -------------------------------------- | ------------------------------------------------- | | `[:gcp_compute, :request, :start]` | `method`, `path`, `project` | | `[:gcp_compute, :request, :stop]` | `+ result` (`:ok`/`:error`), `http_status` | | `[:gcp_compute, :request, :exception]` | `+ kind`, `reason`, `stacktrace` | ```elixir GcpCompute.Telemetry.attach_default_logger() # dev convenience # or wire into Telemetry.Metrics / your reporter ``` ## Testing without GCP Tests stub the network via Req's native `:adapter` option — no Plug, no creds. Req 0.7 takes a **module**, not a function: ```elixir defmodule MyStub do def run(request) do {request, %Req.Response{status: 200, body: %{"name" => "worker-1"}}} end end config = GcpCompute.Config.local!(project: "test", req_options: [adapter: MyStub]) ``` See `test/support/req_stub.ex` for the routing-table helper used by the suite — it asserts the bearer token on every stubbed request and fails loudly on an unrouted one, so a dropped header or a mangled query cannot pass silently. ### …and three that need real GCP Stubs cannot catch everything, and it is worth knowing exactly where the line is. An in-process adapter computes no headers, so it could not reproduce this: the Compute API rejects a bodiless POST with **HTTP 411 Length Required**, which broke `launch/3`, `terminate/3` and every `*_and_wait` helper end to end while the whole stub suite stayed green. Two live scenarios are the counterweight, and between them they have found every bug the stubs could not. **`examples/smoke_test.exs`** — the fast pre-publish check. Launches one `e2-micro` Spot VM, verifies the parse of a real payload, tears it down, and reports per-request timings from the library's own telemetry, including whether a single `operations.wait` long poll exceeded Req's 15 s default `receive_timeout` — the only real proof the long-poll fix works. An observed run had one block for **51 s** and succeed. **`examples/smoke_coverage.exs`** — the full surface, for the parts whose correctness only the real server can confirm, because a stub validates our request construction against our own assumptions. 19 checks: pagination (`max_results`/`page_token` genuinely paginating, not just our translation table agreeing with itself), `stop`/`start`, `spot: false`, `Operations.get/2`, `wait_for_operation/3` called directly, a raw map body, `:request_id`, a server-side `:filter`, and `from_env/2` + `config/2`. **`examples/smoke_advanced.exs`** — 30 checks covering the three things that need real infrastructure to be falsifiable at all: - **Disks.** Boot from a snapshot and from an attached existing disk, then prove `autoDelete: false` really preserves a disk the instance did not create. No stub can tell you whether GCP accepts a bare snapshot name. - **Network.** Firewall rules, ports and tags, verified by opening real TCP connections to the VM — the only test here that leaves both the process *and* the Google API, and the only way to know `:tags` are load-bearing rather than decorative JSON. Expectations are **derived from the project's live rule set**, never hardcoded, because an auto-created default network ships `default-allow-ssh` and a custom VPC ships nothing. The sharp assertion is the difference between a **timeout** (packets dropped by a firewall) and **`:econnrefused`** (packets arrived, nothing listening) — conflating those two is how you convince yourself a firewall works when it does not. - **Spot.** Preempt a VM on purpose with `simulate_maintenance_event/3`, watch it be deleted, then confirm `preemption/3` still finds the evidence once the instance itself is gone. The port assertions refuse to run until the VM's **serial console** reports the test listener actually bound — read with `instance_logs/3`, so the scenario dogfoods the library. That gate exists because it was needed: an early version used `nc`, which Debian 12 images do not ship, and the dead listener passed as a firewall result for two runs. `:timeout` cannot distinguish "firewall dropped it" from "nothing was ever listening", so the listener has to be proven independently. The tag check needs `compute.firewalls.create` (`roles/compute.securityAdmin`). Without it, that one check **SKIPs with the exact grant command** rather than passing quietly. ```bash export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/smoke.json export SMOKE_PROJECT=my-project SMOKE_ZONE=us-central1-a mix run examples/smoke_test.exs # ~1 min, 1 VM mix run examples/smoke_coverage.exs # ~5 min, 3 VMs mix run examples/smoke_advanced.exs # ~8 min, 4 VMs + a snapshot and a disk # needs outbound TCP to the VM ``` All three cost fractions of a cent and clean up after themselves: `e2-micro`, a 10 GB disk, `max_run_duration` as a server-side hard TTL so GCP reclaims the instance even if teardown fails, unique names per run, and a closing orphan sweep. **Run all three before every publish.** ## Roadmap This package is the **Compute client layer**. A higher-level *sandbox orchestration* layer is planned on top of it: - `gen_statem` per sandbox (`:provisioning → :probing → :running → :terminating`) - `DynamicSupervisor` + `Registry`, named profiles, per-user quotas - a reaper for TTL sweeps + orphan reconciliation (cloud resources outlive the BEAM) - the gRPC hop: an in-VM agent reached via `grpc_connection_pool`, lifecycle events published through `pubsub_grpc` ## License MIT.