CrowdControl.Provider.Compose (crowd_control v0.1.1)

Copy Markdown View Source

A per-session Docker stack — sandbox plus sidecars — over the Engine API.

Same job as CrowdControl.Provider.Docker, one container more, and one property that provider provably cannot have: a structural egress block on the sandbox that still leaves the agent reachable from the host.

There is no docker compose dependency and there must never be one. The Engine API has no compose endpoints — compose is a client-side Go plugin that synthesises exactly the calls below — so shelling out would buy a binary dependency, a YAML round trip and a second error vocabulary in exchange for nothing.

The network shape, and why it needs two containers

The measured constraint, confirmed six independent ways against a live daemon:

On one container, Internal: true and a published port are mutually exclusive. Publishing requires at least one non-internal endpoint, and attaching one restores full internet egress.

So the sandbox and the thing the host talks to cannot be the same container:

  1. <project>-sbxInternal: true. The sandbox sits here and only here. This is the one strong egress primitive: no default route exists at all, so the internet, the Docker host, containers on every other Docker network and Docker's own embedded DNS are unreachable structurally rather than by a missing NAT rule.
  2. <project>-pub — non-internal, so PortBindings actually bind, but with com.docker.network.bridge.enable_ip_masquerade=false so the one container attached to it has no internet either.
  3. <project>-egress — a plain NAT bridge, created only if some sidecar declares egress: :allow. If nothing asks, it never exists.

A forwarder sidecar is dual-homed on 1 and 2 and publishes the agent port on 127.0.0.1:0. It runs socat TCP-LISTEN:<port>,fork,reuseaddr TCP:<sandbox>:<port>, addressing the sandbox by its network alias. fork is not decorative: a single-slot forwarder (busybox nc -e) dropped two of three concurrent requests in testing, and CrowdControl.Backend.Sandboxd holds a long-lived chunked stream open for the whole session.

The forwarder's second network is attached with POST /networks/{id}/connect after create, not by listing two endpoints in POST /containers/create: create-time dual attach is HTTP 400 below API 1.44, and daemons still report MinAPIVersion 1.40. The connect-after-create path is verified to produce an identical container.

Options

Stack:

  • :services — sidecar specs, each a map (see below). Default [].
  • :sandbox_service — the name the sandbox container is given, default "sandbox". A spec in :services with this name is the sandbox: it supplies the image, command, env and mounts, and the provider overlays the agent contract on it. With no such spec the sandbox is synthesised from :image alone.
  • :forwarder_service — the name the forwarder is given, default "forwarder". Always synthesised; see :forwarder_image.
  • :forwarder_image — default "alpine/socat:1.8.1.3". Any image with socat on PATH works; the provider sets Entrypoint itself.
  • :network[internal: true, driver: "bridge", options: %{}]. See the warning below.
  • :volumes — named volume declarations, [%{name: "workspace"}]. Created as <project>-<name> and destroyed with the stack.
  • :ready — per-service healthchecks, %{"db" => %{test: [...]}}.
  • :project_name — default "cc-<session_key>".
  • :proxy_service — the sidecar fronting the egress proxy. See below.
  • :health_timeout — deadline for the healthcheck poll, default 60_000.

Agent, exactly as CrowdControl.Provider.Docker takes them: :image, :agent_port, :capture_path, :ready_timeout, :req_adapter, :docker_host, :timeout.

Hardening (:cpus, :memory, :cap_drop, :security_opt, :pids_limit, :user, :readonly_rootfs, :tmpfs) comes from these top-level options and applies to every container in the stack, through CrowdControl.Backend.Docker.HostConfig. There are deliberately no per-service overrides: one posture per stack is the whole reason that module exists, and a sidecar quietly weaker than the sandbox it shares a network with is not a useful thing to be able to express.

Service spec

%{
  name: "proxy",                      # required, unique, DNS-safe
  image: "cc/egress-proxy:1.2.3",     # required
  egress: :allow,                     # required for sidecars, see below
  entrypoint: ["/proxy"],             # optional
  command: ["--listen", ":8080"],     # optional
  env: %{"LOG_LEVEL" => "info"},      # optional
  volumes: [%{name: "cache", target: "/cache", read_only: false}],
  depends_on: ["db"],                 # optional, topologically ordered
  user: "1000:1000",                  # optional
  port: 8080                          # only read for :proxy_service
}

Every service gets its name as an alias on the sandbox network, so services address each other by name and nothing has to learn an IP.

Posture is never inferred

  • :egress on a sidecar is required and has no default. :none keeps it on the internal network only; :allow also attaches it to the NAT bridge. Omitting it is {:error, {:compose, {:egress_required, name}}}, for the same reason CrowdControl.Backend.Docker refuses to guess :network_mode and CrowdControl.Provider.Docker refuses to guess :egress. A sidecar with :allow sits on both networks and can relay the internet into the sandbox — which is exactly what an egress proxy is for, and exactly why saying so is mandatory.
  • The sandbox service can never carry :allow; asking is {:error, {:compose, :sandbox_egress_forbidden}}.
  • network: [internal: false] is accepted, and gives the sandbox a NAT bridge and full internet. It is the one option here that throws away the module's reason to exist, so it exists only as an explicit, typed-out act.

Egress proxy

No proxy ships with this library; SECURITY.md specifies what a conforming one must do. Naming one with :proxy_service wires both halves of that contract:

  • the sandbox's environment goes through CrowdControl.Backend.Credentials.apply_credentials/2ANTHROPIC_BASE_URL points at the proxy's alias and ANTHROPIC_API_KEY becomes a per-session token, with any real :api_key removed rather than overridden;
  • the proxy receives CC_SESSION_TOKEN (the minted token, so it can recognise the session) and the real ANTHROPIC_API_KEY (so it can substitute it upstream).

The proxy must declare egress: :allow, or it could not reach the upstream API and the sandbox would fail in a way that looked like a model bug: {:error, {:compose, {:proxy_needs_egress, name}}}.

With no :proxy_service, :api_key is placed in the sandbox's own environment unchanged and no ANTHROPIC_BASE_URL is set. That is the honest no-proxy posture — the sandbox holds a real provider credential — and it is what makes the removal above observable rather than notional. :api_key and :session_token are both in CrowdControl.Store.secret_keys/0, so neither survives scrub/1 into a Store record.

Unlike CrowdControl.Backend.Docker, this provider needs no explicit :network_mode to make the proxy enforcing. There is no bridge to accidentally choose: the sandbox network is created by this module, is internal by default, and lives and dies with the session.

The published port is never persisted

Every stop/start/restart allocates a new ephemeral host port, and while a container is stopped NetworkSettings.Ports is {} rather than reporting the old one. reconnect/1 therefore always re-reads the forwarder's port. Measured behaviour, not caution.

Summary

Types

t()

@type t() :: %CrowdControl.Provider.Compose{
  agent_port: pos_integer(),
  capture_path: String.t(),
  config: keyword(),
  containers: [{String.t(), String.t()}],
  created_at: integer() | nil,
  egress_network: String.t() | nil,
  forwarder_service: String.t() | nil,
  owner: String.t() | nil,
  project: String.t() | nil,
  publish_network: String.t() | nil,
  sandbox_network: String.t() | nil,
  sandbox_service: String.t() | nil,
  session_key: String.t() | nil,
  volumes: [String.t()]
}