Runs the CLI inside a Kubernetes Pod over the API server.

Requires the optional :kubereq dependency:

{:kubereq, "~> 0.4.4"}

Session-facing semantics are indistinguishable from CrowdControl.Backend.Docker: the same FIFO/tee I/O architecture, the same byte-exact resume, the same reader contract, reattachable?/0 == true. What differs is everything about how, and three of those differences are load-bearing enough to state up front.

How I/O works

provision  POST   /api/v1/namespaces/{ns}/pods
           initContainer: mkfifo -m 600 <fifo> && mkdir -p <teedir>
           container:     wait for <status>, then exit with it

exec       POST   /api/v1/namespaces/{ns}/pods/{pod}/exec   (stdin)
           sh -c 'umask 077; cat > <env>'      <- the secrets channel
           then a second, detaching exec:
           setsid sh -c 'echo $$ > <launcher>; . <env>; rm -f <env>;
                         exec 3<> <fifo>;
                         { <cli> <&3; echo $? > <status>.partial; }
                           | tee <tee>;
                         mv -f <status>.partial <status>'                          </dev/null >/dev/null 2>&1 &

write      POST   .../exec   sh -c 'printf %s <escaped> >> <fifo>'

read       POST   .../exec   tail -c +<byte_offset + 1> -f <tee>
           over a long-lived `Kubereq.PodExec` websocket

destroy    DELETE /api/v1/namespaces/{ns}/pods/{pod}?gracePeriodSeconds=0

Three details are load-bearing and were all established empirically, exactly as under Docker:

• The FIFO is held open read-write (exec 3<> <fifo>). A plain < <fifo> redirect sees EOF the moment the first writer detaches, which collapses the pipeline and kills the CLI — so the second prompt of every session would be lost.
• tail -c +N is 1-indexed, hence byte_offset + 1. Off by one here duplicates a byte per resume, which corrupts the JSON line stream.
• PID 1 relays the CLI's exit status, and is not the CLI itself. The container's process cannot be the CLI, because the tee file has to outlive any individual exec and the CLI is started later by a detaching one. But it must still notice the CLI: setsid makes the CLI a grandchild, so while PID 1 was sleep infinity a crashed CLI left the container Running, tail -f never ended, no :eof was ever cast, and the session waited forever while the Pod billed forever. The launcher writes the CLI's own status after tee drains — never before, or PID 1 could exit while bytes were still buffered — and PID 1 adopts it. A launcher killed before it can report is detected through its pid file and reported as exit 1, because that hang is the same bug one level up.

Secrets travel the exec stdin channel, never argv and never the API object

The Kubernetes exec API has no env parameter — pods/exec has no such field and kubectl exec has no --env. Docker's first-class Env array, which is what keeps provider keys out of the sandbox's own ps there, simply does not exist here. The two obvious replacements are both worse than the problem:

• env in the Pod spec puts the key in the Pod object, i.e. in etcd, readable by anyone with get pods and printed by kubectl describe. That trades an in-sandbox ps leak for a cluster-wide one.
• A Secret plus envFrom has the same etcd residency, plus secrets RBAC, plus a second object left behind on a crash.

So the env arrives as a file written over the exec stdin channel (websocket channel 0) at umask 077, and the launch command sources and unlinks it before the CLI starts. The bytes never enter argv, never enter the API object, and the file is gone by the time the sandbox can read anything. This is the same env-file indirection CrowdControl.Backend.Local uses, with the same CrowdControl.Backend.Shell.escape/1 oracle.

Live and resume are the same code path

start_reader/3 is reattach/2 at offset 0, as under Docker. One addition: because tail -f never ends while the Pod lives, a websocket close frame means the channel dropped, not the stream — so the reader reconnects at byte_offset instead of casting :eof. Resume is free by construction, so a transport blip costs nothing. :eof is cast only once the Pod is confirmed not Running, or after five consecutive fruitless reconnects.

Two hardening regressions versus Docker

Both are stated rather than silently dropped, and both are in SECURITY.md:

• No PidsLimit equivalent. Docker sets a 512-PID fork-bomb ceiling deliberately and separately from Memory. There is no Pod-spec field for it; podPidsLimit is node-level kubelet configuration. A fork bomb in model output is unbounded unless the cluster operator sets it.
• No noexec,nosuid on the writable mounts. Docker's Tmpfs takes mount flags; emptyDir mounts rw,relatime with no flag control, so /tmp can stage and execute a binary even under readOnlyRootFilesystem: true.

Against that, two hardening requirements exist here that Docker has no analogue for, and they are not options:

• automountServiceAccountToken: false — a default Pod is handed a projected API credential and a reachable API server. That is a live cluster credential sitting on disk inside a sandbox running untrusted model-driven code.
• enableServiceLinks: false — otherwise every Service's host and port is injected into the sandbox's environment: free cluster reconnaissance.

Network posture is never inferred

Unlike Docker, where :network_mode defaults to "none" and a Pod therefore starts with no network at all, a Kubernetes Pod always has cluster networking. There is no "none". :network is therefore explicit:

• :deny_all — this backend creates a deny-all Ingress+Egress NetworkPolicy selecting the Pod, before the Pod exists, and deletes it with the Pod.
• {:policy, name} — assert a policy the caller manages. It is fetched and provisioning fails if it is absent, rather than trusting the claim.
• :unrestricted — the Pod can reach the cluster and the internet.

Omitting :network and setting :proxy_url or :api_url is refused with {:error, {:k8s, :network_policy_required}}, for the same reason Docker refuses to infer bridge.

A declaration is not enforcement. NetworkPolicy objects are accepted by any API server, but only enforced by a CNI with a policy controller — OrbStack, for instance, accepts them and enforces nothing. So :deny_all runs a one-time per-cluster enforcement probe (a throwaway Pod under a deny-all policy attempting egress) and refuses to start on {:error, {:k8s, :network_policy_not_enforced}}. Reporting a boundary that does not exist is worse than refusing to start.

Options

• :image — Pod image (required). Needs the CLI plus sh, tail, tee and head on PATH; busybox and coreutils both suffice. head -c is what bounds the credential read — see API.exec_stdin/5 for why stdin EOF cannot be used for that
• :namespace — default: the kubeconfig context's namespace, else "default"
• :kubeconfig — a %Kubereq.Kubeconfig{}, a pipeline module, or {module, opts}; default Kubereq.Kubeconfig.Default, which covers both a developer's ~/.kube/config and an in-cluster ServiceAccount

• :network — :deny_all | {:policy, name} | :unrestricted; see above

• :network_probe — false skips the :deny_all enforcement probe for callers who already know their CNI enforces
• :network_probe_image — probe image, default "busybox:1.36"
• :network_probe_url — probe internet egress instead of the default, which is a TCP connect to the API server's ClusterIP. The default needs no DNS and no internet, so it does not make a security decision depend on external reachability; set this only if internet egress is what you need proven blocked
• :cpus — fractional CPU limit, e.g. 1.5
• :memory — byte limit, e.g. 512 * 1024 * 1024
• :tee_path — default /var/log/cc/out.jsonl
• :fifo_path — default /var/run/cc.fifo
• :env_path — default /var/run/cc.env
• :timeout — HTTP receive timeout, default 30s
• :exec_timeout — wall-clock bound on every short exec, default 15s
• :provision_timeout — wall-clock bound on reaching Running, default 120s
• :pod_poll_ms — reader's idle Pod-liveness poll, default 60s
• :max_inflight_bytes — reader backpressure watermark, default 4 MiB
• :proxy_url, :session_token — see the egress proxy contract in SECURITY.md
• :runtime_class — runtimeClassName, e.g. "gvisor" or "kata". This is the Kubernetes-level sandboxing control and the only option here that changes which kernel the container talks to; see "Sandbox runtimes" below
• :node_selector — map of node labels, e.g. %{"sandbox.gke.io/runtime" => "gvisor"}
• :tolerations — list of raw toleration maps, needed to schedule onto the tainted node pools sandbox runtimes usually run on

Hardening:

• :cap_drop — default ["ALL"]
• :allow_privilege_escalation — default false
• :run_as_user / :run_as_group — opt-in; sets runAsNonRoot too
• :readonly_rootfs — default false. The FIFO and tee directories (Path.dirname/1 of :fifo_path and :tee_path, so /var/run and /var/log/cc by default) are emptyDir volumes either way, because the init container has to hand the FIFO across to the sandbox. Turning this on makes them medium: Memory with a size limit, adds /tmp, and sets readOnlyRootFilesystem on the sandbox container.
• :volume_sizes — per-mount-path size limits, applied under :readonly_rootfs; default 64Mi, 8Mi for /var/run

RBAC

The identity this backend runs as needs, in :namespace:

pods            create, get, list, delete
pods/exec       create
networkpolicies create, get, delete     # only under :network :deny_all