CrowdControl.Backend.Kubernetes.API (crowd_control v0.2.0)

Copy Markdown View Source

Every kubereq call in the project, and the error vocabulary built on top of it.

This module plays the same role for CrowdControl.Backend.Kubernetes that CrowdControl.Backend.Docker.API plays for the Docker backend: one file to audit when the client library moves. That confinement earns more here than it does there, because kubereq 0.4.4 is young and has sharp edges the backend must never see:

  • Kubereq.list/3 with into: :stream silently applies limit: 10 and silently truncates on a mid-pagination error. list_all/4 paginates by hand instead — see the comment there for why a truncated list is the most dangerous value this module can return.
  • Kubereq.PodExec.open?/1 is broken: Kubereq.Connect.handle_call(:open?, …) returns a malformed two-tuple GenServer reply and crashes the connection process. Nothing here calls it; liveness is a monitor's job.
  • Kubereq.PodExec.start_link/1 raises MatchError rather than returning {:error, _} when the websocket upgrade fails, and it links to whoever starts it and stops with the transport error as its exit reason. open_exec/5 contains both: it starts the channel from a dedicated trapping owner, so neither the raise nor a later blip can reach the caller, and callers get {:exec_down, pid, reason} instead of an exit signal.

Non-2xx is {:ok, _}

kubereq installs only a request step (Req.Request.prepend_request_steps), so nothing converts HTTP status into an error. A 404 arrives as {:ok, %Req.Response{status: 404, body: %{"kind" => "Status", …}}}. Every {:error, {:k8s, _}} in this backend is produced by normalize/1 below — this module is where the vocabulary is created, not glue around one that already exists. {:error, _} out of kubereq itself means a transport failure or a %Kubereq.Error.StepError{}.

Summary

Types

Cluster connection config.

Functions

A Req.Request with kubereq attached, pointed at resource.

Send a close frame to a PodExec started by open_exec/5.

The API server URL of the current context — the enforcement-cache key.

POST a NetworkPolicy manifest.

POST a Pod manifest.

DELETE a NetworkPolicy by name.

DELETE a Pod with no grace period.

A client/2 for the exec subresource, negotiating v4.channel.k8s.io.

Run command to completion and return everything it wrote to stdout.

Run command and feed payload to its stdin over the exec websocket.

GET a NetworkPolicy by name.

GET a Pod by name.

The loaded kubeconfig for config.

Every Pod matching label_selectors, following continue to the last page.

The container's logs, as a single binary.

The namespace to operate in.

Start a long-lived Kubereq.PodExec delivering frames to into.

Warning-class events for one Pod.

Poll/watch a Pod until callback returns true, or :timeout ms elapse.

Types

config()

@type config() :: keyword()

Cluster connection config.

:kubeconfig, :namespace, :timeout, :exec_timeout, and the :req_adapter test seam.

Functions

client(config, resource)

@spec client(
  config(),
  keyword()
) :: Req.Request.t()

A Req.Request with kubereq attached, pointed at resource.

Kubereq.attach/2 loads the kubeconfig eagerly — file reads, and possibly an exec auth plugin subprocess — every time it is called with a pipeline module. Every call in this module goes through here, and the reaper calls in a loop, so the loaded %Kubereq.Kubeconfig{} is cached in :persistent_term and the struct is what gets passed to attach/2.

close_exec(pid)

@spec close_exec(pid() | nil) :: :ok

Send a close frame to a PodExec started by open_exec/5.

Never Kubereq.PodExec.open?/1 first: in 0.4.4 Kubereq.Connect.handle_call(:open?, _, _) returns a malformed two-tuple GenServer reply, which crashes the very process being probed.

cluster_url(config)

@spec cluster_url(config()) :: String.t() | nil

The API server URL of the current context — the enforcement-cache key.

create_network_policy(config, manifest)

@spec create_network_policy(config(), map()) :: {:ok, map()} | {:error, term()}

POST a NetworkPolicy manifest.

create_pod(config, manifest)

@spec create_pod(config(), map()) :: {:ok, map()} | {:error, term()}

POST a Pod manifest.

delete_network_policy(config, name)

@spec delete_network_policy(config(), String.t()) :: {:ok, map()} | {:error, term()}

DELETE a NetworkPolicy by name.

delete_pod(config, name)

@spec delete_pod(config(), String.t()) :: {:ok, map()} | {:error, term()}

DELETE a Pod with no grace period.

kubereq has no DeleteOptions option, so gracePeriodSeconds goes through as a plain Req param. Zero because the sandbox holds no state worth draining and a lingering terminating Pod is still a billed Pod.

exec_client(config)

@spec exec_client(config()) :: Req.Request.t()

A client/2 for the exec subresource, negotiating v4.channel.k8s.io.

Requesting the subprotocol is what makes exec exit codes available at all. Without this header the API server falls back to v1 channel.k8s.io, whose channel 3 carries a human string produced by the container runtime — measured: "command terminated with non-zero exit code: Error executing in Docker Container: 7" under this cluster's runtime, but "command terminated with exit code 7" under containerd. Parsing that is parsing a runtime's prose.

Under v4 the same channel carries a JSON Status object instead, so the exit code is a field. Measured against v1.35.6+orb1: exit 7 yields %{"status" => "Failure", "reason" => "NonZeroExitCode", "details" => %{"causes" => [%{"reason" => "ExitCode", "message" => "7"}]}}, and a clean exit yields %{"metadata" => %{}, "status" => "Success"} — note that success also produces a frame, which is what makes "no news is good news" the wrong reading of channel 3.

kubereq never sets this itself, but Kubereq.Connect.connect/1 passes req.headers straight into Mint.WebSocket.upgrade/4, so a header put here reaches the wire.

exec_once(config, pod_name, command, opts \\ [])

@spec exec_once(config(), String.t(), [String.t()], keyword()) ::
  {:ok, binary()} | {:error, term()}

Run command to completion and return everything it wrote to stdout.

Bounded by :exec_timeout (default 15s). Kubereq.exec/4 blocks on a stream whose only deadline is the 10s HTTP-101 upgrade; after the upgrade its Mint.WebSocket.recv/3 waits :infinity. write/2 runs inside the session's own call path, so an unbounded exec here would wedge the session forever.

Exec exit codes are not available: kubereq never negotiates v4.channel.k8s.io, so channel 3 arrives as an undecoded {:error, binary}. This is parity with Docker, whose detached exec also never reports status — not a regression, but it does mean a successful return proves the command was started, not that it succeeded.

exec_stdin(config, pod_name, command, payload, opts \\ [])

@spec exec_stdin(config(), String.t(), [String.t()], iodata(), keyword()) ::
  :ok | {:error, term()}

Run command and feed payload to its stdin over the exec websocket.

This is the secret channel. The Kubernetes exec API has no env parameterpods/exec has no such field and kubectl exec has no --env — so Docker's first-class Env array has no counterpart. The three ways to get a provider key into a sandbox and why only one survives:

  • env in the Pod spec: puts the key in the Pod object, i.e. in etcd, readable by anyone with get pods. Trades an in-sandbox ps leak for a cluster-wide one.
  • Secret + envFrom: same etcd residency, plus secrets RBAC and a second object to leak on crash.
  • stdin: the bytes travel on websocket channel 0. They never enter argv, never enter the API object, and never appear in kubectl describe.

opts takes :container, and passing it is not optional in practice: every other exec in this module pins the container, this one did not, and on a Pod with more than one container the API server picks. The env file is the secret channel, so "whichever container the server picked" is the wrong place for it. The sandbox Pod happens to have one container plus an already-exited init container, so the omission worked by luck rather than by construction.

command must terminate on its own

It must not rely on stdin EOF — no bare cat > file. Closing the websocket to signal EOF makes the API server tear the exec down before it writes the channel-3 Status, so the command's exit code never arrives and every failure reads as success. Measured on v1.35.6+orb1, same Pod, cat > /no-such-dir/x:

with a client-side close:    [:connected, {:close, 1000, ""}]         no channel 3
without a client-side close: [:connected, {:error, "…ExitCode…1"}, ]  channel 3 present

So bound the read instead. head -c <byte_size(payload)> > file consumes exactly the payload and exits, the server sends the status, and the close frame arrives on its own. This is why the caller passes a byte count rather than letting the shell read to EOF.

Bounded by :exec_timeout like exec_once/4.

get_network_policy(config, name)

@spec get_network_policy(config(), String.t()) :: {:ok, map()} | {:error, term()}

GET a NetworkPolicy by name.

get_pod(config, name)

@spec get_pod(config(), String.t()) :: {:ok, map()} | {:error, term()}

GET a Pod by name.

kubeconfig(config)

@spec kubeconfig(config()) :: Kubereq.Kubeconfig.t()

The loaded kubeconfig for config.

Accepts a %Kubereq.Kubeconfig{} struct, a pipeline module, or a {module, opts} tuple under :kubeconfig; defaults to Kubereq.Kubeconfig.Default, which covers both a developer's ~/.kube/config and an in-cluster ServiceAccount with no caller input.

list_all(config, namespace \\ nil, opts \\ [])

@spec list_all(config(), String.t() | nil, keyword()) ::
  {:ok, [map()]} | {:error, term()}

Every Pod matching label_selectors, following continue to the last page.

Never Kubereq.list/3, and never its into: :stream form. Both are wrong here, in the same silent direction:

  • the plain form returns one page and no indication that there were more;
  • do_list_into_stream/4 does Keyword.put_new(params, :limit, 10) and its stream {:halt, :ok}s on a mid-pagination error, so a partial list is indistinguishable from a complete one.

A short list is not a cosmetic bug. CrowdControl.Reaper reads this as the evidence of what is live: under the reconciliation table, a live sandbox missing from this list is live? = no, stored? = yes, and the reaper deletes the store record of a running, billed sandbox — orphaning it permanently. Truncation must therefore be impossible, and any page failure must surface as {:error, _} rather than a shorter list.

logs(config, pod_name, opts \\ [])

@spec logs(config(), String.t(), keyword()) :: {:ok, binary()} | {:error, term()}

The container's logs, as a single binary.

This is the diagnostic channel the backend had none of. A sandbox that dies during provisioning previously produced {:k8s, {:pod_not_ready, "CrashLoopBackOff"}} and nothing else — the operator's next step was kubectl logs by hand, which is only possible if the Pod still exists, and destroy/1 has usually removed it by then.

Bounded by construction, because a log fetch is a diagnostic and must never become the reason a teardown hangs:

  • follow: falsealways, never overridable. Kubereq.logs/4's own docs say follow: true "keeps the connection alive which blocks the current process"; that is Kubereq.PodLogs' job, not this one.
  • tailLines (default 50) and limitBytes (default 65536) so a chatty container cannot return a megabyte into a crash report.
  • :previous for the case that matters most — a container that already restarted, whose current logs are empty precisely because the interesting run is the previous one.

Built on Kubereq.PodLogs, not Kubereq.logs/4. The latter looks like the obvious call and is a trap three ways: its body is a lazy Stream, so a rejected upgrade raises at Enum time rather than at the call and escapes the guard around it; enumerating it can raise WithClauseError from Kubereq.Connect.create_stream/4, whose message inspects a %Mint.HTTP1{}; and its :follow defaults to true, so the obvious call blocks forever.

Returns {:ok, ""} when a container has genuinely produced nothing, so a caller can tell "nothing to say" from "could not ask". A Pod whose container never started answers 400 rather than an empty body — there is nothing to read — so CrowdControl.Backend.Kubernetes falls back to the Pod's own state.waiting.message for that case.

namespace(config)

@spec namespace(config()) :: String.t()

The namespace to operate in.

:namespace, else the current kubeconfig context's own namespace, else "default". Requiring the option would be friction on the dev path for no safety gain: the context namespace is what kubectl would use, and in-cluster Kubereq.Kubeconfig.ServiceAccount populates it from the projected namespace file. Both topologies land on the right answer unasked.

open_exec(config, pod_name, command, into, opts \\ [])

@spec open_exec(config(), String.t(), [String.t()], pid(), keyword()) ::
  {:ok, pid()} | {:error, term()}

Start a long-lived Kubereq.PodExec delivering frames to into.

The channel is not linked to the caller. Two kubereq 0.4.4 hazards make that the only safe contract, and both are handled here and nowhere else:

  • On a failed websocket upgrade Kubereq.Connect's init/1 raises a WithClauseError, GenServer.start_link/3 returns {:error, _}, and Kubereq.Connect's own {:ok, pid} = … turns that into a MatchError in the starting process. A rescue only reaches that if the starter traps exits: otherwise proc_lib's sync_wait never converts the child's abnormal exit into a value and the link signal kills the starter outright, before any rescue can run. (Written unlinked on purpose: that module is @moduledoc false, so an autolink to it is a broken doc reference.)
  • The returned process stops with the transport error as its exit reason, so a routine websocket blip would kill a linked caller mid-session.

So the channel is started by a dedicated owner process, spawned unlinked, which traps exits and holds the only link. The caller gets a plain value back and cannot be killed by either hazard. When the channel dies the owner sends into a {:exec_down, pid, reason} message, which carries the same information the old {:EXIT, pid, reason} did — a consumer that wants to know still learns immediately rather than at its next poll.

The owner monitors into and closes the channel if it dies, so the channel cannot outlive its consumer. It is deliberately not supervised: an exec channel has no meaningful restart, and a supervisor would keep one alive after its reader was gone.

pod_events(config, pod_name)

@spec pod_events(config(), String.t()) :: {:ok, [map()]}

Warning-class events for one Pod.

The only place some failures exist. A Pod that never started has no logs, and often no container waiting.message either — FailedCreatePodSandBox: RuntimeHandler "gvisor" not supported is reported only as an event, so without this a :runtime_class pointing at a runtime the nodes do not have reads as a bare :provision_timeout.

This is diagnosis: the caller already holds a failure, so an unreadable event stream returns {:ok, []} rather than replacing one error with another.

wait_until(config, name, callback, timeout)

@spec wait_until(
  config(),
  String.t(),
  (map() | :deleted -> boolean() | {:error, term()}),
  timeout()
) :: :ok | {:error, term()}

Poll/watch a Pod until callback returns true, or :timeout ms elapse.

Kubereq.wait_until/5's :timeout is a Req receive_timeout on the watch, not a wall-clock deadline — the caller supplies that.