Reconciles live sandboxes against stored session records.

Session.terminate/2 is best-effort and never runs on SIGKILL, a VM crash, or a hard container stop. For a local subprocess that does not matter much — the OS reaps it. For a billed remote sandbox it matters a great deal: the container keeps running and keeps costing money with nothing left to stop it. The reaper is the only real guarantee.

Reconciliation

For each configured backend, list_live/1 is compared against Store.all/0:

Runs once at boot and then every :sweep_interval (default 5 minutes).

Fail-open, always

A backend whose list_live/1 returns an error is skipped with a warning. It is never treated as "nothing is live". That misreading is the single most dangerous bug available here: one unreachable daemon would make every running sandbox look like a stale record, and the sweep would delete the lot. Every destructive branch requires positive evidence.

Two-node safety

Every sandbox carries its owner (CrowdControl.Store.owner_id/0, default to_string(node())), list_live/1 filters on it, and the reaper only ever destroys sandboxes matching its own owner. Two nodes with independent stores therefore cannot reap each other's work. Callers sharing one backend across nodes with a shared store (Ecto, Redis) should set a single shared :owner_id — the owner stamp is the coordination primitive either way.

CrowdControl.Backend.Docker stamps it as a crowd_control.owner label. CrowdControl.Backend.Kubernetes cannot: nonode@nohost is not a legal Kubernetes label value, and sanitizing it is lossy in exactly the way that lets one node's reaper destroy another's Pods. So it puts the raw owner in a crowd_control.owner annotation, whose values are unconstrained, and a sha256 prefix in a crowd_control.owner_hash label for the server-side selector. owned_by?/3's local re-check still compares raw owners exactly, because list_live/1 rebuilds each handle's owner from the annotation.

A :reap_grace_ms window (default 60s), measured against the crowd_control.created_at label, protects a container created by a node that has not yet written its store record from being destroyed mid-provision.

Configuration

config :crowd_control,
  reaper: [
    backends: [{CrowdControl.Backend.Docker, image: "my-cli:latest"}],
    sweep_interval: :timer.minutes(5),
    reap_grace_ms: 60_000,
    reattach: true
  ]

With no :backends configured the reaper starts and does nothing — there is no remote state to reconcile, which is the correct default for the local backend.