# CrowdControl
[](https://github.com/nikoma/crowd_control/actions/workflows/ci.yml)
[](https://hex.pm/packages/crowd_control)
[](https://hexdocs.pm/crowd_control)
[](https://github.com/nyo16/CrowdControl/blob/master/LICENSE)
Orchestrate many [Claude Code](https://github.com/anthropics/claude-code) / [Open Code](https://github.com/anthropics/open-code) / [omp](https://omp.sh/) CLI instances in parallel from Elixir.
Built on [net_runner](https://hex.pm/packages/net_runner) for zero-zombie subprocess management with NIF-based backpressure. Security disclosures: see [`SECURITY.md`](SECURITY.md). Contributing guide: [`CONTRIBUTING.md`](CONTRIBUTING.md). Runnable examples: [`examples/`](https://github.com/nyo16/CrowdControl/tree/master/examples).
**How it works**, beyond the option tables below — architecture, sandbox internals, providers and
day-two operations:
| Guide | Answers |
|-------|---------|
| [`docs/architecture.md`](docs/architecture.md) | What the four layers are, what each is forbidden to know, and how a prompt travels end to end |
| [`docs/sandboxes.md`](docs/sandboxes.md) | How a sandbox actually works: the FIFO and tee file, byte-exact resume, why PID 1 relays the CLI's exit status, and the HTTP-agent alternative |
| [`docs/providers.md`](docs/providers.md) | The three substrates a sandbox can live on — container, Compose stack, GCE spot VM — and what each costs you |
| [`docs/operations.md`](docs/operations.md) | Store and reaper, reattach across a node restart, log redaction, and what to alarm on |
```mermaid
graph LR
CC[CrowdControl] --> S1[claude #1]
CC --> S2[claude #2]
CC --> S3[omp #3]
CC --> SN[... #N]
S1 --> R[Collect Results]
S2 --> R
S3 --> R
SN --> R
style CC fill:#4a9eff,color:#fff
style S1 fill:#51cf66,color:#fff
style S2 fill:#51cf66,color:#fff
style S3 fill:#ff922b,color:#fff
style SN fill:#868e96,color:#fff
style R fill:#ffd43b,color:#333
```
## Features
- Run N Claude Code / Open Code / omp sessions in parallel
- Fan-out the same prompt across different models
- Multi-turn conversations with subscriber-based message delivery
- Fault-isolated sessions via OTP DynamicSupervisor
- Zero zombie OS processes guaranteed by net_runner's Shepherd
- Docker support with project directory mounting
- Sandbox each session in a container, a Compose stack or a GCE spot VM — one transport, pluggable [providers](#sandbox-providers)
- Works with the `claude`, `open-code` and `omp` CLIs — one message contract for all three
- **Security hardened**: non-root Docker container, API key protection, capability dropping, resource limits
- Configurable session concurrency limits and timeouts
- Input validation and structured logging
## Installation
### As a dependency
```elixir
def deps do
[
{:crowd_control, "~> 0.1.1"}
]
end
```
### From source
```bash
git clone git@github.com:nyo16/CrowdControl.git
cd CrowdControl
mix deps.get
mix compile
```
## Quick Start
### Single session
```elixir
result = CrowdControl.run("Explain GenServer in one sentence",
permission_mode: "bypassPermissions"
)
# => {:result, "success", %{"result" => "GenServer is...", "total_cost_usd" => 0.003}}
```
### Parallel sessions
```elixir
# Run 5 sessions with different prompts
opts_list = Enum.map(1..5, fn i ->
[prompt: "Write a haiku about the number #{i}", permission_mode: "bypassPermissions"]
end)
{:ok, sessions} = CrowdControl.start_sessions(opts_list)
results = CrowdControl.collect(sessions, 120_000)
CrowdControl.stop_all(sessions)
# results is [{session_pid, {:result, "success", %{...}}}, ...]
```
### Same prompt, different models
```elixir
results = CrowdControl.run_many("What is the meaning of life?", [
[model: "sonnet", permission_mode: "bypassPermissions"],
[model: "opus", permission_mode: "bypassPermissions"],
[model: "haiku", permission_mode: "bypassPermissions"]
])
```
### Multi-turn conversation
```elixir
{:ok, session} = CrowdControl.start_session(
prompt: "You are a code reviewer. Wait for code to review.",
permission_mode: "bypassPermissions"
)
CrowdControl.Session.subscribe(session)
# Wait for initial result, then send follow-up
receive do
{:crowd_control, ^session, {:result, _, _}} -> :ok
end
CrowdControl.Session.send_prompt(session, "Review this: def add(a, b), do: a + b")
receive do
{:crowd_control, ^session, {:result, _, %{"result" => review}}} ->
IO.puts(review)
end
CrowdControl.Session.stop(session)
```
### Subscribing to streaming messages
```elixir
{:ok, session} = CrowdControl.start_session(
prompt: "Write a short poem",
permission_mode: "bypassPermissions",
include_partial_messages: true
)
CrowdControl.Session.subscribe(session)
# Receive all messages as they stream
defmodule Listener do
def loop do
receive do
{:crowd_control, _pid, {:system_init, %{"session_id" => sid}}} ->
IO.puts("Session started: #{sid}")
loop()
{:crowd_control, _pid, {:assistant, %{"message" => msg}}} ->
IO.puts("Assistant: #{inspect(msg["content"])}")
loop()
{:crowd_control, _pid, {:result, "success", %{"result" => text}}} ->
IO.puts("Done: #{text}")
{:crowd_control, _pid, {:exit, status}} ->
IO.puts("Exited with status: #{status}")
after
30_000 -> IO.puts("Timeout")
end
end
end
Listener.loop()
```
### Using Open Code
```elixir
# Single session with open-code
CrowdControl.run("Hello", executable: "open-code", permission_mode: "bypassPermissions")
# Mix claude and open-code in the same parallel run
CrowdControl.run_many("Explain recursion", [
[executable: "claude", model: "sonnet", permission_mode: "bypassPermissions"],
[executable: "open-code", permission_mode: "bypassPermissions"]
])
```
### Using omp (Oh My Pi)
[omp](https://omp.sh/) speaks its own JSON-RPC protocol over stdio, so it gets
its own adapter (`CrowdControl.Agent.Omp`). Select it with `agent: :omp` — or
just name the binary, which infers the adapter:
```elixir
# Single omp session (agent inferred from the executable)
CrowdControl.run("Hello", executable: "omp", permission_mode: "bypassPermissions")
# Explicit adapter, omp-native options
CrowdControl.run("Summarize this repo",
agent: :omp,
model: "anthropic/claude-haiku-4-5",
approval_mode: "yolo",
thinking: "off",
no_session_persistence: true
)
# Fan the same prompt across all three CLIs
CrowdControl.run_many("Explain recursion", [
[executable: "claude", model: "sonnet", permission_mode: "bypassPermissions"],
[executable: "open-code", permission_mode: "bypassPermissions"],
[agent: :omp, permission_mode: "bypassPermissions"]
])
```
The messages are the same for every agent: `{:system_init, %{"session_id" => id}}`
once the CLI is up, `{:assistant, _}` / `{:stream_event, _}` while it works, and
`{:result, "success", %{"result" => text, "total_cost_usd" => cost}}` at the end
of a turn. `CrowdControl.collect/2` therefore works across a mixed fan-out.
Behind the scenes the adapter runs `omp --mode rpc`, asks for `get_state` to
surface the session id, and treats a terminal `agent_end` frame as the end of a
turn. Claude Code's `:permission_mode` is translated to omp's approval modes
(`"bypassPermissions"` => `yolo`, `"acceptEdits"` => `write`, `"default"` =>
`always-ask`); Claude-Code-only options such as `:mcp_config` or
`:max_budget_usd` raise instead of being silently dropped. See
`CrowdControl.Agent.Omp` for the full option list.
### Custom providers: vLLM, LiteLLM, any OpenAI-compatible endpoint
omp reads a provider's `baseUrl` from `models.yml` under its agent directory —
there is no CLI flag for it. `:custom_provider` renders that file into a
private `0700` temp directory and points the session at it:
```elixir
# Self-hosted vLLM, no auth. Models are discovered from /v1/models,
# including vLLM's max_model_len as the context window.
CrowdControl.run("Review this diff",
agent: :omp,
custom_provider: [base_url: "http://10.0.0.5:8000/v1"],
model: "vllm/Qwen/Qwen3-Coder-30B",
approval_mode: "yolo"
)
# Authenticated endpoint. The key is passed through the same 0600 env-file
# channel as every other credential — it is never written into models.yml
# and never appears in argv or `ps`.
CrowdControl.run("Review this diff",
agent: :omp,
custom_provider: [
id: "my-gateway",
base_url: "https://gateway.internal/v1",
api_key: System.fetch_env!("GATEWAY_KEY")
],
model: "my-gateway/qwen3-coder"
)
# A server without /v1/models: declare the models yourself.
CrowdControl.run("Review this diff",
agent: :omp,
custom_provider: [
base_url: "http://10.0.0.5:8000/v1",
models: [[id: "Qwen/Qwen3-Coder-30B", context_window: 262_144, max_tokens: 65_536]]
],
model: "vllm/Qwen/Qwen3-Coder-30B"
)
```
Fan out across several vLLM hosts, or mix local and hosted models in one run:
```elixir
CrowdControl.run_many("Explain recursion", [
[agent: :omp, custom_provider: [base_url: "http://gpu-a:8000/v1"], model: "vllm/qwen3-coder"],
[agent: :omp, custom_provider: [base_url: "http://gpu-b:8000/v1"], model: "vllm/llama-3.3-70b"],
[agent: :omp, model: "anthropic/claude-haiku-4-5"]
])
```
Spec keys: `:base_url` (required), `:id` (default `"vllm"`, and the `provider/`
prefix in `:model`), `:api` (`"openai-completions"`, `"openai-responses"`, or
`"anthropic-messages"`), `:api_key`, `:api_key_env`, `:models`, `:headers`.
The generated directory is content-addressed, so every session in a fan-out
sharing one spec shares one directory. Build it yourself with
`CrowdControl.Agent.Omp.provider_dir!/1` and pass `:agent_dir` to own the
lifecycle, and `remove_provider_dir/1` to delete it.
> **`PI_CODING_AGENT_DIR` relocates more than `models.yml`.** It moves the whole
> `~/.omp/agent` base for that session — `config.yml`, the auth store, saved
> sessions — so a custom-provider session starts with none of your global omp
> settings and none of your stored logins. Usually what you want for an isolated
> endpoint; add `inherit_auth: true` when you want the logins back (see
> [Authentication](#authentication)). `~/.omp` itself (skills, plugins) is
> unaffected. For the Docker and Kubernetes backends the directory must exist
> *inside* the sandbox: mount your own and pass `:agent_dir`.
### Tuning prefill against a self-hosted endpoint
A coding agent sends its whole system prompt and tool schema on every turn. On a
hosted provider that is absorbed by prompt caching and priced at a discount; on
your own GPU it is prefill you pay for in latency.
Both CLIs default to a large prompt, and both can be trimmed. Measured against
one vLLM box (`deepseek-v4-flash`), three runs per row, prompt tokens and
request count read from vLLM's own `/metrics`:
| Configuration | Prompt tokens | Requests/turn | Wall time |
|---|---|---|---|
| `agent: :claude`, default | 28,631 | 1 | ~2.0s |
| `agent: :claude` + `CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC=1` | **23,850** (−17%) | 1 | ~1.7s |
| `agent: :omp`, default | 20,641 | 1, **sometimes 2** | 1.8–2.6s |
| `agent: :omp` + `no_extensions: true` | 20,641 | **1** | ~1.2s |
| `agent: :omp` + `no_extensions`, `no_skills`, `no_rules` | **17,072** (−17%) | 1 | ~1.2s |
```elixir
# omp, trimmed
CrowdControl.run("…", agent: :omp,
custom_provider: [base_url: "http://10.0.0.5:8000/v1"],
model: "vllm/…",
no_extensions: true, no_skills: true, no_rules: true)
# Claude Code, trimmed
CrowdControl.run("…", api_url: "http://10.0.0.5:8000", auth_token: key,
model: "…", env: %{"CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC" => "1"})
```
Two things worth knowing before you go hunting further:
- **Prefix caching is not the problem.** Both agents sat at a **99% vLLM
prefix-cache hit rate** out of the box, across separate sessions. Nothing in
either CLI's default configuration defeats it.
- **`DISABLE_PROMPT_CACHING=1` does nothing here.** It controls the Anthropic
`cache_control` breakpoints in the request body; vLLM ignores those and does
its own automatic, content-addressed prefix caching. Measured identical
(99.2%, 28,631 tokens) with and without it.
The omp "sometimes 2 requests" row is the one real trap: a discovered extension
fires a second full-size (~20k token) call on some turns, roughly doubling
prefill. `no_extensions: true` removes it and makes latency stable.
`no_skills`/`no_rules`/`no_extensions` trade capability for prefill — drop them
if a session actually needs those. Numbers are from one model on one box; re-measure
with `curl $BASE/metrics | grep prefix_cache` on yours.
### Working with project directories
```elixir
# Point sessions at a specific project
CrowdControl.run("Find and fix the failing test",
add_dir: "/path/to/my/project",
permission_mode: "bypassPermissions",
allowed_tools: ["Read", "Edit", "Bash"]
)
# Fan out across multiple repos
repos = [
"/home/user/api-service",
"/home/user/web-frontend",
"/home/user/mobile-app"
]
opts_list = Enum.map(repos, fn repo ->
[
prompt: "List all TODO comments in the codebase",
add_dir: repo,
permission_mode: "bypassPermissions"
]
end)
{:ok, sessions} = CrowdControl.start_sessions(opts_list)
results = CrowdControl.collect(sessions)
```
### Custom API URL and token
```elixir
# Use a custom API key per session
CrowdControl.run("Hello",
api_key: "sk-ant-your-key-here",
permission_mode: "bypassPermissions"
)
# Point to a custom API endpoint (proxy, gateway, self-hosted)
CrowdControl.run("Hello",
api_url: "https://your-proxy.example.com/v1",
api_key: "sk-ant-your-key-here",
permission_mode: "bypassPermissions"
)
# Different keys per session (e.g. separate billing)
CrowdControl.run_many("Summarize this repo", [
[api_key: "sk-team-alpha", model: "sonnet"],
[api_key: "sk-team-beta", model: "opus"]
])
# Arbitrary environment variables
CrowdControl.run("Debug this",
env: %{
"ANTHROPIC_API_KEY" => "sk-custom",
"ANTHROPIC_BASE_URL" => "https://gateway.internal/v1",
"HTTP_PROXY" => "http://proxy:8080",
"NODE_OPTIONS" => "--max-old-space-size=4096"
},
permission_mode: "bypassPermissions"
)
```
### Settings and config files
```elixir
# Load a custom settings JSON file
CrowdControl.run("Fix the tests",
settings: "/config/claude_settings.json",
add_dir: "/workspace",
permission_mode: "bypassPermissions"
)
# Inline settings as JSON string
CrowdControl.run("Audit this code",
settings: ~s({"permissions":{"allow":["Read","Bash(git:*)"],"deny":["Edit","Write"]}}),
add_dir: "/workspace"
)
# Load MCP server configs
CrowdControl.run("Search the GitHub repo for issues",
mcp_config: "/config/mcp_servers.json",
strict_mcp_config: true,
permission_mode: "bypassPermissions"
)
# Load multiple MCP configs
CrowdControl.run("Analyze the project",
mcp_config: ["/config/mcp_filesystem.json", "/config/mcp_github.json"],
permission_mode: "bypassPermissions"
)
# Custom agents per session
CrowdControl.run_many("Review this PR", [
[
agents: %{
"security" => %{
"description" => "Security reviewer",
"prompt" => "Focus only on security vulnerabilities and data leaks."
}
},
add_dir: "/workspace"
],
[
agents: %{
"perf" => %{
"description" => "Performance reviewer",
"prompt" => "Focus only on performance bottlenecks and N+1 queries."
}
},
add_dir: "/workspace"
]
])
# Bare mode (skip hooks, LSP, plugins, auto-memory)
CrowdControl.run("Quick analysis",
bare: true,
settings: "/config/claude_settings_permissive.json",
add_dir: "/workspace",
permission_mode: "bypassPermissions"
)
# Control which setting sources are loaded
CrowdControl.run("Check the code",
setting_sources: ["user", "project"],
add_dir: "/workspace"
)
# Load plugins from a directory
CrowdControl.run("Process this",
plugin_dir: "/plugins/my-plugin",
permission_mode: "bypassPermissions"
)
```
### Budget control
```elixir
# Cap spending per session
CrowdControl.run_many("Refactor this module for clarity", [
[add_dir: "/path/to/project", model: "sonnet", max_budget_usd: 0.50],
[add_dir: "/path/to/project", model: "opus", max_budget_usd: 2.00]
])
```
### Resource limits
Each session bounds its own memory and lifetime so a misbehaving CLI can't
exhaust the host. All limits are per-session options with safe defaults:
```elixir
CrowdControl.start_session(
# ceiling on a single TURN. Armed at start, re-armed by send_prompt/2 --
# not by output, so a turn that streams for longer is still killed
# mid-flight. Size it against the slowest turn, not the conversation.
timeout: 300_000,
# reject prompts larger than this with {:error, :prompt_too_large}
max_prompt_size: 1_000_000,
# a single newline-free output line over this kills the subprocess and
# broadcasts {:error, :line_too_large} instead of buffering unbounded.
# Defaults to 1 MiB, matching the maxFrameBytes omp advertises.
max_line_bytes: 1_048_576,
# cap on messages kept for get_messages/1 (oldest dropped past the cap);
# live subscribers still receive every message
max_messages: 10_000
)
```
## Authentication
Every credential below reaches the CLI through the session's **environment**, never
through argv: a `0600` env file that is sourced and deleted before the CLI starts
(`Backend.Local`), or the exec `Env` array (Docker, Kubernetes). Nothing shows up in
`ps`.
### Which option do I pass?
| You have | `agent: :claude` / `:open_code` | `agent: :omp` |
|---|---|---|
| **API key** (pay-per-use) | `api_key: "sk-ant-…"` → `ANTHROPIC_API_KEY` | same |
| **Subscription**, headless (Pro/Max/Team) | `oauth_token: "…"` from `claude setup-token` → `CLAUDE_CODE_OAUTH_TOKEN` | `oauth_token: "…"` → `ANTHROPIC_OAUTH_TOKEN` |
| **Subscription**, already logged in on this host | `env: %{"CLAUDE_CONFIG_DIR" => "/home/me/.claude"}` | nothing — omp reads `~/.omp/agent/agent.db` automatically |
| **Self-hosted endpoint** (vLLM, SGLang, LiteLLM) | `api_url:` + `auth_token:` — **only if it serves `/v1/messages`** | `custom_provider: [base_url: …, api_key: …]` |
| **Another hosted provider** (OpenAI, Groq, …) | only through an Anthropic-compatible gateway | `env: %{"OPENAI_API_KEY" => …}` |
| **Gateway / proxy** | `api_url:` + (`auth_token:` for Bearer, `api_key:` for `x-api-key`) | `custom_provider: [base_url: …, api_key: …]` |
The difference in that first row is the wire protocol, not the vendor. **Claude
Code speaks only the Anthropic Messages API**, so an endpoint has to serve
`/v1/messages`; an OpenAI-only `/v1/chat/completions` server will not work with
it. **omp speaks OpenAI-compatible** through `:custom_provider` (and
`api: "anthropic-messages"` when you want the other shape). Recent vLLM builds
serve both, so either agent can drive them:
```bash
# does this endpoint speak Anthropic?
curl -sS $BASE/v1/messages -H "Authorization: Bearer $KEY" \
-H 'content-type: application/json' \
-d '{"model":"…","max_tokens":16,"messages":[{"role":"user","content":"hi"}]}'
# a body with "type": "message" => Claude Code can drive it
```
```elixir
# Claude Code against a self-hosted Anthropic-compatible endpoint
CrowdControl.run("Summarize this repo",
api_url: "http://10.0.0.5:8000", # no /v1 -- the CLI appends it
auth_token: System.fetch_env!("VLLM_KEY"),
model: "deepseek-v4-flash"
)
# omp against the OpenAI-compatible side of the same server
CrowdControl.run("Summarize this repo",
agent: :omp,
custom_provider: [base_url: "http://10.0.0.5:8000/v1", api_key: System.fetch_env!("VLLM_KEY")],
model: "vllm/deepseek-v4-flash"
)
```
`:api_key` sends `x-api-key`, `:auth_token` sends `Authorization: Bearer` — most
self-hosted servers want the latter. `"total_cost_usd"` is meaningless on a
self-hosted model: Claude Code prices it against Anthropic's table, omp reports
`0.0`. For a hosted provider under omp that is not Anthropic, use `:env` with the
variable that provider documents.
### Subscription passthrough, in one line each
```elixir
# omp, host already logged in: nothing to pass.
CrowdControl.run("Explain this repo", agent: :omp)
# omp, headless (CI, container, another user):
CrowdControl.run("Explain this repo", agent: :omp, oauth_token: System.fetch_env!("OMP_OAUTH"))
# Claude Code, headless: token from `claude setup-token`
CrowdControl.run("Explain this repo", oauth_token: System.fetch_env!("CLAUDE_OAUTH"))
# Claude Code, inherit an existing ~/.claude login
CrowdControl.run("Explain this repo", env: %{"CLAUDE_CONFIG_DIR" => "/home/me/.claude"})
```
A subscription and a self-hosted endpoint can coexist in **one** omp session. The
catch: `:custom_provider` relocates omp's agent directory, which is where the stored
login lives, so opt back in with `inherit_auth: true`:
```elixir
CrowdControl.run("Compare these two approaches",
agent: :omp,
custom_provider: [base_url: "http://10.0.0.5:8000/v1", inherit_auth: true],
model: "anthropic/claude-haiku-4-5" # or "vllm/…" — both resolve
)
```
> `inherit_auth` symlinks your OAuth store into a directory the session's own `bash`
> tool can read, while that session may be talking to a third-party endpoint. It is
> off by default for that reason. Turn it on for endpoints you trust.
### Claude Code auth methods
```mermaid
graph LR
subgraph "Auth Methods"
A[API Key
ANTHROPIC_API_KEY]
B[Subscription Token
claude setup-token]
C[OAuth Session
~/.claude/]
D[Console API
claude auth login --console]
end
A --> CC[CrowdControl]
B --> CC
C --> CC
D --> CC
style A fill:#51cf66,color:#fff
style B fill:#4a9eff,color:#fff
style C fill:#ff922b,color:#fff
style D fill:#cc5de8,color:#fff
```
### API key (pay-per-use)
The simplest method. Set per-session or via environment:
```elixir
# Via environment variable (all sessions inherit)
# export ANTHROPIC_API_KEY=sk-ant-...
# Or per-session
CrowdControl.run("Hello", api_key: "sk-ant-your-key-here")
# Different keys per session (separate billing)
CrowdControl.run_many("Summarize this", [
[api_key: "sk-team-alpha"],
[api_key: "sk-team-beta"]
])
```
### Claude subscription (Pro/Max/Team)
For subscription-based billing without an API key. Two options:
**Option 1: Interactive login (local machine)**
```bash
# Login once — stores OAuth token in ~/.claude/
claude auth login
```
Then pass your `~/.claude` directory so sessions pick up the token:
```elixir
# Sessions inherit the subscription auth from ~/.claude
CrowdControl.run("Hello",
env: %{"CLAUDE_CONFIG_DIR" => "/home/crowdctl/.claude"},
permission_mode: "bypassPermissions"
)
```
**Option 2: Setup token (headless / Docker / CI)**
```bash
# Generate a long-lived token (requires Claude subscription)
claude setup-token
# Paste the token when prompted — stored in ~/.claude/
```
For Docker, mount `~/.claude` into the container:
```bash
docker run -it \
-v ~/.claude:/home/crowdctl/.claude \
-v "$(pwd)":/workspace \
crowd_control
```
### Console API (Anthropic Console billing)
```bash
claude auth login --console
```
Works the same as subscription — mount `~/.claude/` to share the session.
### Auth in Docker
```mermaid
graph TD
subgraph Host
KEY["~/.claude/
OAuth tokens"]
API["ANTHROPIC_API_KEY"]
CFG["/my/config/
settings.json"]
end
subgraph Docker Container
ROOT["/home/crowdctl/.claude/"]
ENV["$ANTHROPIC_API_KEY"]
CONF["/config/"]
CC[CrowdControl]
end
KEY -->|"-v mount"| ROOT
API -->|"-e env"| ENV
CFG -->|"-v mount"| CONF
ROOT --> CC
ENV --> CC
CONF --> CC
style CC fill:#4a9eff,color:#fff
style KEY fill:#ff922b,color:#fff
style API fill:#51cf66,color:#fff
```
| Method | Docker flag | Notes |
|--------|-------------|-------|
| API key | `-e ANTHROPIC_API_KEY=sk-...` | Simplest, pay-per-use |
| Subscription | `-v ~/.claude:/home/crowdctl/.claude` | Mount OAuth tokens |
| Setup token | `-v ~/.claude:/home/crowdctl/.claude` | For CI/headless |
| Per-session key | Set via `:api_key` option in IEx | Different keys per session |
## Docker
Run CrowdControl in a hardened Linux container with your project directory and config mounted in.
The container is security-hardened out of the box:
- **Non-root user**: runs as `crowdctl` (not root)
- **Capabilities dropped**: all Linux capabilities removed (`cap_drop: ALL`)
- **No privilege escalation**: `no-new-privileges` security option
- **Read-only filesystem**: root FS is read-only with tmpfs for `/tmp`, `~/.claude`, `~/.npm`
- **Resource limits**: memory and CPU limits configured in docker-compose
- **Health check**: built-in `HEALTHCHECK` via `CrowdControl.healthy?/0`
- **Pinnable CLI version**: `CLAUDE_CODE_VERSION` build arg for reproducible builds
### Build the image
```bash
# Build with latest Claude Code CLI
docker build -t crowd_control .
# Pin a specific Claude Code version
docker build --build-arg CLAUDE_CODE_VERSION=1.0.16 -t crowd_control .
```
### Run with API key
```bash
docker run -it \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v "$(pwd)":/workspace \
crowd_control
```
### Run with subscription auth
```bash
docker run -it \
-v ~/.claude:/home/crowdctl/.claude \
-v "$(pwd)":/workspace \
crowd_control
```
### Mount custom config files
```bash
docker run -it \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v "$(pwd)":/workspace \
-v /path/to/my/configs:/config \
crowd_control
```
The container has built-in template configs at `/config/templates/`:
| Template | Description |
|----------|-------------|
| `claude_settings.json` | Default settings with common tool permissions |
| `claude_settings_permissive.json` | Permissive settings (all tools allowed) |
| `mcp_servers.json` | MCP server config (filesystem + GitHub) |
| `agents.json` | Pre-built agents (reviewer, refactorer, documenter, tester) |
| `opencode_settings.json` | Open Code provider/tool configuration |
Use them directly or as a starting point:
```elixir
# Use built-in template
CrowdControl.run("Review this",
settings: "/config/templates/claude_settings_permissive.json",
add_dir: "/workspace"
)
# Use your own mounted config
CrowdControl.run("Analyze with MCP tools",
settings: "/config/my_settings.json",
mcp_config: "/config/my_mcp.json",
add_dir: "/workspace"
)
```
### Mount everything (full setup)
```bash
docker run -it \
-v ~/.claude:/home/crowdctl/.claude \
-v "$(pwd)":/workspace \
-v /path/to/configs:/config \
-v /path/to/plugins:/plugins \
crowd_control
```
```elixir
# Full-featured session with subscription auth, config, MCP, and plugins
CrowdControl.run("Deep code review",
settings: "/config/claude_settings.json",
mcp_config: "/config/mcp_servers.json",
plugin_dir: "/plugins/my-plugin",
add_dir: "/workspace",
permission_mode: "bypassPermissions"
)
```
This drops you into an IEx shell. Your project is available at `/workspace`:
```elixir
CrowdControl.run("Analyze the code in /workspace",
add_dir: "/workspace",
permission_mode: "bypassPermissions"
)
```
### Custom API endpoint
```bash
docker run -it \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-e ANTHROPIC_BASE_URL=https://your-proxy.example.com/v1 \
-v "$(pwd)":/workspace \
crowd_control
```
Or set per-session inside IEx:
```elixir
# Each session can target a different API endpoint
CrowdControl.run_many("Hello", [
[api_url: "https://gateway-us.internal/v1", api_key: "sk-us-key"],
[api_url: "https://gateway-eu.internal/v1", api_key: "sk-eu-key"]
])
```
### Mount a different directory
```bash
docker run -it \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v /path/to/your/project:/workspace \
crowd_control
```
### Mount multiple project directories
```bash
docker run -it \
-e ANTHROPIC_API_KEY=$ANTHROPIC_API_KEY \
-v /home/user/api:/projects/api \
-v /home/user/web:/projects/web \
-v /home/user/docs:/projects/docs \
crowd_control
```
Then inside IEx:
```elixir
# Scan all three repos in parallel
opts_list = [
[prompt: "Find security issues", add_dir: "/projects/api"],
[prompt: "Find security issues", add_dir: "/projects/web"],
[prompt: "Check for outdated links", add_dir: "/projects/docs"]
]
|> Enum.map(&Keyword.merge(&1, permission_mode: "bypassPermissions"))
{:ok, sessions} = CrowdControl.start_sessions(opts_list)
results = CrowdControl.collect(sessions, 300_000)
```
### Docker Compose
```bash
# Mount current directory
ANTHROPIC_API_KEY=sk-... docker compose run crowd_control
# Mount a specific project
ANTHROPIC_API_KEY=sk-... PROJECT_DIR=/path/to/project docker compose run crowd_control
# Run multiple isolated workers
ANTHROPIC_API_KEY=sk-... WORKER_COUNT=5 docker compose up worker
```
### Docker architecture
```mermaid
graph TD
subgraph Host
PD1[Project Dir A]
PD2[Project Dir B]
KEY[ANTHROPIC_API_KEY]
end
subgraph Docker Container
IEX[IEx Shell]
CC[CrowdControl]
CC --> S1[Session 1
claude CLI]
CC --> S2[Session 2
claude CLI]
CC --> S3[Session 3
open-code CLI]
WS1[/workspace]
WS2[/projects/api]
WS3[/projects/web]
end
PD1 -->|"-v mount"| WS1
PD1 -->|"-v mount"| WS2
PD2 -->|"-v mount"| WS3
KEY -->|"-e env"| CC
S1 --> WS2
S2 --> WS3
S3 --> WS1
style CC fill:#4a9eff,color:#fff
style S1 fill:#51cf66,color:#fff
style S2 fill:#51cf66,color:#fff
style S3 fill:#ff922b,color:#fff
```
## Sandbox Backends
> **Not the same as the "Docker" section above.** That section is about running
> *CrowdControl itself* inside a container. This section is about CrowdControl
> putting *each CLI session* inside its own container. They are independent —
> you can use either, both, or neither.
By default a session runs the CLI as a local subprocess. A `backend` option
swaps that for something else:
```elixir
# Default — a local subprocess, exactly as before.
CrowdControl.start_session(prompt: "hi")
# Each session gets its own container.
CrowdControl.start_session(
backend: {CrowdControl.Backend.Docker, image: "my-cli:latest"},
prompt: "hi"
)
```
Implement `CrowdControl.Backend` for anything else: nine required callbacks,
plus `scrub/1` — optional in the behaviour, but mandatory in practice for any
backend whose handle carries credentials — and `push_workspace/2` /
`pull_artifacts/2`, which no backend ships yet.
| Backend | Sandbox | Survives a VM restart? |
|---------|---------|------------------------|
| `Backend.Local` (default) | local subprocess | no — dies with the VM |
| `Backend.Docker` | container | yes — reattachable |
| `Backend.Kubernetes` | Pod, over the API server | yes — reattachable |
| `Backend.Sandboxd` | whatever its **provider** supplies — a container, a stack, or a VM | yes — reattachable |
`Backend.Sandboxd` is the odd one out: it is a transport with no substrate of
its own, and pairs with a `CrowdControl.Provider` that supplies one. If you want
a Compose stack or a cloud VM rather than a single container, that is the section
you want — see [Sandbox Providers](#sandbox-providers).
### Docker backend
Requires the optional `:req` dependency:
```elixir
{:req, "~> 0.5"}
```
```elixir
CrowdControl.start_session(
backend: {CrowdControl.Backend.Docker,
image: "my-cli:latest",
network_mode: "none",
cpus: 1.5,
memory: 512 * 1024 * 1024,
max_stream_bytes: 100 * 1024 * 1024
},
prompt: "Refactor lib/foo.ex"
)
```
| Option | Default | Notes |
|--------|---------|-------|
| `:image` | — | **Required.** Needs the CLI and `sh` on `PATH` |
| `:docker_host` | `unix:///var/run/docker.sock` | `tcp://host:port` also works |
| `:network_mode` | `"none"` | **Required** when `:proxy_url`/`:api_url` is set — see below |
| `:cpus` | unset | Fractional, e.g. `1.5` |
| `:memory` | unset | Bytes |
| `:tee_path` | `/var/log/cc/out.jsonl` | Where output is recorded |
| `:fifo_path` | `/var/run/cc.fifo` | Where prompts are written |
| `:max_stream_bytes` | `nil` (unbounded) | Destroys the sandbox past this much total output |
| `:max_inflight_bytes` | 4 MiB | Reader backpressure watermark |
| `:proxy_url`, `:session_token` | unset | Egress proxy — see [SECURITY.md](SECURITY.md#egress-proxy-contract) |
Hardening:
| Option | Default | Notes |
|--------|---------|-------|
| `:cap_drop` | `["ALL"]` | On by default; a CLI needs no capabilities |
| `:security_opt` | `["no-new-privileges:true"]` | On by default |
| `:pids_limit` | `512` | On by default — `:memory`/`:cpus` do **not** bound PIDs |
| `:user` | image default | Opt-in, e.g. `"1000:1000"`; recommended |
| `:readonly_rootfs` | `false` | Opt-in; `:tmpfs` keeps the fifo and tee writable |
The three capability settings default on because the sandbox runs untrusted
model-driven code and none of them breaks an ordinary CLI. `:user` and
`:readonly_rootfs` are opt-in because both genuinely break images that expect
root or write outside the tmpfs mounts — enable them where your image allows.
No custom image is required beyond having the CLI and `sh` available —
CrowdControl injects everything else it needs at provision time.
**Networking is never inferred.** Setting `:proxy_url` or `:api_url` without an
explicit `:network_mode` returns `{:error, {:docker, :network_mode_required}}`
rather than silently picking `bridge`, which would give the sandbox general
outbound access and make an egress proxy advisory rather than enforcing.
### Kubernetes backend
Requires the optional `:kubereq` dependency:
```elixir
{:kubereq, "~> 0.4.4"}
```
One Pod per session, driven over the API server. Session-facing behaviour is
indistinguishable from the Docker backend — same FIFO/tee I/O, same byte-exact
resume, same reader contract, `reattachable?/0 == true`.
```elixir
CrowdControl.start_session(
backend: {CrowdControl.Backend.Kubernetes,
image: "my-cli:latest",
namespace: "sandboxes",
network: :deny_all,
cpus: 1.5,
memory: 512 * 1024 * 1024
},
prompt: "Refactor lib/foo.ex"
)
```
| Option | Default | Notes |
|--------|---------|-------|
| `:image` | — | **Required.** Needs the CLI plus `sh`, `tail`, `tee` and `head` on `PATH` — busybox and coreutils both suffice |
| `:namespace` | kubeconfig context's namespace, else `"default"` | **Must already exist — the backend never creates one.** Creating namespaces needs cluster-scoped RBAC, which is a larger grant than running sandboxes justifies; see below |
| `:kubeconfig` | `Kubereq.Kubeconfig.Default` | A `%Kubereq.Kubeconfig{}`, a pipeline module, or `{module, opts}`. The default covers both a developer's `~/.kube/config` and an in-cluster ServiceAccount |
| `:network` | — | `:deny_all` \| `{:policy, name}` \| `:unrestricted`. **Required** when `:proxy_url`/`:api_url` is set — see below |
| `:network_probe` | `true` | `false` skips the `:deny_all` enforcement probe, for callers who already know their CNI enforces |
| `:network_probe_image` | `"busybox:1.36"` | Image the enforcement probe runs |
| `:network_probe_url` | unset | Probe *internet* egress instead of the default, which is a TCP connect to the API server's ClusterIP. The default needs no DNS and no internet, so a security decision does not depend on external reachability — see [SECURITY.md](SECURITY.md) |
| `:cpus` | unset | Fractional, e.g. `1.5` |
| `:memory` | unset | Bytes |
| `:tee_path` | `/var/log/cc/out.jsonl` | Where output is recorded |
| `:fifo_path` | `/var/run/cc.fifo` | Where prompts are written |
| `:env_path` | `/var/run/cc.env` | Env file, written over exec stdin and unlinked before the CLI starts |
| `:timeout` | 30s | HTTP receive timeout |
| `:exec_timeout` | 15s | Wall-clock bound on every short exec |
| `:provision_timeout` | 120s | Wall-clock bound on reaching `Running` |
| `:pod_poll_ms` | 60s | Reader's idle Pod-liveness poll |
| `:max_inflight_bytes` | 4 MiB | Reader backpressure watermark |
| `:proxy_url`, `:session_token` | unset | Egress proxy — see [SECURITY.md](SECURITY.md#egress-proxy-contract) |
Hardening:
| Option | Default | Notes |
|--------|---------|-------|
| `:cap_drop` | `["ALL"]` | On by default; the sandbox container adds nothing back |
| `:allow_privilege_escalation` | `false` | On by default |
| `:run_as_user` / `:run_as_group` | image default | Opt-in; also sets `runAsNonRoot` (for a non-zero uid) and the Pod's `fsGroup`, so the `emptyDir` volumes are writable |
| `:readonly_rootfs` | `false` | Opt-in. The fifo and tee directories are `emptyDir` volumes either way (the init container has to hand the FIFO across), so this is a pure toggle: it makes them in-memory (`medium: Memory`) and adds `/tmp` |
| `:volume_sizes` | `64Mi`, `/var/run` `8Mi` | `sizeLimit` per mount path; used only under `:readonly_rootfs` |
| `:runtime_class` | unset | `runtimeClassName` — `"gvisor"`, `"kata"`. The only option here that changes **which kernel** the container talks to; see below |
| `:node_selector` | unset | Map of node labels, e.g. `%{"sandbox.gke.io/runtime" => "gvisor"}` |
| `:tolerations` | unset | Raw toleration maps, for the tainted node pools sandbox runtimes run on |
`automountServiceAccountToken: false` and `enableServiceLinks: false` are not
options — a projected API token inside a sandbox running untrusted model-driven
code is a sandbox escape, and service links map the namespace into its
environment. Two Docker hardening measures also have **no Kubernetes
equivalent**: there is no `PidsLimit` (`podPidsLimit` is node-level kubelet
config) and `emptyDir` cannot be mounted `noexec,nosuid`. Both are stated in
[SECURITY.md](SECURITY.md#the-kubernetes-backend) rather than silently dropped;
read that before running this against untrusted output.
**Sandbox runtimes are the kernel boundary, and they are opt-in.** Everything in
the table above narrows what the container may *ask* of a kernel it still shares
with the host; one container escape defeats the lot together. A RuntimeClass
changes which kernel answers the syscall at all — gVisor's `runsc` handles them
in userspace, Kata gives the Pod its own VM:
```elixir
{CrowdControl.Backend.Kubernetes,
image: "my-cli:latest",
network: :deny_all,
runtime_class: "gvisor",
# A RuntimeClass alone is usually not enough: sandbox node pools are tainted,
# and a Pod that cannot tolerate the taint sits Pending until
# :provision_timeout. GKE Sandbox uses this pair.
node_selector: %{"sandbox.gke.io/runtime" => "gvisor"},
tolerations: [
%{"key" => "sandbox.gke.io/runtime", "operator" => "Equal",
"value" => "gvisor", "effect" => "NoSchedule"}
]}
```
It is never defaulted, because the RuntimeClass admission controller rejects a
Pod naming one the cluster does not have — measured, verbatim: `pods "cc-…" is
forbidden: pod rejected: RuntimeClass "gvisor" not found`. Guessing would turn
every cluster without gVisor into a provisioning failure. What you need on the
cluster side is a node pool running `runsc` or Kata and a matching `RuntimeClass`
object; GKE Sandbox, and the SIG-Apps
[Agent Sandbox](https://github.com/kubernetes-sigs/agent-sandbox) project, both
configure exactly this field.
If the RuntimeClass exists but the *nodes* do not implement its handler, the Pod
is admitted and never starts. That used to read as a bare
`{:k8s, :provision_timeout}`; the backend now reports the cluster's own reason —
`FailedCreatePodSandBox: … RuntimeHandler "gvisor" not supported` — because a
Pod that never started has no logs and no container `waiting` message, and the
explanation exists only in its events.
**Network posture is never inferred.** A Pod always has cluster networking —
there is no `NetworkMode: "none"` here — so `:network` is explicit, and omitting
it while setting `:proxy_url`/`:api_url` returns
`{:error, {:k8s, :network_policy_required}}`. Under `:deny_all` the backend also
*proves* the policy is enforced with a one-time per-cluster probe, and refuses
to provision with `{:error, {:k8s, :network_policy_not_enforced}}` if it is not.
Any API server accepts a NetworkPolicy object; only a CNI with a policy
controller acts on one.
Runnable examples, neither needing an API key or a custom image:
- [`examples/kubernetes_task.exs`](examples/kubernetes_task.exs) — fan out N
sandboxes as concurrent tasks, one Pod each, then ask the API server whether
anything leaked.
- [`examples/kubernetes_fanout.exs`](examples/kubernetes_fanout.exs) — the batch
shape: a queue of tasks with different parameters each, a max-concurrency
ceiling, and a report of every result at the end.
RBAC the backend's identity needs, in `:namespace`:
pods create, get, list, delete
pods/exec create
events list # only to explain a failed provision
networkpolicies create, get, delete # only under network: :deny_all
**Namespace isolation is yours to provision, deliberately.** The backend runs
every sandbox in `:namespace` and never creates, labels or deletes one: that
would need cluster-scoped `namespaces` RBAC, and a library that can create
namespaces can create them anywhere. Isolating a tenant is therefore a
namespace you make, with a `ResourceQuota` and `LimitRange` on it, plus the
`pod-security.kubernetes.io/enforce=restricted` label if you want the API
server enforcing what this backend already sets:
```bash
kubectl create namespace sandboxes
kubectl label namespace sandboxes pod-security.kubernetes.io/enforce=restricted
kubectl -n sandboxes create quota cc --hard=pods=50
```
Then pass `namespace: "sandboxes"`. Per-*session* isolation is the Pod itself —
one Pod per session, a deny-all NetworkPolicy under `:network`, and optionally a
sandbox runtime above.
Reaper configuration is the same shape as Docker's — the backend entry needs
whatever `list_live/1` requires to reach the cluster and rebuild handles:
```elixir
config :crowd_control,
store: {CrowdControl.Store.DETS, path: "/var/lib/crowd_control/sessions.dets"},
owner_id: "worker-1",
reaper: [
backends: [
{CrowdControl.Backend.Kubernetes,
image: "my-cli:latest", namespace: "sandboxes", network: :deny_all}
],
sweep_interval: :timer.minutes(5),
reap_grace_ms: 60_000
]
```
Pods are selected by a `crowd_control.owner_hash` label rather than the raw
owner, because `nonode@nohost` is not a legal label value. The raw owner rides
along in a `crowd_control.owner` annotation, so the reaper's local ownership
re-check still compares exact strings.
### Durability and reattach
A container outlives the session that created it, which is the whole reason
the rest of this machinery exists.
```elixir
config :crowd_control,
# Where session records live. ETS (default) survives a session crash;
# DETS also survives a node restart.
store: {CrowdControl.Store.DETS, path: "/var/lib/crowd_control/sessions.dets"},
# Scopes ownership. Defaults to the node name.
owner_id: "worker-1",
reaper: [
backends: [{CrowdControl.Backend.Docker, image: "my-cli:latest"}],
sweep_interval: :timer.minutes(5),
reap_grace_ms: 60_000
]
```
`CrowdControl.Reaper` reconciles live containers against stored records at
boot and on a timer:
| live? | stored? | action |
|-------|---------|--------|
| yes | yes | reattach a session to it |
| yes | no | orphan — destroy it |
| no | yes | stale record — delete it |
Only `Backend.Local` writes nothing to the store: a local subprocess dies with
the VM, so there is nothing to reattach and the per-chunk write would be pure
overhead.
Two safety properties are worth knowing about, because both are load-bearing:
- A backend whose container listing **fails** is skipped, never read as
"nothing is live". The latter would destroy every running sandbox.
- Containers are labelled with `:owner_id` and the reaper only destroys its
own. Two nodes cannot reap each other's work.
### The tee-file contract
This is the part to understand before changing anything in the Docker backend.
Each session's output is appended to a single file inside the container
(`:tee_path`), and the session persists a **byte offset** into that file plus
whatever partial line it was holding. Resuming means reading the file from that
offset and re-joining the partial line — so a session interrupted in the middle
of a JSON line resumes byte-exactly, with nothing lost and nothing duplicated.
That only holds while offsets stay valid, which is why:
- **Containers are created with `RestartPolicy: no`.** A restart truncates the
tee file and invalidates every persisted offset.
- **The tee file is capped, never rotated.** `:max_stream_bytes` destroys the
sandbox when output grows too large. Rotation would silently invalidate every
offset — corrupting resume in exactly the way the offset cursor exists to
prevent. A hard cap is the correct answer for a bounded resource.
If you are tempted to add log rotation here, this is the reason not to.
## Sandbox Providers
A **backend** moves bytes; a **provider** owns the sandbox those bytes come out
of. Every backend above pairs one transport with one substrate, so a new
substrate meant writing a new transport too — and a VM has no exec API to build
one on. `CrowdControl.Backend.Sandboxd` breaks that pairing: it speaks one HTTP
protocol to one in-sandbox agent, and `CrowdControl.Provider` supplies the
substrate, so adding a substrate is provisioning code and nothing else.
```elixir
CrowdControl.start_session(
backend:
{CrowdControl.Backend.Sandboxd,
provider: {CrowdControl.Provider.Docker, image: "crowd_control/sandbox:dev", egress: :allow}},
prompt: "Refactor lib/foo.ex"
)
```
`Backend.Docker` is not deprecated and is not going anywhere: it works with any
image that has `sh` and `tail`, while this path needs an image containing the
agent. That is the trade.
| Provider | Sandbox | Reattach | Egress blocked? |
|----------|---------|----------|-----------------|
| `Provider.Docker` | one container | yes | **no** — see below |
| `Provider.Compose` | a per-session stack | yes | yes, structurally |
| `Provider.Gce` | a Compute Engine spot VM | yes | n/a — VM-level |
All three require `:req`, and a configured secret the agent token is derived
from:
```elixir
config :crowd_control, sandboxd_secret: System.fetch_env!("CC_SANDBOXD_SECRET")
```
Use at least 32 random bytes and keep it stable across restarts. It is never
persisted and never sent anywhere; each sandbox's token is
`HMAC-SHA256(secret, session_key)`, recomputed on reattach from the session key
the store already holds. Rotating it therefore fails reattach closed for every
existing sandbox — the deliberate cost of keeping no credential at rest.
### The sandboxd agent
`sandboxd` is a small OTP release (nested Mix project in `sandboxd/`, four
dependencies) that runs inside the sandbox and exposes seven routes:
| Method | Path | Notes |
|--------|------|-------|
| `GET` | `/v1/health` | readiness; the only unauthenticated route, returns `{"ok": true}` and nothing else |
| `POST` | `/v1/exec` | `{executable, args, env}` — env in the **body**, never argv; one exec per sandbox |
| `POST` | `/v1/stdin` | `{data: base64}` |
| `GET` | `/v1/stream` | `?offset=N`, chunked, **0-indexed** |
| `GET` | `/v1/status` | `{alive, exit_status, bytes}`, long-polled with `?wait_ms=` |
| `PUT` | `/v1/files/*path` | raw bytes, `?mode=0600`; traversal refused |
| `POST` | `/v1/shutdown` | kills the CLI; destroying the *sandbox* is the provider's job |
Build an image containing it:
```sh
# minimal, for exercising the transport
docker build --target sandbox-dev -t crowd_control/sandbox:dev .
# the full image, which also carries the agent CLI
docker build -t crowd_control:latest .
```
The release embeds its own ERTS, so it also runs on a bare VM with no Erlang
installed — that is what the GCE provider depends on. It needs `libssl3` present
(ERTS's crypto NIF links against the system OpenSSL) and reads its entire
configuration from the environment: `CC_SANDBOXD_TOKEN` (required; boot fails
without it), `CC_SANDBOXD_PORT`, `CC_SANDBOXD_BIND`, `CC_SANDBOXD_CAPTURE`.
The capture file is byte-for-byte the same artifact as the Docker backend's tee
file, so [the tee-file contract](#the-tee-file-contract) applies here unchanged
— including the reason it is capped and never rotated.
### Docker provider
```elixir
{CrowdControl.Provider.Docker,
image: "crowd_control/sandbox:dev",
egress: :allow,
cpus: 1.5,
memory: 512 * 1024 * 1024}
```
| Option | Default | Notes |
|--------|---------|-------|
| `:image` | — | **Required.** Must contain the sandboxd release |
| `:egress` | — | **Required.** `:allow` or `:no_nat`; never inferred |
| `:agent_port` | `8080` | Port the agent listens on inside the container |
| `:capture_path` | `/var/log/cc/out.jsonl` | |
| `:ready_timeout` | `30_000` | How long `acquire/1` waits for `GET /v1/health` |
| `:docker_host`, `:timeout` | as `Backend.Docker` | |
| `:agent_env` | `%{}` | Extra env for the *agent*, not the CLI |
Hardening options are identical to the Docker backend's and come from the same
module, so the two cannot drift.
**This provider does not block egress, and does not claim to.** On one
container, `Internal: true` and a published port are mutually exclusive:
publishing requires a non-internal endpoint, and attaching one restores full
internet access. That is measured, not assumed, and the failure is silent —
Docker answers `201` with no warning and simply discards the binding. So
`:egress` is required, exactly as `Backend.Docker` requires an explicit
`:network_mode`:
- `egress: :allow` — a private per-sandbox bridge, full outbound access.
- `egress: :no_nat` — masquerade disabled. Blocks the internet, but the Docker
host, sibling containers and embedded DNS stay reachable. It is "no NAT", not
"dropped" — see [SECURITY.md](SECURITY.md#the-docker-provider).
For a structural egress block *and* a reachable agent, use the Compose
provider; it takes a second container to get both.
### Compose provider
A per-session stack over the Engine API. There is no `docker compose` CLI
dependency and there never will be one — the Engine API has no compose
endpoints, so the stack is synthesized directly.
```elixir
{CrowdControl.Provider.Compose,
image: "crowd_control/sandbox:dev",
services: [
%{name: "proxy", image: "cc/egress-proxy:1.2.3", egress: :allow, port: 8080}
],
proxy_service: "proxy",
volumes: [%{name: "workspace"}]}
```
| Option | Default | Notes |
|--------|---------|-------|
| `:services` | `[]` | Sidecar specs; a spec named `:sandbox_service` *is* the sandbox |
| `:sandbox_service` | `"sandbox"` | |
| `:forwarder_service` / `:forwarder_image` | `"forwarder"` / `alpine/socat:1.8.1.3` | Always synthesized, never caller-supplied |
| `:network` | `[internal: true, driver: "bridge"]` | |
| `:volumes` | `[]` | Named `-`, destroyed with the stack |
| `:ready` | `%{}` | Per-service healthchecks, gating start order |
| `:project_name` | `cc-` | |
| `:proxy_service` | unset | Sidecar fronting the egress proxy |
| `:health_timeout` | `60_000` | |
Agent and hardening options are the Docker provider's, and hardening applies to
**every** container in the stack — there are deliberately no per-service
overrides, because a sidecar quietly weaker than the sandbox it shares a network
with is not a useful thing to express.
The sandbox sits on an `Internal: true` network with no port bindings at all: no
default route exists, so the internet, the Docker host, sibling containers and
embedded DNS are unreachable structurally rather than by a missing NAT rule. A
synthesized `socat` forwarder is dual-homed onto a publishing bridge and is the
only part of the stack the host can reach. `:egress` is required on every
sidecar and refused on the sandbox. Services address each other by name.
### GCE provider
Requires the optional `:gcp_compute` dependency:
```elixir
{:gcp_compute, "~> 0.2"}
```
```elixir
{CrowdControl.Provider.Gce,
project: "my-project",
zone: "us-central1-a",
sandboxd_url: "https://github.com/.../sandboxd-linux-amd64.tar.gz",
sandboxd_sha256: "…",
machine_type: "e2-standard-2",
spot: true}
```
Runnable example: [`examples/gce_spot_vm.exs`](examples/gce_spot_vm.exs) — one
spot VM per sandbox, end to end. It is **billable** and guarded by an explicit
`CC_GCE_CONFIRM=yes`.
| Option | Default | Notes |
|--------|---------|-------|
| `:project`, `:zone`, `:token_provider` | — | Or a ready `%GcpCompute.Config{}` as `:gce_config` |
| `:sandboxd_url` | — | **Required.** Agent tarball, published under the `sandboxd-v*` release channel: `https://github.com/nyo16/CrowdControl/releases/download/sandboxd-v0.1.0/sandboxd-linux-amd64.tar.gz`. Any URL the VM can reach over plain HTTPS works — a GCS object or your own mirror is fine |
| `:sandboxd_sha256` | — | **Required and never skipped.** The VM fetches the tarball with no credential, so the checksum is the only thing that makes an untrusted source safe. A `.sha256` sidecar ships with every release |
| `:bootstrap_script` | unset | Shell run as root before the agent installs |
| `:spot` | `true` | |
| `:external_ip` | `true` | The agent stays on loopback regardless — see below |
| `:max_run_duration` | `ready_timeout + session timeout + 5 min` | Server-side orphan backstop |
| `:ready_timeout` | `180_000` | Operation DONE → healthy agent. Measured requirement is **31s** with no bootstrap script (8.9s insert, 0s to RUNNING, 23.8s ssh, 7.3s health; 39.9s end to end on a spot `e2-small`). Attach to `[:crowd_control, :gce, :phase]` to measure your own image |
| `:ssh_port` | `22` | |
| `:host_key_fp` | unset | Pin the VM's host key |
The agent binds the VM's **loopback** and is reached through an OTP `:ssh`
local-port-forward, using a per-session ed25519 key generated in memory that
never touches disk. Port 22 is the only reachable port. `external_ip: false` is
the hardened mode and needs same-VPC connectivity plus Cloud NAT.
`max_run_duration` with `instanceTerminationAction: DELETE` is a **server-side**
backstop: the reaper runs on the BEAM, so if the node dies mid-provision nothing
local knows the VM exists, and a leaked spot VM bills forever. No service
account is attached unless you ask for one — with one, the sandboxed CLI can
mint project credentials from the metadata server.
> `:ready_timeout`'s default is an **estimate, not a measurement.** The billable
> end-to-end spike that would have measured operation-DONE → SSH-ready →
> agent-healthy was never run. Measure it against your own image and machine
> type before trusting it in production.
See [SECURITY.md](SECURITY.md#sandbox-agent-transport) for the full posture of
all three, including the regressions each carries relative to the others.
## CLI Options
Every session picks an **agent adapter** (`CrowdControl.Agent`), which decides
both the argv the CLI is launched with and the wire format it speaks:
| `:agent` | Adapter | CLI |
|----------|---------|-----|
| `:claude` (default), `:claude_code` | `CrowdControl.Agent.ClaudeCode` | `claude` — `--output-format stream-json` |
| `:open_code`, `:opencode` | `CrowdControl.Agent.ClaudeCode` | `open-code` — same wire format |
| `:omp` | `CrowdControl.Agent.Omp` | `omp --mode rpc` — see [Using omp](#using-omp-oh-my-pi) |
Omit `:agent` and it is inferred from the `:executable` basename (`"omp"` =>
the omp adapter), defaulting to Claude Code.
The options below come from `CrowdControl.CLI.build_command/1` (Claude Code)
and can be passed to `start_session`, `run`, and `run_many`. The omp adapter
accepts the shared subset plus its own flags, and raises on Claude-Code-only
options — see `CrowdControl.Agent.Omp`.
| Option | Description |
|--------|-------------|
| `:agent` | Agent adapter (see table above) |
| `:executable` | CLI binary name or path (default: `"claude"`, or `"omp"` for `agent: :omp`) |
| `:prompt` | Initial prompt to send |
| `:model` | Model to use (`"sonnet"`, `"opus"`, `"haiku"`) |
| `:system_prompt` | Custom system prompt |
| `:allowed_tools` | List of allowed tool names |
| `:permission_mode` | Permission mode string |
| `:max_budget_usd` | Spending ceiling |
| `:session_id` | Session ID for new sessions |
| `:resume` | Session ID to resume |
| `:continue` | `true` to continue most recent session |
| `:add_dir` | Additional project directory |
| `:include_partial_messages` | `true` for streaming deltas |
| `:no_session_persistence` | `true` to skip saving to disk |
| `:settings` | Path to settings JSON file or inline JSON string |
| `:setting_sources` | List of setting sources to load (`["user", "project", "local"]`) |
| `:mcp_config` | Path to MCP config JSON file(s) (string or list) |
| `:strict_mcp_config` | `true` to only use MCP servers from `:mcp_config` |
| `:agents` | JSON string or map defining custom agents |
| `:plugin_dir` | Path to a plugin directory |
| `:bare` | `true` for minimal mode (skip hooks, LSP, plugins, auto-memory) |
| `:extra_args` | List of additional CLI arguments |
| `:api_key` | Anthropic API key (sets `ANTHROPIC_API_KEY` for the subprocess) |
| `:oauth_token` | Claude subscription token (`CLAUDE_CODE_OAUTH_TOKEN` for Claude Code, `ANTHROPIC_OAUTH_TOKEN` for omp) |
| `:auth_token` | Bearer credential for a gateway or self-hosted endpoint (`ANTHROPIC_AUTH_TOKEN`; Claude Code only) |
| `:api_url` | Custom API base URL (sets `ANTHROPIC_BASE_URL` for the subprocess) |
| `:env` | Map of arbitrary environment variables for the subprocess |
| `:timeout` | Session timeout in milliseconds (default: `nil` / no timeout) |
| `:max_prompt_size` | Maximum prompt size in bytes (default: `nil` / no limit) |
## API Reference
### CrowdControl (orchestration)
| Function | Description |
|----------|-------------|
| `run(prompt, opts)` | Single-shot: start session, send prompt, collect result, stop |
| `run_many(prompt, opts_list)` | Same prompt across N sessions with different options |
| `start_session(opts)` | Start one supervised session (returns `{:error, :max_sessions_reached}` at limit) |
| `start_sessions(opts_list)` | Start N sessions in parallel |
| `broadcast(sessions, prompt)` | Send the same prompt to all sessions |
| `collect(sessions, timeout)` | Wait for result messages from all sessions |
| `stop_all(sessions)` | Gracefully stop all sessions |
| `healthy?()` | Returns `true` if the session supervisor is alive |
### CrowdControl.Session (per-instance)
| Function | Description |
|----------|-------------|
| `send_prompt(session, prompt)` | Send a user prompt |
| `subscribe(session)` | Receive messages as `{:crowd_control, pid, msg}` |
| `get_status(session)` | Returns `:starting`, `:running`, `:completed`, or `:error` |
| `get_session_id(session)` | CLI-assigned session ID |
| `get_messages(session)` | All accumulated messages |
| `current_turn(session)` | Turn currently in flight (prompts written so far) |
| `stop(session)` | Graceful shutdown |
### Message types
Subscribers receive `{:crowd_control, session_pid, message}` where message is
the same shape for every agent adapter:
| Message | When |
|---------|------|
| `{:system_init, map}` | CLI initialized, contains `session_id`, `tools`, `model` |
| `{:assistant, map}` | Assistant response with `content` blocks |
| `{:user, map}` | Tool execution results |
| `{:result, subtype, map}` | Turn complete. Subtype: `"success"`, `"error_max_turns"`, `"error_max_budget_usd"` (Claude Code), `"error_prompt_failed"` (omp). `map["turn"]` is the 1-based turn the result belongs to; omp adds `map["local_only"]` for slash commands answered without a model call |
| `{:stream_event, map}` | Partial message delta (Claude Code: requires `:include_partial_messages`; omp: always) |
| `{:timeout, :session_expired}` | Session timed out (requires `:timeout` option) |
| `{:exit, status}` | CLI process exited with OS status code |
## Architecture
### Supervision Tree
```mermaid
graph TD
A[CrowdControl.Application] --> B[DynamicSupervisor
SessionSupervisor]
B --> C1[Session GenServer #1]
B --> C2[Session GenServer #2]
B --> C3[Session GenServer #N]
C1 --> D1[NetRunner.Process
claude CLI]
C1 --> E1[Reader Process
stdout drain]
C2 --> D2[NetRunner.Process
claude CLI]
C2 --> E2[Reader Process
stdout drain]
C3 --> D3[NetRunner.Process
open-code CLI]
C3 --> E3[Reader Process
stdout drain]
style A fill:#4a9eff,color:#fff
style B fill:#ff6b6b,color:#fff
style C1 fill:#51cf66,color:#fff
style C2 fill:#51cf66,color:#fff
style C3 fill:#51cf66,color:#fff
style D1 fill:#ffd43b,color:#333
style D2 fill:#ffd43b,color:#333
style D3 fill:#ffd43b,color:#333
style E1 fill:#cc5de8,color:#fff
style E2 fill:#cc5de8,color:#fff
style E3 fill:#cc5de8,color:#fff
```
### Session Lifecycle
```mermaid
stateDiagram-v2
[*] --> starting: start_link(opts)
starting --> running: system_init received
running --> running: assistant / user messages
running --> completed: result message received
running --> error: CLI crash / timeout
starting --> error: spawn failure
completed --> [*]: stop
error --> [*]: stop / supervisor restart
```
### Message Flow
```mermaid
sequenceDiagram
participant Caller
participant Session as Session GenServer
participant Reader as Reader Process
participant NR as NetRunner.Process
participant CLI as claude CLI
Caller->>Session: start_session(prompt: "...")
Session->>NR: start_link("claude", args)
NR->>CLI: spawn OS process
Session->>Reader: spawn_link(reader_loop)
Session->>NR: write(encoded_prompt)
NR->>CLI: stdin: {"type":"user",...}\n
CLI->>NR: stdout: {"type":"system","subtype":"init",...}\n
NR->>Reader: read() -> {:ok, data}
Reader->>Session: cast {:stdout_data, data}
Session->>Caller: send {:crowd_control, pid, {:system_init, ...}}
CLI->>NR: stdout: {"type":"assistant",...}\n
NR->>Reader: read() -> {:ok, data}
Reader->>Session: cast {:stdout_data, data}
Session->>Caller: send {:crowd_control, pid, {:assistant, ...}}
CLI->>NR: stdout: {"type":"result","subtype":"success",...}\n
NR->>Reader: read() -> {:ok, data}
Reader->>Session: cast {:stdout_data, data}
Session->>Caller: send {:crowd_control, pid, {:result, ...}}
```
### Parallel Execution
```mermaid
graph LR
A[CrowdControl.run_many] --> B[Task.async_stream]
B --> S1[Session 1
model: sonnet]
B --> S2[Session 2
model: opus]
B --> S3[Session 3
model: haiku]
S1 --> C1[claude --model sonnet]
S2 --> C2[claude --model opus]
S3 --> C3[claude --model haiku]
C1 --> R[collect results]
C2 --> R
C3 --> R
R --> OUT["{session, {:result, ...}}"]
style A fill:#4a9eff,color:#fff
style B fill:#ff6b6b,color:#fff
style S1 fill:#51cf66,color:#fff
style S2 fill:#51cf66,color:#fff
style S3 fill:#51cf66,color:#fff
style R fill:#ffd43b,color:#333
```
### Wire Protocol — Claude Code (stream-json)
```mermaid
sequenceDiagram
participant E as Elixir
participant C as Claude CLI
Note over E,C: stdin: newline-delimited JSON
E->>C: {"type":"user","message":{"role":"user","content":"Hello"}}\n
Note over E,C: stdout: newline-delimited JSON
C->>E: {"type":"system","subtype":"init","session_id":"..."}\n
C->>E: {"type":"assistant","message":{"content":[...]}}\n
C->>E: {"type":"result","subtype":"success","result":"..."}\n
Note over E,C: Multi-turn: send another prompt after result
E->>C: {"type":"user","message":{"role":"user","content":"Follow up"}}\n
C->>E: {"type":"assistant","message":{"content":[...]}}\n
C->>E: {"type":"result","subtype":"success","result":"..."}\n
```
### Wire Protocol — omp (`--mode rpc`)
```mermaid
sequenceDiagram
participant E as Elixir
participant O as omp --mode rpc
O->>E: {"type":"ready","protocolVersion":1,...}\n
Note over E,O: handshake: ask for the session id
E->>O: {"id":"cc-init","type":"get_state"}\n
O->>E: {"id":"cc-init","type":"response","command":"get_state","data":{"sessionId":"..."}}\n
Note over E,O: prompt is acked before the turn runs
E->>O: {"id":"cc-prompt-0","type":"prompt","message":"Hello","streamingBehavior":"followUp"}\n
O->>E: {"id":"cc-prompt-0","type":"response","command":"prompt","success":true}\n
O->>E: {"type":"message_update","assistantMessageEvent":{...}}\n
O->>E: {"type":"message_end","message":{"role":"assistant",...}}\n
O->>E: {"type":"agent_end","isTerminal":true,"messages":[...]}\n
Note over E,O: agent_end (isTerminal) is the turn boundary
```
### Module Dependency
```mermaid
graph BT
P[Protocol
pure functions]
CLI[CLI
pure functions]
A[Agent
adapter behaviour] --> P
A --> CLI
S[Session
GenServer] --> A
S --> NR[NetRunner.Process]
CC[CrowdControl
public API] --> S
CC --> DS[DynamicSupervisor]
APP[Application] --> DS
style P fill:#51cf66,color:#fff
style CLI fill:#51cf66,color:#fff
style S fill:#4a9eff,color:#fff
style CC fill:#ff6b6b,color:#fff
style APP fill:#ffd43b,color:#333
style A fill:#51cf66,color:#fff
style NR fill:#cc5de8,color:#fff
style DS fill:#ffd43b,color:#333
```
### Fault Tolerance
```mermaid
graph TD
SUP[DynamicSupervisor
one_for_one] -->|supervises| S1[Session 1]
SUP -->|supervises| S2[Session 2]
SUP -->|supervises| S3[Session 3]
S2 -->|crash| X["Session 2 crashes"]
X -->|restart| S2R[Session 2
restarted]
SUP -->|supervises| S2R
S1 -.->|unaffected| S1
S3 -.->|unaffected| S3
NR[NetRunner Shepherd] -->|zero zombies| Z["OS process cleanup
guaranteed even on
BEAM crash"]
style X fill:#ff6b6b,color:#fff
style S2R fill:#51cf66,color:#fff
style Z fill:#ffd43b,color:#333
```
## Security
### API key protection
Per-session API keys (passed via `:api_key` or `:env`) are written to a temporary file with `0600` permissions, sourced by the shell, and deleted before the CLI process starts. Keys never appear in process arguments or `ps` output.
### Session limits
The `DynamicSupervisor` enforces a maximum number of concurrent sessions (default: 50). Configure via application environment:
```elixir
# config/runtime.exs
config :crowd_control, max_sessions: 100
```
When the limit is reached, `start_session/1` returns `{:error, :max_sessions_reached}`.
### Session timeouts
Sessions can be configured with an automatic timeout to prevent runaway processes:
```elixir
# A turn gets 5 minutes to finish
CrowdControl.run("Analyze this code",
timeout: 300_000,
add_dir: "/workspace"
)
```
The timer is armed at start and re-armed by each `send_prompt/2` call — it is **not** refreshed by output, so a single turn that streams for longer than the timeout is killed mid-flight. Size it against the slowest turn you expect rather than the length of the conversation; long autonomous tasks against a self-hosted model routinely need more than the default. On expiry, subscribers receive `{:timeout, :session_expired}` and the session shuts down.
### Input validation
- `send_prompt/2` rejects non-string prompts with `{:error, :invalid_prompt}`
- Optional `:max_prompt_size` (bytes) returns `{:error, :prompt_too_large}` when exceeded
### Docker security
See the [Docker](#docker) section for container hardening details (non-root user, capability dropping, read-only filesystem, resource limits).
## Requirements
- Elixir >= 1.18
- Erlang/OTP >= 27
- C compiler (gcc or clang) for net_runner NIF
- At least one agent CLI: `claude` ([install](https://docs.anthropic.com/en/docs/claude-code)), `open-code`, or `omp` ([omp.sh](https://omp.sh/))
- `ANTHROPIC_API_KEY` environment variable set
## License
Apache License 2.0. See [LICENSE](https://github.com/nyo16/CrowdControl/blob/master/LICENSE).