Standalone Deployment

Copy Markdown View Source

The standalone OTP release owns its HTTP listener, DuckDB-backed storage, provider token pools, model catalog, and optional SafeRPC socket.

This guide covers the release itself. Package deployment systems should wrap these steps with immutable artifacts, secret injection, service supervision, backups, and rollback appropriate to the host.

Build a Mix release

From the LLMProxy source checkout:

MIX_ENV=prod mix deps.get --only prod
MIX_ENV=prod mix compile
MIX_ENV=prod mix release llm_proxy

The release is written to:

_build/prod/rel/llm_proxy

Its runtime configuration:

  • uses DuckDB through the bundled QuackDB adapter;
  • starts a managed local QuackDB process;
  • binds the HTTP listener to 127.0.0.1;
  • optionally loads /etc/llm-proxy/config.toml;
  • reads secrets and operational values from the environment.

Build a ReleaseKit artifact

ReleaseKit can package the Mix release as an immutable tarball with a machine-readable manifest:

MIX_ENV=prod mix release_kit.artifact --out-dir _build/prod/artifacts

Output:

_build/prod/artifacts/llm_proxy-<version>.tar.gz
_build/prod/artifacts/llm_proxy.etf

Use the standard Mix release directly when your deployment system does not consume ReleaseKit manifests.

Filesystem layout

A typical host uses:

/opt/llm-proxy/releases/<release>/    immutable unpacked release
/opt/llm-proxy/current/               symlink to active release
/etc/llm-proxy/config.toml            non-secret provider and model data
/var/lib/llm-proxy/llm_proxy.duckdb   persistent database
/run/llm-proxy/rpc.sock               optional SafeRPC socket

Keep the database and runtime directory outside immutable release paths. Do not store provider secrets in the TOML file or release archive.

Runtime environment

Minimal production variables:

MASTER_KEY="replace-with-a-long-random-key"
DATABASE_PATH="/var/lib/llm-proxy/llm_proxy.duckdb"
PORT="4000"
PUBLIC_URL="https://llm.example.com"

At least one provider pool also needs credentials:

OPENAI_API_KEYS="sk-..."
ANTHROPIC_API_KEYS="sk-ant-..."
OPENROUTER_API_KEYS="sk-or-..."
LLM_PROXY_PROVIDER_KEYS='{"custom-production":["secret"]}'

Optional service integration:

LLM_PROXY_CONFIG_TOML="/etc/llm-proxy/config.toml"
LLM_PROXY_RPC_SOCKET="/run/llm-proxy/rpc.sock"
OTEL_EXPORTER_OTLP_ENDPOINT="http://127.0.0.1:4318"

See the Configuration Cheatsheet for all supported variables.

Migrate before startup

Run migrations from the new release with the same storage environment the service will use:

/opt/llm-proxy/current/bin/llm_proxy eval 'LLMProxy.ReleaseTasks.migrate()'

The migration task starts its own temporary QuackDB dependency, applies all pending migrations, checkpoints the database, and stops.

Do not run migration and the long-lived service against the same exclusive local database concurrently unless the storage deployment explicitly supports it.

Start and verify

Start under your service supervisor, or manually for a smoke test:

/opt/llm-proxy/current/bin/llm_proxy start

Verify health:

curl --fail --silent http://127.0.0.1:4000/health

A healthy response reports readiness, drain state, and active work counts. Then verify:

  1. /v1/models contains intended public aliases.
  2. An authenticated non-streaming completion succeeds.
  3. A streaming completion emits text and terminates normally.
  4. Usage, provider identity, and trace ID are recorded.
  5. The reverse proxy preserves streaming and request IDs.
  6. OTLP export works when configured.
  7. The SafeRPC socket has expected ownership and mode when enabled.

Reverse proxy

LLMProxy binds loopback only. Terminate public TLS and enforce network policy in a reverse proxy or ingress.

The proxy must:

  • support long-lived streaming responses;
  • avoid buffering SSE bodies;
  • allow the configured request body limit;
  • preserve Authorization, x-api-key, and request-ID headers;
  • use an idle timeout longer than the expected stream silence window.

LLMProxy sends SSE comment heartbeats during upstream silence, but every intermediary must still permit streaming flushes.

Graceful replacement

When LLM_PROXY_RPC_SOCKET is configured, release tasks can coordinate drain state through the running service:

bin/llm_proxy eval 'LLMProxy.ReleaseTasks.drain_start()'
bin/llm_proxy eval 'LLMProxy.ReleaseTasks.drain_await(1_800_000)'

Drain mode stops accepting new tracked work while existing requests, streams, and agents finish. Check status with:

bin/llm_proxy eval 'LLMProxy.ReleaseTasks.drain_status()'

Cancel a drain if replacement is aborted:

bin/llm_proxy eval 'LLMProxy.ReleaseTasks.drain_cancel()'

After active work reaches zero, stop the old release and start the new one. Your service supervisor should enforce a bounded shutdown timeout longer than the application's drain window.

Rollback

Application rollback and schema rollback are different decisions.

  1. Stop or drain the current release.
  2. Point the active symlink at the previous immutable release.
  3. Start it with the same persistent database and secret environment.
  4. Verify health, catalog, completion, and storage access.

Bundled migrations are forward migrations. Do not reverse database changes automatically unless the release documents and tests a safe down migration. Take a database backup before any migration that could make rollback incompatible.

Backups

Back up the DuckDB file while storage is in a consistent state. Include:

  • API-key hashes and policies;
  • provider API keys and OAuth refresh material;
  • catalog-independent operational state;
  • usage, traces, messages, and feedback required by retention policy.

Exclude:

  • release archives already stored elsewhere;
  • SafeRPC socket files;
  • logs, caches, and temporary QuackDB runtime files.

Document restore order: restore data, install release, recreate runtime directories and permissions, migrate, start, and verify.