Configuration

Repo

config :llm_proxy, repo: MyApp.Repo

Ecto repo used by the default storage adapter. Library hosts normally provide an already supervised repo. Standalone production uses LLMProxy.Storage.Repo.QuackDB.

Storage Adapter

config :llm_proxy, storage: MyApp.LLMProxyStorage

Module implementing LLMProxy.Storage.Adapter. Default: LLMProxy.Storage.Ecto.

HTTP Enabled

config :llm_proxy, http_enabled: false

Whether LLMProxy starts its own loopback Cowboy listener. Default: true. Disable in library mode unless a separate listener is intentional.

HTTP Port

config :llm_proxy, http: [port: 4000]

Port for the bundled listener. The listener binds 127.0.0.1.

Master Key

config :llm_proxy, master_key: System.fetch_env!("LLM_PROXY_MASTER_KEY")

Bootstrap/operator credential. It bypasses ordinary model and quota checks.

RPC Socket

config :llm_proxy, rpc_socket: "/run/llm-proxy/rpc.sock"

Starts the SafeRPC server for chat, operations, and optional Incant admin calls.

Public URL

config :llm_proxy, public_url: "https://llm.example.com"

Public base URL used by setup helpers and provider headers.

Request Body Limit

config :llm_proxy, body_limit_bytes: 32_000_000

Maximum authenticated JSON request body. Must be a positive integer. Default: 32 MB.

Remote Timeout

config :llm_proxy, remote_timeout_ms: 30_000

Default SafeRPC request timeout. Default: 30 seconds.

Provider Receive Timeout

config :llm_proxy, provider_receive_timeout_ms: :timer.minutes(10)

Provider transport timeout and standalone Cowboy idle ceiling. Default: 10 minutes.

Token Cooldown

config :llm_proxy, token_cooldown_ms: :timer.hours(4)

Default credential cooldown after rate limiting. Must be a positive integer no greater than 31 days. Default: 4 hours.

Provider Token Selection

config :llm_proxy, token_selection_strategy: :fill_first

:affinity is the default and preserves stable user-to-token affinity. :fill_first selects the highest-priority healthy token within the preferred credential kind, breaking ties by token ID. OAuth remains preferred over API keys. Standalone TOML:

[provider_tokens]
selection_strategy = "fill_first"

This non-secret setting is not read from the environment. Provider-token priorities are non-negative integers and can be edited through Incant's priority-only form. Set an initial priority when adding a token:

LLMProxy.Storage.add_token("openai-codex", "oauth", access_token, %{priority: 100})

Provider Token Codec

config :llm_proxy,
  provider_token_codec: {
    LLMProxy.Provider.TokenCodec.AESGCM,
    active_key_id: "2026-08",
    keys: %{
      "2026-08" => System.fetch_env!("PROVIDER_TOKEN_KEY_2026_08"),
      "2026-02" => System.fetch_env!("PROVIDER_TOKEN_KEY_2026_02")
    },
    allow_plaintext: true
  }

Codec for provider API keys and OAuth tokens. AES-GCM keys must decode to 32 bytes. This keyring is separate from :master_key. Writes use only the active key ID. Reads accept all key IDs in the keyring. The default plaintext codec is for compatibility and does not protect stored credentials.

Deployment Failure Threshold

config :llm_proxy, deployment_failure_threshold: 3

Default consecutive retryable failures before opening a deployment circuit. Default: 3.

Deployment Cooldown

config :llm_proxy, deployment_cooldown_ms: 30_000

Default open-circuit cooldown. Default: 30 seconds.

Max Retries

config :llm_proxy, max_retries: 1

Compatibility retry/fallback limit for configured fallback chains. Default: 1. The strict provider-dispatch budget is max_retries + 1; open-circuit and unsupported-protocol route skips do not consume it.

Standalone TOML:

[routing]
max_retries = 1

Replay Policy

config :llm_proxy, replay_policy: :safe_only

:safe_only permits fallback only when LLMProxy has evidence that the upstream did not accept work, such as unavailable credentials, a clear connection failure, or a 429 refusal. This is the default.

:allow_uncertain also permits replay after timeouts and 5xx responses. Use it only when duplicate cost and side effects are acceptable. It restores the earlier fallback behavior.

Standalone TOML:

[routing]
replay_policy = "safe_only"

Fallbacks

config :llm_proxy,
  fallbacks: %{"primary" => ["secondary"]}

Compatibility map for model fallback chains. Catalog routes are preferred for new configurations.

Cache Adapter

config :llm_proxy, cache: MyApp.LLMCache

Module implementing LLMProxy.Cache. Unset by default.

Cache Policy

config :llm_proxy,
  cache_policy: [
    enabled: true,
    ttl_ms: 60_000,
    models: %{"fresh" => [enabled: false]}
  ]

Default and per-model deterministic cache policy.

Guardrails

config :llm_proxy, guardrails: [MyApp.RequestPolicy]

Ordered modules implementing LLMProxy.Guardrail.

Public Model Allowlist

config :llm_proxy, public_models: ["fast", "codex"]

Filters model discovery and request admission. Unset by default, which exposes all registered models for compatibility. In standalone mode, configure visible catalog aliases in TOML; direct provider model IDs are not exposed by the allowlist:

[catalog]
public_models = ["fast", "codex"]

An explicit empty list exposes no models. This non-secret setting is not read from the environment.

Built-in Provider Keys

config :llm_proxy,
  providers: %{
    "openai" => %{api_keys: "sk-a,sk-b"},
    "anthropic" => %{api_keys: "sk-ant-..."},
    "openrouter" => %{api_keys: "sk-or-..."},
    "openai-codex" => %{oauth_tokens: "access|refresh|expires_ms|account"}
  }

Runtime bootstrap values are persisted into provider-token storage.

Named Provider

config :llm_proxy,
  providers: %{
    "example-service" => %{
      adapter: "openai",
      base_url: "https://api.example.com/v1",
      token_pool: "example-production"
    }
  }

Uses an existing ReqLLM adapter without a custom provider module.

Provider Base URL

%{base_url: "https://api.example.com/v1"}

Default endpoint for a named or built-in provider.

Provider Token Pool

%{token_pool: "example-production"}

Default credential pool for provider routes.

GLM Usage Source

%{
  adapter: "zai_coding_plan",
  base_url: "https://api.z.ai/api/coding/paas/v4",
  token_pool: "glm-production",
  usage_adapter: "glm",
  usage_auth_scheme: "raw",
  usage_paths: ["/api/monitor/usage/quota/limit", "/api/monitor/usage"]
}

usage_adapter is optional for zai, zai_coder, and zai_coding_plan; when present it must be the string "glm". usage_auth_scheme must be "raw" or "bearer". usage_paths must be a non-empty list of at most three distinct absolute origin paths. Raw whitespace and control bytes are rejected; encode valid path characters when needed. Invalid values fail configuration loading rather than being coerced or ignored.

Anthropic Defaults

%{
  api_version: "2023-06-01",
  beta: "feature-a,feature-b",
  conversion_defaults: %{max_tokens: 4096}
}

Provider-specific headers and request-conversion defaults.

Single Route

config :llm_proxy,
  models: [
    fast: [route: [to: :openai, model: "gpt-4.1-mini"]]
  ]

Public alias fast targets one upstream deployment.

Multiple Routes

config :llm_proxy,
  models: [
    fast: [
      routing: :ordered,
      routes: [
        [to: :openai, model: "gpt-4.1-mini"],
        [to: :anthropic, model: "claude-3-5-haiku-20241022", order: 2]
      ]
    ]
  ]

Routing

routing: :round_robin

:ordered, :shuffle, :round_robin, :weighted_shuffle, :lowest_cost, or :latency_aware.

Route Timeout

timeout: 30_000

Per-attempt timeout in milliseconds. timeout_ms is also accepted.

Route Circuit Breaker

failure_threshold: 3,
cooldown_ms: 30_000

Overrides deployment defaults.

Route Order

order: 2

Fallback group. Lower groups are attempted first.

Route Weight

weight: 3

Relative weight for :weighted_shuffle.

Route Token Pool

token_pool: "special-production"

Overrides the provider's default pool.

Master Key

MASTER_KEY=...

Bootstrap/operator credential.

Provider API-Key Pools

LLM_PROXY_PROVIDER_KEYS={"example-production":["secret"]}

Secret JSON object mapping token-pool names to API-key arrays. Use pool names such as openai, anthropic, or openrouter for built-in providers. Codex OAuth credentials are provisioned through the admin login flow and persisted in provider-token storage.

Provider Token Keyring

LLM_PROXY_PROVIDER_TOKEN_KEYRING={"active_key_id":"2026-08","keys":{"2026-08":"base64-encoded-32-byte-key"}}

Secret JSON keyring for standalone provider-token encryption. Keys must decode to 32 bytes. Keep this keyring separate from MASTER_KEY; writes use the active key ID and reads accept every retained key.

TOML Path

LLM_PROXY_CONFIG_TOML=/etc/llm-proxy/config.toml

Standalone TOML data file. Default: /etc/llm-proxy/config.toml.

Server

[server]
port = 4000
public_url = "https://llm.example.com"
body_limit_bytes = 32000000
rpc_socket = "/run/llm-proxy/rpc.sock"

The listener remains bound to loopback. Omit rpc_socket to disable SafeRPC.

Storage

[storage]
database = "/var/lib/llm-proxy/llm_proxy.duckdb"
quackdb_uri = "http://127.0.0.1:9494"
quackdb_endpoint = "quack:localhost:9494"

Routing Runtime

[routing]
max_retries = 1
replay_policy = "safe_only"
provider_connect_timeout_ms = 10000
token_selection_strategy = "affinity"

Fallback deployments belong in model routes, not a separate fallback map.

Provider Usage

[provider_usage]
auto_refresh = true
refresh_interval_ms = 300000
request_timeout_ms = 10000
stale_after_ms = 600000

The refresh interval must be from 60000 through 3600000. The request timeout must be from 1000 through 30000. The stale time must be at least the refresh interval and no more than 86400000. Omitted values use the built-in defaults.

Telemetry

[telemetry]
otlp_endpoint = "http://127.0.0.1:4318"

Omit the endpoint to keep trace export disabled.

Provider Token Rollout

[provider_tokens]
allow_plaintext = true

Keep plaintext compatibility enabled only during an explicit encryption rollout. After migration and verification, set it to false; a missing keyring then fails startup.

Provider

[providers.example-service]
adapter = "openai"
base_url = "https://api.example.com/v1"
token_pool = "example-production"

Optional provider usage keys are usage_adapter, usage_auth_scheme, and usage_paths.

Model

[[models]]
name = "example/model"
routing = "ordered"

Route

[[models.routes]]
to = "example-service"
model = "upstream-model"
timeout = 30000
failure_threshold = 3
cooldown_ms = 30000
order = 1
weight = 1

TOML contains data only. Unknown keys are rejected; arbitrary atoms are not created.