Getting Started

Copy Markdown View Source

LLMProxy can run inside an existing Elixir application or as its own OTP service. Both modes use the same provider execution, routing, quota, usage, and tracing code.

Requirements

  • Elixir 1.17 or later
  • An Ecto repo in library mode, or the bundled DuckDB/QuackDB storage in standalone mode
  • At least one upstream provider credential
  • A master key for bootstrap and operator access

Phoenix, SQLite, Igniter, and Incant are optional integrations.

Install the package

Add LLMProxy to mix.exs:

def deps do
  [
    {:llm_proxy, "~> 0.1"}
  ]
end

Fetch dependencies:

mix deps.get

Choose a mode

Library mode

Choose library mode when LLMProxy belongs to an existing Elixir or Phoenix application. The host owns the repo, release, endpoint, and configuration.

Configure the host repo and disable LLMProxy's separate Cowboy listener:

# config/runtime.exs
config :llm_proxy,
  repo: MyApp.Repo,
  http_enabled: false,
  master_key: System.fetch_env!("LLM_PROXY_MASTER_KEY")

Run the installer to add migration aliases, then migrate:

mix igniter.install llm_proxy
mix llm_proxy.migrate

Continue with Library Mode.

Standalone mode

Choose standalone mode when multiple applications or external clients need one gateway. The bundled release owns HTTP serving, DuckDB storage, provider credentials, and operational state.

A minimal secret environment is:

export MASTER_KEY="replace-with-a-long-random-key"
export LLM_PROXY_PROVIDER_KEYS='{"openai":["sk-..."]}'

Set the port, database, routing policy, and other non-secret runtime values in the standalone TOML file described in the next guide.

Continue with Standalone Mode and Standalone Deployment.

Configure a model

Clients call public model names. Each public name routes to one or more upstream deployments.

Library configuration:

config :llm_proxy,
  providers: %{
    "openai" => %{api_keys: System.fetch_env!("OPENAI_API_KEYS")}
  },
  models: [
    fast: [route: [to: :openai, model: "gpt-4.1-mini"]]
  ]

Standalone TOML:

[[models]]
name = "fast"

[[models.routes]]
to = "openai"
model = "gpt-4.1-mini"

Provider credentials are secrets. In standalone mode, seed API-key pools with LLM_PROXY_PROVIDER_KEYS or use persisted provider-token storage. Library hosts may source credentials through their own runtime .exs configuration. Never put credentials in committed Elixir or TOML files.

Make a request

In-process:

{:ok, response} =
  LLMProxy.chat("Hello",
    model: "fast",
    api_key: System.fetch_env!("LLM_PROXY_MASTER_KEY")
  )

ReqLLM.Response.text(response.message)

HTTP:

curl http://127.0.0.1:4000/v1/chat/completions \
  -H "authorization: Bearer $MASTER_KEY" \
  -H "content-type: application/json" \
  -d '{"model":"fast","messages":[{"role":"user","content":"Hello"}]}'

The master key bypasses ordinary per-key quotas and is intended for bootstrap and operator access. Create scoped API keys for applications and users before production traffic.

Verify the installation

Check the service and catalog:

curl http://127.0.0.1:4000/health
curl http://127.0.0.1:4000/v1/models

A successful generation should also produce usage and trace records in the configured storage. If OpenTelemetry export is enabled, the provider attempt carries the same LLMProxy trace ID returned to the caller.

Next steps