Architecture

LLMProxy has one provider-execution boundary with several transport adapters around it, including local Unix-socket calls through SafeRPC. Authentication, quota checks, policy, routing, accounting, and tracing do not belong in HTTP-only code.

Request path

LLMProxy.chat/2    ReqLLM provider    HTTP routes    SafeRPC
       │                 │                │             │
       └─────────────────┴────────┬───────┴─────────────┘
                                  ▼
                         LLMProxy.Provider
                                  │
                       actor and API-key lookup
                                  │
                    quota and public-model access
                                  │
                       before-request guardrails
                                  │
                      cache key and catalog plan
                                  │
                  timeout / retry / fallback / circuit
                                  │
                    response and stream normalization
                                  │
                  guardrails / cache / usage / tracing
                                  │
                         caller-specific response

Core request contracts use ReqLLM.Context, ReqLLM.Message, and LLMProxy.Protocol.Request. OpenAI and Anthropic wire maps stay at protocol and transport boundaries.

Module map

LLMProxy
├── Provider                 in-process execution boundary and ReqLLM provider
├── HTTP
│   ├── Router               standalone Plug router
│   ├── RouteSpec            route table shared with Phoenix
│   └── Routes               protocol adapters
├── Phoenix.Router           Phoenix forwarding macros
├── Providers
│   ├── Registry             provider/model lookup and catalog resolution
│   ├── Execution            attempts, fallback, timeout, and retry handling
│   ├── CircuitBreaker       deployment health state
│   └── Routing              deployment ordering strategies
├── Protocol                 OpenAI/Anthropic request normalization
├── Catalog                  public model aliases and deployment data
├── TokenPool                credential selection and cooldown state
├── Cache                    adapter, deterministic key, and policy
├── GuardrailPipeline        request, response, and stream policy hooks
├── Accounting               usage, spend, traces, and message recording
├── Telemetry                telemetry events and OpenTelemetry spans
├── Stream                   normalized events, SSE writing, and heartbeats
├── Storage                  public facade and adapter boundary
├── Admin                    optional Incant resources and dashboard
└── RPC                      SafeRPC server for admin and operations

Transport adapters

Local Elixir

LLMProxy.chat/2 normalizes a prompt or ReqLLM context, then calls LLMProxy.Provider. It returns %LLMProxy.Response{} with the normalized ReqLLM response, provider identity, usage, trace ID, and cache metadata.

ReqLLM

LLMProxy.Provider registers as ReqLLM provider :llm_proxy. The adapter builds an internal request and halts the Req pipeline with the response produced by the same provider boundary.

A ReqLLM call can execute locally or pass its request through SafeRPC when a client/socket option is provided.

HTTP

LLMProxy.HTTP.RouteSpec is shared by the standalone Plug router and Phoenix macros. Route modules authenticate, parse wire requests, call the provider boundary, and render protocol-shaped responses or SSE events.

HTTP modules should not implement provider selection, accounting, or fallback.

SafeRPC

SafeRPC exposes typed operation tuples over a local Unix socket. Ordinary chat calls target LLMProxy; remote admin operations target LLMProxy.Admin; drain operations target LLMProxy.Ops.

The socket transports portable request and response data. Repos, schemas, callbacks, policies, and side effects remain in the LLMProxy service VM.

Catalog and providers

The catalog separates public model identity from deployment identity:

%LLMProxy.Catalog.Model{
  name: "fast",
  deployments: [
    %LLMProxy.Catalog.Deployment{
      provider_name: "openai-primary",
      provider: LLMProxy.Providers.ReqLLM,
      upstream_model: "gpt-4.1-mini",
      token_pool: "openai-production",
      timeout_ms: 30_000
    }
  ]
}

Named providers with an adapter execute through LLMProxy.Providers.ReqLLM. Built-in modules remain for native or specialized integrations.

Deployment attempts carry provider identity, upstream model, token pool, timeout, order, weight, and circuit-breaker settings. This keeps credential selection and health state attached to the route that needs them.

Storage

LLMProxy.Storage delegates to config :llm_proxy, storage: .... The default LLMProxy.Storage.Ecto implementation uses LLMProxy.Storage.Repo, which delegates to the configured Ecto repo.

LLMProxy.Storage
       │
       ▼
configured storage adapter
       │
       ▼
LLMProxy.Storage.Ecto
       │
       ▼
LLMProxy.Storage.Repo
       │
       ▼
host repo / bundled SQLite / bundled QuackDB

Library hosts usually provide their existing repo. The standalone production release uses the bundled QuackDB repo and supervises a managed local QuackDB process.

Database-specific queries and migrations branch on the configured repo adapter. Provider execution depends on the storage facade, not a concrete database.

Optional integrations

Optional modules use compile-time availability checks:

• Incant admin modules load only when Incant is installed.
• SQLite repo modules load only when ecto_sqlite3 is installed.
• Phoenix router integration requires Phoenix in the host.
• Igniter installation support requires Igniter.

The gateway core remains usable without those packages.

Ownership rules

• Provider execution and fallback belong under LLMProxy.Provider and LLMProxy.Providers.
• HTTP modules parse and render only.
• Phoenix embedding helpers belong under LLMProxy.Phoenix.
• Deployment selection belongs under LLMProxy.Providers.Routing, not Phoenix routing.
• Endpoint/model differences belong in provider data unless they require a new protocol.
• Storage access goes through LLMProxy.Storage or LLMProxy.Storage.Repo facades.
• LLMProxy.Admin is the only service-owned admin surface; do not add a second public admin API.
• SafeRPC transports operations but does not move executable service internals between VMs.