The model catalog: which models exist, at which versions, and where traffic goes.
Why this is a GenServer that is never called on the hot path
MLServe.predict/3 runs in whatever process is handling the request — a Phoenix controller, an
Oban worker, a Task. If every prediction had to GenServer.call this process just to find out
which worker to talk to, this single process would serialise the entire node's inference
traffic before any inference happened.
So the registry owns a :protected ETS table with read_concurrency: true, and:
- Reads —
route/2,status/2,list/0— run:ets.lookup/2directly in the calling process. Lock-free, and they scale with schedulers. - Writes — registering, promoting, unregistering — go through the GenServer, which is the only process with write access. Serialising writes is free because they happen at load and deploy time, not per request.
Table layout
| Key | Value | Purpose |
|---|---|---|
{:route, name, version} | MLServe.Route | Hot path. Slim by design — see MLServe.Route. |
{:model, name, version} | map | Full detail for MLServe.model_status/2. |
{:default, name} | version | Which version unpinned traffic gets. |
{:canary, name} | {version, weight} | Progressive rollout. |
Splitting the hot-path route from the full entry keeps the per-prediction ETS copy to a few dozen words instead of copying the backend's entire configuration.
Summary
Functions
Returns the active canary as {version, weight}, or nil.
Returns a specification to start this module under a supervisor.
Returns the version unpinned traffic is routed to.
Fetches the route for an exact {name, version} pair, ready or not.
Returns every registered model name, sorted and deduplicated.
Fetches a route regardless of status. Used by lifecycle code, not by dispatch.
Resolves a model name to the route that should serve this request.
Returns the full status entry for a model version.
Returns every registered version of name, newest registration last.
Functions
Returns the active canary as {version, weight}, or nil.
Returns a specification to start this module under a supervisor.
See Supervisor.
Returns the version unpinned traffic is routed to.
@spec fetch_route(atom(), String.t()) :: {:ok, MLServe.Route.t()} | {:error, :model_not_found | :model_not_ready}
Fetches the route for an exact {name, version} pair, ready or not.
@spec list() :: [atom()]
Returns every registered model name, sorted and deduplicated.
@spec peek_route(atom(), String.t()) :: {:ok, MLServe.Route.t()} | {:error, :model_not_found}
Fetches a route regardless of status. Used by lifecycle code, not by dispatch.
@spec route( atom(), keyword() ) :: {:ok, MLServe.Route.t()} | {:error, :model_not_found | :model_not_ready}
Resolves a model name to the route that should serve this request.
Runs entirely in the calling process. Version selection, in order:
- An explicit
version: "2.1.0"option pins that version exactly. - Otherwise, if a canary is configured, a per-request roll sends
weight% of traffic to the candidate.:rand.uniform/1uses the caller's own seed — no shared state, no coordination. - Otherwise the default version.
Returns {:error, :model_not_ready} rather than :model_not_found when the model exists but is
loading, draining or failed, because those are different operational problems.
Returns the full status entry for a model version.
Returns every registered version of name, newest registration last.