@spec await_ready(model() | {model(), version()} | :all, timeout()) ::
  :ok | {:error, :timeout}

Blocks until a model is ready, or the timeout expires.

Models load asynchronously, which makes both tests and startup scripts racy without this.

Takes the same argument as ready?/1, so a specific version can be awaited while an older one is still the default — which is exactly the situation during a version rollout.

Examples

MLServe.load_model(:fraud_detection, backend: MyApp.FraudModel)
:ok = MLServe.await_ready(:fraud_detection)

MLServe.load_model(:fraud_detection, backend: MyApp.FraudModel, version: "2.1.0")
:ok = MLServe.await_ready({:fraud_detection, "2.1.0"})

@spec batch_predict(model(), [input()], keyword()) ::
  {:ok, [result()]} | {:error, reason()}

Runs inference for a list of inputs, returning results in the same order.

When the backend implements MLServe.Model.batch_predict/2 this is a single backend call, which is where the win is — one round-trip and one vectorised computation instead of N. Otherwise MLServe maps MLServe.Model.predict/2 over the inputs and stops at the first error, since a batch result is all-or-nothing.

This is for one caller with many inputs. To coalesce many concurrent callers into shared batches, configure :batching instead — see MLServe.Batcher.

Parameters

• name: the registered model name
• inputs: a list of inputs
• opts: as predict/3, except :cache, which does not apply to batches

Examples

MLServe.batch_predict(:fraud_detection, [%{amount: 10}, %{amount: 20_000}])
#=> {:ok, [%{prediction: :legitimate, ...}, %{prediction: :fraud, ...}]}

@spec batch_predict!(model(), [input()], keyword()) :: [result()]

Same as batch_predict/3 but raises MLServe.Error on failure.

@spec canary(model(), version(), 1..100) :: :ok | {:error, term()}

Routes percent of unpinned traffic to version for progressive rollout.

Each request rolls independently in the calling process, so there is no coordination point and no shared counter. Telemetry metadata carries both version and canary?, which is what makes the rollout decidable: your metrics backend can compare error rate and latency per version without extra instrumentation.

A candidate that is not ready never black-holes its share — those requests fall back to the default version.

Parameters

• name: the model name
• version: the candidate version, already loaded
• percent: 1..100

Examples

MLServe.canary(:fraud_detection, "2.1.0", 5)   # 5% of traffic
# ... watch [:ml_serve, :prediction, :stop] grouped by version ...
MLServe.promote(:fraud_detection, "2.1.0")     # ship it, clears the canary

@spec clear_canary(model()) :: :ok

Aborts an in-progress canary, sending all unpinned traffic back to the default version.

@spec load_model(
  model(),
  keyword()
) :: {:ok, {model(), version()}} | {:error, reason()}

Registers and loads a model.

Returns as soon as the model is registered; loading proceeds in the background and the model reports :loading until it is ready. Use await_ready/2 to block, or ready?/1 for a readiness probe. A load failure is retried with backoff before the model is marked :failed.

Loading a version that is already loaded returns {:error, {:already_loaded, name, version}}. Loading a different version of the same name is how zero-downtime upgrades start: both run side by side until you promote/2.

Parameters

• name: the atom to register the model under
• opts: the model options documented in MLServe.Config

Examples

MLServe.load_model(:fraud_detection,
  backend: MyApp.FraudModel,
  version: "2.1.0",
  workers: 8,
  config: [threshold: 0.8]
)
#=> {:ok, {:fraud_detection, "2.1.0"}}

@spec model_status(
  model(),
  keyword()
) :: {:ok, map()} | {:error, :model_not_found}

Returns operational status for a model version.

Parameters

• name: the model name
• opts: :version, defaulting to the current default version

Examples

MLServe.model_status(:fraud_detection)
#=> {:ok, %{
#=>   name: :fraud_detection,
#=>   version: "1.0.0",
#=>   status: :ready,
#=>   backend: MyApp.FraudModel,
#=>   workers: 4,
#=>   concurrency: :exclusive,
#=>   in_flight: 3,
#=>   requests: 154223,
#=>   errors: 12,
#=>   default?: true,
#=>   canary: nil,
#=>   ...
#=> }}

requests, errors and in_flight come from atomic counters updated on the hot path — no process is asked, and reading them costs nothing.

@spec models() :: [model()]

Lists every registered model name.

Examples

MLServe.models()
#=> [:fraud_detection, :recommendations]

@spec predict(model(), input(), keyword()) :: {:ok, result()} | {:error, reason()}

Runs inference for a single input.

Executes in the calling process up to the point of dispatch: cache lookup, the :preprocess hook and admission control all happen here, so a worker is occupied only for actual inference.

Parameters

• name: the registered model name
• input: whatever the backend accepts
• opts: see below

Options

• :version — pin an exact version. Without it, routing follows the default version and any active canary.
• :timeout — milliseconds, defaulting to the model's :timeout. The deadline travels with the request: a worker that picks up already-expired work drops it instead of running it. Not enforced for concurrency: :shared backends — see the note below.
• :cache — true, false, or a TTL in milliseconds. Off unless asked.
• :cache_key — an explicit cache key. Use it when hashing the full input is more expensive than the inference you are skipping.
• :cache_ttl — TTL override in milliseconds.

Examples

MLServe.predict(:fraud_detection, %{amount: 1500.50})
#=> {:ok, %{prediction: :fraud, probability: 0.94}}

MLServe.predict(:fraud_detection, features, version: "2.1.0", timeout: 250)

MLServe.predict(:embeddings, "some text", cache: true, cache_ttl: :timer.minutes(10))

@spec predict!(model(), input(), keyword()) :: result()

Same as predict/3 but raises MLServe.Error instead of returning {:error, reason}.

Examples

MLServe.predict!(:fraud_detection, features)
#=> %{prediction: :fraud, probability: 0.94}

@spec promote(model(), version()) :: :ok | {:error, :model_not_found}

Makes version the default for unpinned traffic, and clears any canary.

The switch is a single ETS write: in-flight requests finish on whatever version they started on, and the next request routes to the new one. No restart, no downtime.

Examples

MLServe.load_model(:fraud_detection, backend: MyApp.FraudModel, version: "2.1.0")
MLServe.await_ready(:fraud_detection)
MLServe.promote(:fraud_detection, "2.1.0")
MLServe.unload_model(:fraud_detection, version: "1.0.0")

@spec ready?(model() | {model(), version()} | :all) :: boolean()

Returns true when the model is loaded and serving.

Accepts a model name, a {name, version} tuple to check one exact version, or :all.

With :all (the default), returns true only when every registered model is ready — which is what a container readiness probe wants:

# in a Phoenix router
get "/health/ready", HealthController, :ready

def ready(conn, _params) do
  if MLServe.ready?(), do: send_resp(conn, 200, "ok"), else: send_resp(conn, 503, "loading")
end

@spec reload_model(
  model(),
  keyword()
) :: {:ok, {model(), version()}} | {:error, reason()}

Reloads a model version in place, picking up a changed artifact or configuration.

Equivalent to unload_model/2 followed by load_model/2 with the same options, which means it is a gap in availability for that version. For a zero-downtime change, load a new version alongside and promote/2 instead.

Parameters

• name: the model name
• opts: :version, plus any model options to override

@spec unload_model(
  model(),
  keyword()
) :: :ok | {:error, reason()}

Drains and unloads a model version.

New requests stop routing to it immediately. Requests already accepted are given up to :drain_timeout to finish before the workers are terminated, and the number still outstanding is reported as the drained measurement on [:ml_serve, :model, :unload].

If the unloaded version was the default and other versions remain, the newest survivor becomes the default so unpinned calls keep working.

Parameters

• name: the model name
• opts: :version (defaults to the current default version), :timeout (drain timeout)

Examples

MLServe.unload_model(:fraud_detection, version: "1.0.0")
#=> :ok

@spec versions(model()) :: {:ok, [version()]} | {:error, :model_not_found}

Lists every loaded version of a model, oldest first.

Examples

MLServe.versions(:fraud_detection)
#=> {:ok, ["1.0.0", "2.1.0"]}