Model Versioning

Models change more often than the applications that serve them, and a retrained model should not require a deploy. MLServe keys models by {name, version} so several versions run side by side, and routing between them is a pointer you move.

Versions are independent models

MLServe.load_model(:fraud_detection, backend: MyApp.Backends.ONNX, path: "fraud-v1.onnx", version: "1.0.0")
MLServe.load_model(:fraud_detection, backend: MyApp.Backends.ONNX, path: "fraud-v2.onnx", version: "2.1.0")

Each gets its own supervision subtree, its own workers, its own counters and its own telemetry tags. They share nothing but a name.

MLServe.versions(:fraud_detection)
#=> {:ok, ["1.0.0", "2.1.0"]}

MLServe.predict(:fraud_detection, features, version: "2.1.0")   # pinned
MLServe.predict(:fraud_detection, features)                     # default version

The first version registered under a name becomes the default. Loading another does not move it — promotion is always explicit.

The rollout

# 1. Load the candidate beside the live model. Nothing changes for callers yet.
{:ok, _} = MLServe.load_model(:fraud_detection,
  backend: MyApp.Backends.ONNX,
  path: "fraud-v2.onnx",
  version: "2.1.0",
  workers: 4
)

:ok = MLServe.await_ready({:fraud_detection, "2.1.0"})

# 2. Send it a slice of traffic.
:ok = MLServe.canary(:fraud_detection, "2.1.0", 5)

# 3. Watch. Telemetry is already tagged by version.

# 4. Promote — an atomic pointer flip. No restart, no dropped request.
:ok = MLServe.promote(:fraud_detection, "2.1.0")

# 5. Drain and remove the old version.
:ok = MLServe.unload_model(:fraud_detection, version: "1.0.0")

At no point does the application restart, and at no point does a request fail because of the change. MLServe's own test suite asserts exactly this: 600 concurrent predictions across a full load → canary → promote → drain cycle, with zero failures.

Canary routing

MLServe.canary(:fraud_detection, "2.1.0", 5)     # 5% of unpinned traffic
MLServe.clear_canary(:fraud_detection)           # abort

Each request rolls independently in the calling process against its own random seed. There is no shared counter and no coordination point, so the split costs nothing and cannot become a bottleneck.

Rules worth knowing:

• Pinned requests ignore the canary. version: "1.0.0" goes exactly there.
• A candidate that is not ready never black-holes its share. If it is still loading or has failed, those requests fall back to the default version.
• promote/2 clears the canary. Promoting is the end of the rollout, not another state to clean up.
• Unloading the candidate clears the canary too.

Deciding with telemetry

Every prediction event carries version and canary?, so per-version comparison needs no extra instrumentation. That is the whole reason the canary exists — a rollout you cannot measure is just a slower way to ship a bad model.

:telemetry.attach_many("canary-watch", [[:ml_serve, :prediction, :stop]], fn _e, m, meta, _c ->
  :telemetry.execute(
    [:my_app, :inference],
    %{duration: m.duration},
    %{model: meta.model, version: meta.version, canary: meta.canary?, result: meta.result}
  )
end, nil)

With MLServe.Telemetry.Metrics.metrics/0, latency and error-rate series are already split by version. Compare, then promote or clear.

What to look at before promoting:

Graceful drain

unload_model/2 does not kill in-flight work:

1. The version is marked :draining, so the registry stops routing new requests to it.
2. MLServe waits for the in-flight atomic counter to reach zero, bounded by :drain_timeout.
3. Workers terminate; the backend's unload/1 runs; cached entries for that version are dropped.
4. [:ml_serve, :model, :unload] reports drained — the number still outstanding when the wait ended. 0 is a clean drain.

models: [fraud_detection: [drain_timeout: 15_000]]

Set :drain_timeout above your p99 inference latency. Alert on drained > 0: it means a deploy abandoned live requests.

If you unload the version that is currently the default and others remain, the newest survivor becomes the default automatically, so an unpinned predict/3 keeps working rather than starting to return :model_not_found.

Version strings

Any non-empty string works. MLServe.versions/1 sorts dot- and dash-separated integer components numerically, so "10.0.0" correctly follows "9.0.0" rather than sorting before it. Anything non-numeric falls back to string ordering.

Semver is the obvious choice, but a training run id or a date works equally well:

version: "2026-08-22-run-4417"

Whatever you choose, make it the thing you would want to see in an alert at 3am.

Reload versus a new version

MLServe.reload_model(:fraud_detection)                  # same version, re-read the artifact
MLServe.reload_model(:fraud_detection, workers: 8)      # with overrides

reload_model/2 is unload then load on the same version, which means it is a gap in availability. It is right for picking up a changed config or a hot-swapped file in development.

For anything user-facing, load a new version and promote. That path has no gap at all.

Loading versions at boot

config :ml_serve,
  models: [
    fraud_detection: [
      backend: MyApp.Backends.ONNX,
      path: "fraud-v2.onnx",
      version: "2.1.0",
      checksum: {:sha256, System.get_env("FRAUD_MODEL_SHA256")},
      workers: 4
    ]
  ]

Pinning the version and checksum in config makes deploys reproducible: the same release always loads the same artifact, and a mismatch fails loudly at load time rather than serving predictions from a file you did not intend to ship.

Registering versions from a database

Runtime registration goes through exactly the same validation as configuration, so a model registry table works naturally:

defmodule MyApp.ModelLoader do
  use Task, restart: :transient

  def start_link(_), do: Task.start_link(__MODULE__, :run, [])

  def run do
    for model <- MyApp.Repo.all(MyApp.ModelVersion.active()) do
      MLServe.load_model(model.name,
        backend: MyApp.Backends.ONNX,
        path: model.path,
        version: model.version,
        checksum: {:sha256, model.sha256},
        workers: model.workers
      )
    end

    for model <- MyApp.Repo.all(MyApp.ModelVersion.canaries()) do
      MLServe.canary(model.name, model.version, model.traffic_percent)
    end
  end
end

Add it to your own supervision tree after MLServe has started. Because loading is asynchronous and retried with backoff, a temporarily unreachable artifact store delays a model rather than taking your application down.