MLServe.Model behaviour (MLServe v0.1.0)

The behaviour every MLServe model backend implements.

A backend is the adapter between MLServe's runtime — supervision, pooling, batching, caching, telemetry — and an actual inference engine. MLServe never interprets a model artifact itself; it hands the configuration to load/1 and calls predict/2 with whatever state you return.

Only load/1 and predict/2 are required. The smallest useful backend is two functions:

defmodule MyApp.Backends.Threshold do
  @behaviour MLServe.Model

  @impl true
  def load(config), do: {:ok, Keyword.fetch!(config, :threshold)}

  @impl true
  def predict(threshold, %{score: score}) do
    {:ok, %{fraud?: score > threshold}}
  end
end

Capabilities

capabilities/0 tells MLServe how the backend may be executed. It is the single most important thing a backend declares, because it changes the execution strategy entirely:

`:concurrency`	Effect
`:exclusive` (default)	State lives in pooled `MLServe.Worker` processes; one request per worker at a time. Use for anything not safe to call concurrently — ONNX Runtime sessions, ports, stateful handles.
`:shared`	`predict/2` runs in the calling process and state is read from `:persistent_term`. No worker processes are started and large inputs are never copied between processes. Use for `Nx.Serving`, pure functions, and remote HTTP services.

`:load`	Effect
`:once` (default)	`load/1` is called once and the resulting state term is shared by every worker. Correct for NIF-resource-backed models, where the term is a cheap handle and the real memory is shared.
`:per_worker`	`load/1` is called once per worker, each getting independent state. Correct for ports and per-worker sessions.

Choosing :once by default matters: loading a 2 GB model separately into eight workers is an out-of-memory crash, not a pool.

Optional callbacks

MLServe detects optional callbacks with function_exported?/3, so implementing one is enough — there is no registration step.

batch_predict/2 — receives a list of inputs and returns a list of results in the same order. Implement it whenever the engine can vectorise; both MLServe.batch_predict/3 and dynamic batching route through it. Without it MLServe maps predict/2 over the inputs. Return {:error, :not_supported} to fall back to that mapping for a particular model.
unload/1 — release ports, files, or NIF resources. Called on unload and on reload.
metadata/1 — arbitrary map surfaced under :metadata in MLServe.model_status/2. Good place for input shapes, label names, or a training run id.
capabilities/0 — as described above.

Error handling

Return {:error, reason} for expected failures. MLServe also traps raises, throws and exits around every callback and converts them to {:error, {:backend_error, %MLServe.BackendError{}}} with the original exception and stacktrace preserved — a backend never needs its own try purely to protect the runtime.

See the Creating a Model Backend guide for complete Nx, Bumblebee, ONNX and Python-port implementations.

Summary

Types

capabilities()

How MLServe may execute this backend.

input()

A single inference input, in whatever shape the backend accepts.

result()

A single inference result, in whatever shape the backend produces.

state()

Backend-private state returned by load/1 and passed to every other callback.

Callbacks

batch_predict(state, list)

Runs inference for a list of inputs, returning results in the same order.

capabilities()

Declares how MLServe may execute this backend. Optional; defaults to %{concurrency: :exclusive, load: :once}.

load(config)

Loads the model and returns its state.

metadata(state)

Returns a map of backend-specific information for MLServe.model_status/2. Optional.

predict(state, input)

Runs inference for a single input.

unload(state)

Releases any resources held by the state. Optional.

Types

capabilities()

@type capabilities() :: %{
  optional(:concurrency) => :shared | :exclusive,
  optional(:load) => :once | :per_worker
}

How MLServe may execute this backend.

input()

@type input() :: term()

A single inference input, in whatever shape the backend accepts.

result()

@type result() :: term()

A single inference result, in whatever shape the backend produces.

state()

@type state() :: term()

Backend-private state returned by load/1 and passed to every other callback.

Callbacks

batch_predict(state, list)

(optional)

@callback batch_predict(state(), [input()]) :: {:ok, [result()]} | {:error, term()}

Runs inference for a list of inputs, returning results in the same order.

Optional. When absent MLServe maps predict/2 over the inputs.

capabilities()

(optional)

@callback capabilities() :: capabilities()

Declares how MLServe may execute this backend. Optional; defaults to %{concurrency: :exclusive, load: :once}.

load(config)

@callback load(config :: keyword()) :: {:ok, state()} | {:error, term()}

Loads the model and returns its state.

Receives the model's :config keyword list, with :path (already validated and expanded when configured) and :version injected. Called once, or once per worker when capabilities/0 declares load: :per_worker.

metadata(state)

(optional)

@callback metadata(state()) :: map()

Returns a map of backend-specific information for MLServe.model_status/2. Optional.

predict(state, input)

@callback predict(state(), input()) :: {:ok, result()} | {:error, term()}

Runs inference for a single input.

unload(state)

(optional)

@callback unload(state()) :: :ok

Releases any resources held by the state. Optional.