MLServe.Backend (MLServe v0.1.3)

Guarded invocation of MLServe.Model callbacks.

Every call into a backend goes through this module. It exists for three reasons:

Nothing raises out of a backend into MLServe's runtime. A raise, throw or exit is converted to {:error, {:backend_error, %MLServe.BackendError{}}} carrying the original exception and stacktrace. Errors are surfaced, never swallowed — the point of catching is to attach context, not to hide the failure.
Optional callbacks are resolved once. MLServe.Model.batch_predict/2 and friends are detected with function_exported?/3 and given sane fallbacks.
Return values are validated. A backend that returns a bare value instead of an {:ok, term} tuple gets a clear error naming the callback, rather than a confusing mismatch three layers up.

Application code does not normally call this module; it is public because backend authors and the guides refer to its semantics.

Summary

Functions

batch_predict(spec, state, inputs)

Calls MLServe.Model.batch_predict/2, falling back to a mapped MLServe.Model.predict/2.

load(spec)

Calls MLServe.Model.load/1.

metadata(spec, state)

Calls MLServe.Model.metadata/1 when exported, returning %{} otherwise.

predict(spec, state, input)

Calls MLServe.Model.predict/2.

supports_batching?(backend)

Returns whether the backend exports MLServe.Model.batch_predict/2.

unload(spec, state)

Calls MLServe.Model.unload/1 when exported. Always returns :ok.

Types

invocation_result()

@type invocation_result() :: {:ok, term()} | {:error, term()}

Functions

batch_predict(spec, state, inputs)

@spec batch_predict(MLServe.ModelSpec.t(), term(), [term()]) ::
  {:ok, [term()]} | {:error, term()}

Calls MLServe.Model.batch_predict/2, falling back to a mapped MLServe.Model.predict/2.

The fallback stops at the first error rather than running the remaining inputs, since a batch result is all-or-nothing. Backends that can vectorise should implement batch_predict/2; the whole point of MLServe.batch_predict/3 is one backend round-trip instead of N.

A backend that exports batch_predict/2 but cannot batch this particular model returns {:error, :not_supported} to opt back into the mapped fallback.