MLServe.ModelSpec (MLServe v0.1.3)

Copy Markdown View Source

The normalised, validated description of one loaded model version.

MLServe.Config.build/2 turns a keyword list — from config :ml_serve, :models or from MLServe.load_model/2 — into this struct. Everything downstream (the registry, the supervisor, the workers, the dispatcher) reads the struct rather than re-interpreting raw options, so there is exactly one place where option handling lives.

A spec is immutable for the lifetime of a loaded model version. Mutable runtime facts — status, worker pids, request counters — live in MLServe.ModelRegistry, not here.

Summary

Types

hook()

An {module, function, extra_args} hook invoked in the calling process.

t()

Functions

key(model_spec)

Returns the registry key identifying this model version.

shared?(model_spec)

Returns true when this model runs inference in the calling process rather than in a worker pool.

worker_count(model_spec)

Returns the number of worker processes this model will start.

Types

hook()

@type hook() :: {module(), atom(), list()} | (term() -> term()) | nil

An {module, function, extra_args} hook invoked in the calling process.

t()

@type t() :: %MLServe.ModelSpec{
  backend: module(),
  batching: %{max_size: pos_integer(), timeout: pos_integer()} | nil,
  cache: %{enabled: boolean(), ttl: pos_integer()} | nil,
  checksum: {:sha256 | :sha512, String.t()} | nil,
  concurrency: :shared | :exclusive,
  config: keyword(),
  drain_timeout: timeout(),
  load: :once | :per_worker,
  max_batch_size: pos_integer(),
  max_concurrency: pos_integer() | :infinity,
  name: atom(),
  path: String.t() | nil,
  postprocess: hook(),
  preprocess: hook(),
  restart_on_error: boolean(),
  selection: :round_robin | :least_loaded | :random,
  timeout: timeout(),
  version: String.t(),
  workers: pos_integer()
}

Functions

key(model_spec)

@spec key(t()) :: {atom(), String.t()}

Returns the registry key identifying this model version.

Examples

iex> spec = %MLServe.ModelSpec{name: :fraud, version: "1.0.0"}
iex> MLServe.ModelSpec.key(spec)
{:fraud, "1.0.0"}

shared?(model_spec)

@spec shared?(t()) :: boolean()

Returns true when this model runs inference in the calling process rather than in a worker pool.

Examples

iex> MLServe.ModelSpec.shared?(%MLServe.ModelSpec{concurrency: :shared})
true

iex> MLServe.ModelSpec.shared?(%MLServe.ModelSpec{concurrency: :exclusive})
false

worker_count(model_spec)

@spec worker_count(t()) :: non_neg_integer()

Returns the number of worker processes this model will start.

Shared-concurrency models start none: inference runs in the caller.

Examples

iex> MLServe.ModelSpec.worker_count(%MLServe.ModelSpec{concurrency: :shared, workers: 8})
0

iex> MLServe.ModelSpec.worker_count(%MLServe.ModelSpec{concurrency: :exclusive, workers: 8})
8