Reads and validates MLServe configuration.
Two jobs: expose the application-wide settings as typed accessors, and turn a per-model keyword
list into a validated MLServe.ModelSpec. The same build/2 path serves both static
configuration and runtime MLServe.load_model/2, so a model registered at runtime is validated
exactly as strictly as one declared in config.exs.
Validation is hand-rolled rather than delegated to a schema library. MLServe has one runtime dependency on purpose, and a few hundred lines of explicit validation buys clearer error messages than a generic schema violation.
Application configuration
config :ml_serve,
model_root: "priv/models",
start_mode: :async,
default_timeout: 5_000,
max_batch_size: 1_000,
max_model_bytes: 2_147_483_648,
cache: [enabled: true, max_size: 10_000, ttl: :timer.minutes(5)],
models: [
fraud_detection: [backend: MyApp.Backends.ONNX, path: "fraud.onnx", workers: 4]
]Model options
| Option | Default | Meaning |
|---|---|---|
:backend | required | Module implementing MLServe.Model |
:version | "1.0.0" | Version string; models are keyed by {name, version} |
:path | nil | Model artifact, validated by MLServe.Security |
:checksum | nil | {:sha256, hex} integrity check |
:workers | System.schedulers_online() | Pool size; ignored for :shared backends |
:timeout | :default_timeout | Per-request inference timeout |
:drain_timeout | 5_000 | How long unload_model/2 waits for in-flight requests |
:max_concurrency | :infinity | Admission limit; excess returns {:error, :overloaded} |
:max_batch_size | :max_batch_size | Largest accepted batch_predict/3 list |
:batching | nil | [max_size: 16, timeout: 10] enables dynamic batching |
:cache | nil | [enabled: true, ttl: 60_000] |
:preprocess / :postprocess | nil | {Mod, :fun, args} hooks run in the caller |
:restart_on_error | false | Crash the worker on a backend exception instead of returning it |
:selection | :round_robin | :round_robin, :least_loaded or :random |
:config | [] | Opaque keyword list passed to the backend's load/1 |
Summary
Functions
Builds a validated MLServe.ModelSpec from a model name and options.
Same as build/2 but raises MLServe.Error on invalid configuration.
Inference cache settings.
Models declared in application configuration, as {name, opts} pairs.
Default per-request inference timeout in milliseconds.
Default upper bound on MLServe.batch_predict/3 list length.
Largest model artifact MLServe.Security will accept, in bytes.
Root directory model paths must resolve inside. Defaults to priv/models of the app.
Whether models declared in configuration load in the background (:async) or block boot.
Functions
@spec build( atom(), keyword() ) :: {:ok, MLServe.ModelSpec.t()} | {:error, MLServe.Error.t()}
Builds a validated MLServe.ModelSpec from a model name and options.
Parameters
name: the model's atom nameopts: the model options documented in the moduledoc
Examples
iex> {:ok, spec} = MLServe.Config.build(:fraud, backend: MLServe.Backend.Static, config: [result: :ok])
iex> {spec.name, spec.version, spec.backend}
{:fraud, "1.0.0", MLServe.Backend.Static}@spec build!( atom(), keyword() ) :: MLServe.ModelSpec.t()
Same as build/2 but raises MLServe.Error on invalid configuration.
@spec cache() :: %{ enabled: boolean(), max_size: pos_integer(), ttl: pos_integer(), sweep_interval: pos_integer() }
Inference cache settings.
@spec configured_models() :: keyword()
Models declared in application configuration, as {name, opts} pairs.
@spec default_timeout() :: timeout()
Default per-request inference timeout in milliseconds.
@spec max_batch_size() :: pos_integer()
Default upper bound on MLServe.batch_predict/3 list length.
@spec max_model_bytes() :: pos_integer() | :infinity
Largest model artifact MLServe.Security will accept, in bytes.
@spec model_root() :: String.t()
Root directory model paths must resolve inside. Defaults to priv/models of the app.
@spec start_mode() :: :async | :sync
Whether models declared in configuration load in the background (:async) or block boot.