The slim record read from ETS on every prediction.
Deliberately separate from MLServe.ModelSpec. A spec carries the backend's :config — which
may hold closures, large keyword lists, or preprocessing tables — and reading it out of ETS
copies the whole thing into the calling process. A route carries only what dispatch needs, so
the per-prediction copy stays at a few dozen words.
Everything here is either an atom, a small integer, or a reference. Nothing that grows with model size ever enters this struct.
Summary
Functions
Reserves an admission slot, returning :ok or {:error, :overloaded}.
Increments the in-flight request count.
Total failed requests since the model was loaded.
Builds a route from a validated spec.
Current number of in-flight requests.
Decrements the in-flight request count, recording an error when ok? is false.
Atomically increments and returns the next round-robin worker index, 0-based.
Total requests accepted since the model was loaded.
The :persistent_term key under which a shared backend's state is stored.
Types
@type t() :: %MLServe.Route{ atomics: :atomics.atomics_ref() | nil, backend: module(), batching: boolean(), cache: %{enabled: boolean(), ttl: pos_integer()} | nil, canary?: boolean(), concurrency: :shared | :exclusive, counters: :counters.counters_ref() | nil, max_batch_size: pos_integer(), max_concurrency: pos_integer() | :infinity, name: atom(), postprocess: MLServe.ModelSpec.hook(), preprocess: MLServe.ModelSpec.hook(), restart_on_error: boolean(), selection: :round_robin | :least_loaded | :random, status: :loading | :ready | :draining | :failed | :unloading, timeout: timeout(), version: String.t(), workers: non_neg_integer() }
Functions
@spec admit(t()) :: :ok | {:error, :overloaded}
Reserves an admission slot, returning :ok or {:error, :overloaded}.
Admission control is a compare-then-increment on the in-flight counter with no lock and no
process. Under a race two callers can both observe max - 1 and both proceed, so the limit is
approximate at the boundary. That is the right trade: an exact limiter would need serialisation
through a process, which is the very bottleneck the limit exists to prevent.
@spec enter(t()) :: :ok
Increments the in-flight request count.
@spec errors(t() | :counters.counters_ref()) :: non_neg_integer()
Total failed requests since the model was loaded.
@spec from_spec( MLServe.ModelSpec.t(), keyword() ) :: t()
Builds a route from a validated spec.
@spec in_flight(t() | :counters.counters_ref()) :: non_neg_integer()
Current number of in-flight requests.
Decrements the in-flight request count, recording an error when ok? is false.
@spec next_index(t()) :: non_neg_integer()
Atomically increments and returns the next round-robin worker index, 0-based.
@spec requests(t() | :counters.counters_ref()) :: non_neg_integer()
Total requests accepted since the model was loaded.
The :persistent_term key under which a shared backend's state is stored.