Batch Inference

Two different things share the word "batch", and MLServe supports both because they solve different problems.

Explicit batching

MLServe.batch_predict(:fraud_detection, [features_a, features_b, features_c])
#=> {:ok, [result_a, result_b, result_c]}

Results come back in input order. When the backend implements MLServe.Model.batch_predict/2 this is a single backend call — one round-trip and one vectorised computation instead of N, which is the entire point.

Without that callback, MLServe maps predict/2 over the inputs and stops at the first error, since a batch result is all-or-nothing.

Check which you are getting:

MLServe.model_status(:fraud_detection)
#=> {:ok, %{native_batching: true, ...}}

Implementing it

@impl true
def batch_predict(state, inputs) do
  tensor = inputs |> Enum.map(&encode/1) |> Nx.tensor(type: :f32)
  {probabilities} = Ortex.run(state.session, {tensor})
  {:ok, probabilities |> Nx.to_flat_list() |> Enum.map(&decode/1)}
end

Two rules, both enforced: results must be in input order and of equal length. A backend returning the wrong count gets a clear MLServe.BackendError naming the mismatch rather than handing callers each other's results.

A backend that exports batch_predict/2 but cannot batch a particular model returns {:error, :not_supported} to fall back to the mapped path.

Limits

config :ml_serve, max_batch_size: 1_000
models: [fraud_detection: [max_batch_size: 256]]

Oversized batches are rejected before any work happens:

MLServe.batch_predict(:fraud_detection, Enum.to_list(1..5_000))
#=> {:error, {:batch_too_large, 256}}

This is a real safety limit, not a formality: an unbounded batch is an unbounded allocation, and one request can otherwise exhaust the memory of the node.

Dynamic batching

The far more common production shape is many independent callers, each with a single input, arriving within milliseconds of each other. A GPU that processes 32 rows in barely more time than one row is being wasted by a pool feeding it one row at a time.

models: [
  fraud_detection: [
    backend: MyApp.Backends.ONNX,
    workers: 2,
    batching: [max_size: 16, timeout: 10]
  ]
]

Nothing changes at the call site. Independent predict/3 calls are transparently coalesced:

# 16 separate Phoenix requests, arriving within 10ms → one backend call
MLServe.predict(:fraud_detection, features)

The window

A batch flushes when either trigger fires:

• :max_size inputs have accumulated → reason: :full
• :timeout milliseconds have passed since the first input → reason: :timeout

Timing from the first input rather than the last bounds the added latency at :timeout for every caller. A sliding window timed from the most recent arrival can starve the earliest caller indefinitely under steady traffic.

Tuning it

Watch [:ml_serve, :batch, :flush]:

:telemetry.attach("batch-watch", [:ml_serve, :batch, :flush], fn _e, m, meta, _c ->
  IO.puts("#{meta.model}: #{m.size} rows on #{meta.reason}")
end, nil)

Start with max_size matching the batch size your model was compiled for, and timeout at the latency budget you can spare — 5–20ms is typical for an interactive request path.

Why it does not block

Running inference inside the batcher would stop it accumulating the next batch for the whole duration of the current one, serialising exactly what it was built to parallelise. Instead a flush hands the batch to a task under MLServe.TaskSupervisor, which calls a worker and replies to every caller with GenServer.reply/2. The batcher stays responsive throughout.

Backpressure

In-flight batches are capped at the model's worker count. Beyond that, flushes wait — there is no free worker to take them, and queueing more would only build a backlog of work whose callers will have timed out by the time it runs. That cap is the backpressure.

Expired requests are dropped from a batch at flush time rather than being sent to the backend.

Batching and batch_predict/3 together

MLServe.batch_predict/3 bypasses the batcher and calls the backend directly. That is correct: you already have a full batch, so there is nothing to wait for.

Choosing

Batching is not free. It adds up to :timeout milliseconds of latency to every request. Enable it when the per-call overhead of your backend genuinely dominates the per-row cost, which is the case for GPUs and rarely the case for a small CPU model.

Chunking large jobs

def score_all(rows) do
  rows
  |> Stream.chunk_every(256)
  |> Task.async_stream(
    fn chunk -> MLServe.batch_predict(:fraud_detection, chunk, timeout: 30_000) end,
    max_concurrency: 4,
    timeout: 35_000
  )
  |> Enum.reduce({:ok, []}, fn
    {:ok, {:ok, results}}, {:ok, acc} -> {:ok, acc ++ results}
    {:ok, {:error, reason}}, _acc -> {:error, reason}
    {:exit, reason}, _acc -> {:error, {:exit, reason}}
  end)
end

Keep max_concurrency at or below the model's worker count — going higher just queues work that :max_concurrency may then reject as :overloaded.