Coalesces concurrent single predictions into one backend batch call.

Started only for models configured with batching: [max_size: 16, timeout: 10].

Why this exists

MLServe.batch_predict/3 helps when one caller has many inputs. Dynamic batching helps the far more common production shape: many independent callers, each with one input, arriving within milliseconds of each other. A GPU that processes 32 rows in barely more time than one row is being wasted by a pool that feeds it one row at a time. This process collects arrivals into a window and hands the backend a full batch.

The window

A batch flushes when either trigger fires:

• :max_size inputs have accumulated (reason: :full)
• :timeout milliseconds have passed since the first input in the batch (reason: :timeout)

Timing from the first input rather than the last bounds the added latency at :timeout for every caller. A sliding window timed from the last arrival can starve the earliest caller indefinitely under steady traffic.

Watch [:ml_serve, :batch, :flush]: a healthy configuration flushes mostly on :full. Mostly :timeout means the window is longer than your arrival rate justifies, and you are adding latency for batches that never fill.

Why the batcher never blocks

Running inference inside handle_info(:flush, ...) would stop the batcher accumulating the next batch for the whole duration of the current one — serialising exactly what it was built to parallelise. Instead a flush hands the batch to a task under MLServe.TaskSupervisor, which calls a worker and replies to every caller with GenServer.reply/2.

Backpressure

In-flight batches are capped at the model's worker count. Beyond that, flushes wait: there is no worker free to take them, and queueing more would only build an unbounded backlog of work whose callers will have timed out by the time it runs. That cap is the backpressure.