Per-model gen_statem that drives the request flow and wires the cache subsystem into the model lifecycle.

State machine

  ┌──── admit (idle_seq_ids non-empty)
  │     ┌──── admit (queues in pending when idle_seq_ids empty)
  │     │     ┌──── cast tick (self)
  ▼     ▼     │
 idle ──────▶ running ─────┐
  ▲                        │
  └────── all reqs finished (req_table = #{} AND pending = [])

Two states only:

• idle/3 — req_table is empty AND pending is empty. Accepts admit events (complete, prefill_only, infer) and transitions to running. Verifies and other read-only ops are allowed.
• running/3 — one or more #req{} are in flight. Accepts further admit events (allocate seq_id from idle_seq_ids or enqueue in pending), cancel casts, and the internal tick cast. Verify is refused with {error, busy} because it mutates the context.

Per-request lifecycle

  admit                  step_tick                step_tick               finish_req
  ─────▶ req_table       ──────▶ prefilling       ──────▶ decoding        ────────▶
         (new #req,              (prefill_cursor          (prefill_cursor
          seq_id popped           non-empty,              undefined,
          from                    backend:step             backend:step
          idle_seq_ids,           pushes slice             samples one
          sampler built,          to KV)                   token + decodes)
          warm/cold path
          chosen)

Each #req records its own seq_id, sampler_ref, prompt_tokens, context_tokens, prefill_cursor, generated, response_target, cache_hit_kind, and finishing flag. The req_table map is keyed by seq_id.

step_tick driver

Every tick (one llama_decode call) builds a co-batched op list:

• For each #req with prefill_cursor =/= undefined, append {seq_id, {prefill, Slice}}.
• For each #req with prefill_cursor =:= undefined and a sampler_ref, append {seq_id, {decode, sampler_ref}}.

The op list is bounded by total_batch_budget (context_opts.n_batch): decode rows are kept whole, prefill rows are sliced head-first until the sum fits. The NIF returns {seq_id, prefilled} or {seq_id, {token, T, EogFlag}} per row; results land back in the respective #req. Reqs that reach response_target tokens, an eog flag, or a cancel get finishing = true and finalise in the post-step finisher walk (finish_marked_reqs/2).

Per-tick batch budget

step_tick/1 enforces total tokens ≤ total_batch_budget (mirrors context_opts.n_batch, default 512). If total_batch_budget is smaller than the number of in-flight decoders, the gen_statem crashes deliberately so the supervisor restarts and the operator fixes n_batch / n_seq_max. Otherwise prefill rows are sliced head-first to fit; truncated tails resume next tick.

Chunked prefill

Each prefill row is additionally capped by the prefill_chunk_size policy knob (default max(64, n_batch div 4), or infinity to disable). The effective slice per prefill row is min(length(remaining), prefill_chunk_size, available_budget). A long prompt is therefore sliced across several ticks even when the batch budget alone would have accommodated it in one, leaving room for concurrent decoders to make progress between chunks.

Cache save reasons

• cold: fired right after a fresh prompt's prefill completes, before any decoding. Saves the trimmed prefix the policy produces from cold_save_split/2. Per-#req — each new admit fires its own cold save at most once.
• continued: fired every continued_interval tokens during decode. Per-#req, gated on the request's last_save_at.
• finish: fired when the request finishes (success, length limit, eog, or cancel). Saves the full context_tokens.
• evict / shutdown: fired by external triggers and walks every in-flight #req, firing one save per non-empty context_tokens.

All saves go through fire_save_if/5 which calls backend:kv_pack/3 against the request's seq_id and hands the binary off to erllama_cache_writer.

Concurrency contract

The gen_statem is the sole writer of the context's KV cells: every backend:step/2 and kv_pack / kv_unpack / seq_rm call runs inside a state callback, so the AGENTS.md paused-context invariant holds — kv_pack only runs between ticks when no llama_decode is in flight. Default n_seq_max => 1 collapses this to the v0.2 single-tenant flow bit-identically; opting in via context_opts.n_seq_max > 1 lets up to N requests run concurrently through one decode call per tick.

Backwards compatibility

• Public API (complete/2,3, prefill_only/2, infer/4, cancel/1, status/1, model_info/1, verify/4, etc.) is unchanged.
• Default n_seq_max => 1 keeps single-tenant behaviour bit-identical to v0.2; multi-tenancy is opt-in.
• phase on the obs row and in model_info/1 is still idle | prefilling | generating, computed from the dominant phase across in-flight reqs (dominant_phase/1).