Request lifecycle

This is the contributor-facing walkthrough of a single inference request. The public usage patterns live in Examples; this page explains how admission, cache restore, batching, decode, and saving fit together inside the model process.

Model process states

Each loaded model is a supervised gen_statem with one underlying llama_context*. It has two broad runtime states:

• idle when no request is active.
• running when one or more requests are prefilling, decoding, or queued for the multi-sequence scheduler.

The default context has n_seq_max => 1, which keeps single-request behavior simple and deterministic. Setting context_opts.n_seq_max > 1 opts into concurrent prefill/decode across multiple active sequences.

Admission

Every request starts by taking a sequence id from the model's idle pool. If the caller supplied a session_id, the model tries to reuse that session's pinned sequence instead. If the session is already active in another request, admission returns {error, sticky_busy}.

continue/3 is a separate admission path used by callers that have a pinned session and want to extend its live KV with a caller-asserted token tail. It short-circuits the cache resolution stage entirely: no parent_key lookup, no longest-prefix walk, no prompt prefix-equality check. The only admission checks are that the session_id is known (otherwise {error, no_session}) and that the pinned seq is not in flight (otherwise {error, sticky_busy}).

At admission, the model also prepares request-local state: token cursor, sampling options, optional grammar, optional thinking/tool-call markers, and the reply destination for streaming calls.

Cache resolution

The admitted request resolves its warm prefix before any decode work starts. The result is reported as cache_hit_kind:

The cache is byte-addressed (ds4-style): the key is SHA-256 over the rendered prompt bytes (detokenize(tokens)), not the token-id list. The lookup order is:

1. Exact byte-key hit on the whole rendered prompt.
2. Supplied parent_key, if present (session fast path), accepted when its rendered bytes are a byte-prefix of this prompt; it also waits up to session_resume_wait_ms for an in-flight finish save.
3. Longest-byte-prefix lookup over the new prompt's rendered bytes.

Any failure of 1 or 2 falls through to 3; only a byte-prefix miss is cold. The longest-byte-prefix lookup is still exact: it picks the longest stored rendered-byte prefix whose recomputed SHA-256 key matches and resumes from that checkpoint's exact tokens. The suffix beyond the match reuses the caller's ORIGINAL tokens when the byte boundary lands on a token boundary (token-exact, the common case); only a mid-token boundary re-tokenises the byte remainder (checkpoint_tokens ++ tokenize(byte_suffix) - identical byte stream, ds4's contract).

Restore or prefill

After cache resolution, the request takes one of four paths:

• Warm restore. kv_unpack restores the saved KV bytes into the sequence. The model then prefills the last token again as a logits primer so generation starts from fresh logits.
• Cold prefill. The sequence is cleared and the full prompt is prefed.
• Sticky continuation. The live sequence already holds the prefix, so the model truncates or extends in place and only prefills the new suffix.
• Caller-asserted continuation. continue/3 enters with the session's stored tokens already resident on the pinned seq. The supplied suffix is queued for prefill without any prefix equality check; correctness is the caller's contract.

Long prompts are sliced by prefill_chunk_size, defaulting to max(64, n_batch div 4), so one large prefill cannot monopolize every scheduler tick when other sequences are decoding.

Decode

The scheduler batches active rows into one llama_decode call per tick, bounded by n_batch. A tick can contain a mix of prefill tokens and decode tokens across different sequences.

Request-local samplers stay isolated. One request can use a grammar or a fixed seed while another uses a different temperature. Tool-call and thinking markers are recognized inline and surfaced on the streaming wire without changing unrelated requests.

Save points

The model fires cache saves at stable boundaries:

Async saves go through erllama_cache_writer. Synchronous saves block the caller because the backing resource is about to be released.

Disk publication is handled by the reservation protocol described in Publish protocol.

Completion

When a request finishes, the model reports final stats, including:

• finish_reason
• finish_key
• cache_hit_kind
• cache_delta
• cancellation state, when applicable

One-shot requests release their sequence back to the idle pool. Sticky requests keep the sequence pinned until a later request diverges or the caller invokes erllama:end_session/2.

Observability

Routers can inspect model state without crossing the model gen_statem:

erllama:phase(ModelId).
erllama:pending_len(ModelId).
erllama:queue_depth(ModelId).
erllama:last_cache_hit(ModelId).

These read a public ETS row and are safe to call from a hot path.