-spec cached_prefix_len(model(), [token_id()]) -> {ok, non_neg_integer()} | {error, error_reason()}.

How many bytes of the detokenised PromptTokens are already cached for this model, across all tiers. {ok, 0} when nothing matches. Attached adapters are honoured through the effective fingerprint.

-spec cancel(reference()) -> ok.

Cancel an in-flight streaming request. Idempotent. The caller still receives the terminal {erllama, Ref, {done, Stats}} with finish_reason => cancelled. The running decode is interrupted through the backend's abort callback; an interrupt that fires recreates the context in place, which also resets co-batched requests on other seqs. The persistent cache is untouched.

-spec chat(model(), [chat_message()], chat_opts()) -> {ok, chat_result()} | {error, error_reason()}.

One chat turn: render Messages (and the tools in Opts) through the model's template, generate, and parse the output into a structured assistant message with content, reasoning and tool calls. Messages and tools are Erlang maps; see chat_message() and chat_tool().

{ok, #{message := #{content := Text, tool_calls := Calls}}} =
    erllama:chat(Model, [#{role => user, content => <<"hi">>}],
                 #{tools => [#{name => <<"weather">>, parameters => Schema}]}).

Streaming callers use chat_apply/2 + stream/3 + chat_parse/3.

-spec chat_apply(model(), [chat_message()], chat_opts()) ->
                    {ok, #{prompt := binary(), params := chat_params()}} | {error, error_reason()}.

Render the prompt and build the output parser for one request with llama.cpp's common_chat_templates_apply. Messages and Opts are those of chat/3 (only the chat keys of Opts are used). Returns the prompt bytes and the chat_params() to hand to chat_parse/3 for this request's output; tokenise the prompt with tokenize/3 and #{add_special => false, parse_special => true}. The parser is not reusable across requests.

-spec chat_parse(chat_params(), binary(), boolean()) -> {ok, parsed_message()} | {error, error_reason()}.

Parse model output into a structured assistant message (content, reasoning, tool calls) with the parser from chat_apply/3. IsPartial = true accepts a streaming prefix.

-spec collect(reference(), timeout()) -> {ok, stream_result()} | {error, error_reason()}.

Wait for a streaming request started with stream/3 or continue/3 and fold its events into a stream_result(). {error, timeout} after Timeout milliseconds without any event (the request is cancelled and its remaining events drained).

-spec complete(model(), binary()) -> {ok, completion_result()} | {error, error_reason()}.

complete/3 with default options.

-spec complete(model(), binary(), request_opts()) -> {ok, completion_result()} | {error, error_reason()}.

Run a completion and wait for the result.

Result carries reply (detokenised text, trimmed at a matched stop string), generated (token ids produced), context_tokens (prompt ++ generated), committed_tokens, finish_key (cache key of the full context, or undefined when the finish save was suppressed), cache_hit_kind, finish_reason (stop | length | cancelled), cache_delta (#{read := N, created := N}), stop_sequence (only when one fired) and stats.

Pass the previous turn's finish_key as parent_key to resume from the cached row instead of walking the longest cached prefix. See request_opts() for every option.

-spec continue(model(), [token_id()], request_opts()) -> {ok, reference()} | {error, error_reason()}.

Extend a pinned session by prefilling SuffixTokens on top of its live KV cells, skipping the prefix-equality check and any cache lookup. Use it when the chat template renders history-dependent prefixes that would defeat stream/3's sticky path.

Opts must carry session_id; events go to to (default the caller). expect_committed => [token_id()] makes the engine verify the session's stored tokens first and fail with {error, {transcript_mismatch, #{stored_len, expected_len, diverge_at}}} on divergence, leaving the session pinned for a retry. parent_key is ignored. Events are those of stream/3; Stats.cache_hit_kind is continuation.

-spec counters() -> #{atom() => non_neg_integer()}.

Snapshot of the cache counters; see erllama_cache:get_counters/0 for the keys.

-spec detokenize(model(), [token_id()]) -> {ok, binary()} | {error, error_reason()}.

Detokenise token ids back to text.

-spec draft_tokens(model(), [token_id()], #{max => pos_integer()}) ->
                      {ok, [token_id()]} | {error, error_reason()}.

Draft up to max next tokens after PrefixTokens and return their ids. Shorter lists (EOS, response_tokens) are valid. Built on stream/3; a 30 s silence cancels the request and returns {error, timeout}.

-spec embed(model(), binary() | [token_id()]) -> {ok, [float()]} | {error, error_reason()}.

Embedding vector for a text or a token list. The model must be loaded with context_opts => #{embeddings => true}.

-spec embed_batch(model(), [binary() | [token_id()]]) -> {ok, [[float()]]} | {error, error_reason()}.

Embedding vectors for several inputs (texts or token lists) in one round-trip to the model process. Stops at the first error.

-spec end_session(model(), term()) -> ok | {error, not_loaded | timeout}.

Release a sticky session: free its KV cells and return the seq to the idle pool. Unknown session ids are a no-op.

-spec evict(model()) -> ok | {error, not_loaded | timeout}.

Fire an evict save and release the model's live KV state without unloading it. Bounded by the evict_save_timeout_ms application environment key (default 30 s).

-spec last_cache_hit(model_id()) -> {ok, cache_hit() | undefined} | {error, not_loaded}.

Lock-free summary of the model's most recent admission: the cache hit kind and the warm prefix length in tokens. {ok, undefined} before the first admission.

-spec list_adapters(model()) ->
                       {ok, [#{adapter := adapter(), scale := float()}]} | {error, not_loaded | timeout}.

Attached adapters with their scales.

-spec list_models() -> [model_info()].

Loaded models as model_info() maps (id, status, backend, context size, quantisation).

-spec load_adapter(model(), file:name_all()) -> {ok, adapter()} | {error, error_reason()}.

Load a LoRA adapter from a GGUF file and attach it with scale 1.0. The adapter's sha256 is folded into the model's effective fingerprint, so cached rows never mix across adapter sets. In-flight requests keep the previous fingerprint.

-spec load_model(load_config()) -> {ok, model_id()} | {error, error_reason()}.

Load a model. The id is taken from model_id in Config or generated (<<"erllama_model_N">>). Returns {error, {missing_config, model_path}} when no path is given for the llama backend and {error, {invalid_config, model_path, Path}} when the file does not exist; the model process is never started on a bad config.

-spec load_model(model_id(), load_config()) -> {ok, model_id()} | {error, error_reason()}.

Load a model under an explicit id. {error, already_loaded} if the id is in use.

-spec model_info(model()) -> {ok, model_info()} | {error, not_loaded | timeout}.

Inspect one loaded model (same map shape as list_models/0).

-spec pending_len(model_id()) -> {ok, non_neg_integer()} | {error, not_loaded}.

Lock-free count of calls queued behind the model's current request (complete/2,3, prefill_only/2,3, stream/3).

-spec phase(model_id()) -> {ok, phase()} | {error, not_loaded}.

Lock-free phase snapshot from the model's observability row; answers without crossing the model process, so it returns instantly while a decode step is in flight.

-spec prefill_only(model(), [token_id()]) -> {ok, prefill_result()} | {error, error_reason()}.

prefill_only/3 with default options.

-spec prefill_only(model(), [token_id()], prefill_opts()) ->
                      {ok, prefill_result()} | {error, error_reason()}.

Decode a prompt into KV state and fire a finish save without sampling any tokens. Returns finish_key for a later parent_key, or undefined when the prompt is shorter than the policy's min_tokens. With parent_key set and PromptTokens extending that context, only the new suffix is prefilled.

-spec pressure() ->
                  {ok,
                   #{source := erllama_pressure:source(),
                     used_b := non_neg_integer(),
                     total_b := non_neg_integer()}} |
                  {error, term()}.

Host or accelerator memory pressure from the scheduler's configured pressure_source (system when the scheduler is off or on noop): used and total bytes plus the source that produced them.

-spec pressure_sources() -> [erllama_pressure:source()].

Pressure sources the scheduler accepts (noop, system, nvidia_smi, ...).

-spec queue_depth() -> non_neg_integer().

Number of admitted streaming requests across all loaded models.

-spec queue_depth(model_id()) -> {ok, non_neg_integer()} | {error, not_loaded}.

Number of admitted streaming requests for one model.

-spec render_chat_template(model(), chat_request()) -> {ok, [token_id()]} | {error, error_reason()}.

Render a chat request through the model's built-in template with the legacy renderer and tokenise it. Request carries messages, system and tools. Fallback for models whose template the autoparser (chat_apply/2) cannot handle; {error, no_template} when the GGUF ships none.

-spec request_info(reference()) ->
                      {ok, #{ref := reference(), pid := pid(), model := model_id() | undefined}} |
                      {error, not_found}.

The model serving a streaming request, or {error, not_found} once it has finished.

-spec requests() -> [#{ref := reference(), pid := pid(), model := model_id() | undefined}].

Admitted streaming requests across all models: the ref and the model pid and id.

-spec reset_session(model(), term()) -> {ok, recovered | not_found} | {error, not_loaded | timeout}.

Forcibly drop a session's live KV cells and fail any in-flight request on its seq (the caller gets {erllama, Ref, {error, engine_reset}}). Bounded by a 5 s timeout so it stays usable when the engine is wedged: {error, timeout} means the model process itself is unreachable. Returns {ok, recovered} or {ok, not_found}. Prefer end_session/2 for normal teardown.

-spec set_adapter_scale(model(), adapter(), number()) -> ok | {error, error_reason()}.

Change an attached adapter's scale (also splits the cache namespace).

-spec shutdown(model()) -> ok | {error, not_loaded | timeout}.

Fire a shutdown save and return; same bound as evict/1.

-spec status(model()) -> {ok, phase()} | {error, not_loaded | timeout}.

Current phase, read through the model process.

-spec stream(model(), binary() | [token_id()], request_opts()) ->
                {ok, reference()} | {error, error_reason()}.

Streaming inference. Returns {ok, Ref} at once; events arrive at the to process (default the caller) as {erllama, Ref, stream_event()}. Prompt is text (tokenised with tokenize/2) or a token list.

session_id pins the KV cells to a session so the next request with the same id extends them in place; release it with end_session/2. A concurrent request on a pinned session returns {error, sticky_busy}; on_full => error returns {error, seq_capacity} instead of queueing when no seq is free. Stats.generated in the done event is the exact generated token list, usable as the suffix for continue/3. Use collect/2 to wait for the result without writing the receive loop.

-spec tokenize(model(), binary()) -> {ok, [token_id()]} | {error, error_reason()}.

Tokenise text (add_special => true, parse_special => false). Safe during inference.

-spec tokenize(model(), binary(), #{add_special => boolean(), parse_special => boolean()}) ->
                  {ok, [token_id()]} | {error, error_reason()}.

Tokenise with explicit options. parse_special => true turns chat-template markers in Text into their special token ids; use it for prompts rendered by chat_apply/2. add_special => false skips the BOS token.

-spec unload(model()) -> ok | {error, not_loaded}.

Unload a model and free its context. {error, not_loaded} if it is not running.

-spec unload_adapter(model(), adapter()) -> ok | {error, error_reason()}.

Detach and free an adapter. Idempotent.

-spec verify(model(), [token_id()], [token_id()], pos_integer()) ->
                {ok, #{accepted := non_neg_integer(), next := token_id() | eos}} |
                {error, error_reason()}.

Verify speculative Candidates (first K) after PrefixTokens in one forward pass. Returns the accepted prefix length and the model's own next token (eos at end of generation). The model must be idle; concurrent requests get {error, busy}. KV state is restored before returning.

-spec vram_info() ->
                   {ok,
                    #{total_b := non_neg_integer(),
                      free_b := non_neg_integer(),
                      used_b := non_neg_integer()}} |
                   {error, no_gpu | error_reason()}.

Free / total / used bytes summed over the non-CPU ggml devices. {error, no_gpu} on a CPU-only build; fall back to a system memory probe in that case.

-spec whereis(model()) -> {ok, pid()} | {error, not_loaded}.

Pid of a loaded model, e.g. to erlang:monitor/2 it.