@spec cancel(GenServer.server(), reference()) :: :ok

Cancels an in-flight or queued stream request by its subscription reference.

The slot stops being scheduled immediately and is freed for other requests (its prefix cache is retained per the request's :cache_prompt). Consumer death is detected automatically via monitors — explicit cancel is for consumers that stop reading without exiting. Server.stream/3 and stream_tokens/3 call this from their cleanup, so halting those streams early (e.g. Enum.take/2) cancels generation instead of burning batch budget to max_tokens.

Returns a specification to start this module under a supervisor.

See Supervisor.

@spec complete_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok,
   %{
     text: String.t(),
     completion_tokens: non_neg_integer(),
     finish_reason: atom()
   }}
  | {:error, term()}

Like generate_tokens/3, but returns completion metadata alongside the text.

Returns {:ok, %{text: text, completion_tokens: n, finish_reason: reason}} where reason is :eog or :max_tokens. Used by the OpenAI-shaped LlamaCppEx.chat_completion/3 when routed through a server.

@spec fetch_model(GenServer.server()) ::
  {:ok, LlamaCppEx.Model.t()} | {:error, term()}

Returns the model struct for external tokenization, or an error.

The model resource is reference-counted and thread-safe for read-only operations like tokenization. Served from LlamaCppEx.Registry — an ETS read, no round-trip through the server's mailbox.

Returns {:error, :noproc} when server does not resolve to a live process or died on its way down, and {:error, :not_ready} when the server is still loading its model (the window between start_link/1 returning and handle_continue/2 finishing).

A server whose model load failed returns {:error, {:load_failed, reason}}: handle_continue/2 stops with that reason, so the :get_model call exits with it. That escaped the previous three catch clauses — from inside both Stream.resource start-functions, where nothing can catch it — even though the @spec promised a total function.

@spec generate(GenServer.server(), String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Generates text synchronously. Blocks until generation is complete.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Call timeout in ms. Defaults to 60000.
• :cache_prompt - Reuse/retain this request's KV prefix on the slot. Defaults to the server-level :cache_prompt setting.
• :session - Any term identifying a conversation. Requests with the same session are routed to the same slot whenever it is free, keeping their cached prefix intact under concurrency.
• :cache_scope - Trust boundary for KV prefix reuse. A request only reuses a cached prefix — from its own slot, from another slot, or from the RAM prompt cache — when the cached content was produced under the same scope. Defaults to nil, a single shared pool, which is only safe when every caller of this server is in one trust domain. In a multi-tenant deployment set it to the tenant id: prefix reuse is a KV read, so two tenants whose prompts share a system prompt would otherwise be able to inherit each other's cache. Unlike :session, this does not affect slot routing, only what may be reused.
• Sampling options (:temp, :top_k, :top_p, :min_p, :seed, :penalty_repeat, :penalty_freq, :penalty_present, :grammar, :grammar_root) - override the server-level defaults for this request.

@spec generate_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok, String.t()} | {:error, term()}

Generates text from pre-tokenized input. Blocks until generation is complete.

Use get_model/1 to obtain the model for tokenization outside the server.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Call timeout in ms. Defaults to 60000.

@spec get_model(GenServer.server()) :: LlamaCppEx.Model.t()

Returns the model struct for external tokenization.

Raises when the server is not running or has not finished loading. Use fetch_model/1 when either is a possibility.

The previous @spec claimed a total function while the implementation exited with {:noproc, ...} on a dead server, so callers had no documented way to handle it.

@spec get_stats(GenServer.server()) :: map()

Returns a snapshot of the server's current state.

@spec request_option_keys() :: [atom()]

The per-request options this server accepts, beyond :max_tokens/:timeout.

Follows the same ownership rule as LlamaCppEx.Sampler.option_keys/0: callers that forward user options into a server — LlamaCppEx.chat_completion/3 and LlamaCppEx.stream_chat_completion/3 — select them with this function instead of keeping their own copy. Their copy had already drifted once, silently rejecting :cache_scope.

@spec start_link(keyword()) :: GenServer.on_start()

Starts the server.

Options

• :model_path (required) - Path to the GGUF model file.
• :n_gpu_layers - GPU layers. Defaults to 99.
• :n_ctx - Total context size (shared across slots). Defaults to 8192.
• :n_parallel - Number of concurrent slots. Defaults to 4.
• :n_batch - Max tokens per forward pass. Defaults to min(n_ctx, 2048). Bounds worst-case tick latency: one huge prompt can occupy at most n_batch tokens of a tick, so decode tokens of other slots are never delayed by more than one n_batch-sized pass. Raise it for pure batch throughput (fewer, larger passes); lower it (or lower :chunk_size) for smoother streaming latency under mixed load.
• :chunk_size - Max prefill tokens per slot per tick. Defaults to 512.
• :max_queue - Max queued requests waiting for a slot. When the bound is hit, calls return {:error, :queue_full} immediately and streams emit a single {:error, :queue_full} element — no silent queueing until the call timeout. 0 means unlimited, which is not the default: at 0 the reject branch is dead code, the documented :queue_full error can never fire, and each queued entry holds a full token list, so a burst is bounded only by memory. Defaults to 64.
• :cache_prompt - Retain KV cache between requests on the same slot for prefix reuse. Defaults to true (matching llama-server). Overridable per request via the :cache_prompt option on generate/3 and friends.
• :kv_unified - Share one KV buffer across all slots instead of splitting n_ctx evenly (n_ctx/n_parallel each). Enables cross-slot prefix sharing: a system prompt cached by any slot is adopted by every other slot via a metadata-only copy. Slots then compete for the shared n_ctx budget, and idle slots' caches are purged under KV pressure. Defaults to true. Set false for strictly isolated per-slot budgets.
• :prompt_cache_ram_mb - Byte budget (in MB) for the level-2 RAM prompt cache: when a slot's cached prefix is about to be destroyed it is serialized to RAM and can be restored later instead of re-prefilling. State blobs are KV-sized (up to hundreds of MB for long contexts) — entries larger than the budget are never stored, so a small budget degrades to "disabled" rather than OOM. Defaults to 0 (off).
• :batch_strategy - Batch building strategy module. Defaults to LlamaCppEx.Server.Strategy.DecodeMaximal. See LlamaCppEx.Server.BatchStrategy.
• Sampling options: :temp, :top_k, :top_p, :min_p, :seed, :penalty_repeat, :penalty_freq, :penalty_present, :grammar, :grammar_root.
• Context tuning options are forwarded to LlamaCppEx.Context.create/2 — :n_threads, :n_threads_batch, :n_ubatch, :type_k, :type_v, :flash_attn, :offload_kqv, :op_offload, the RoPE/YaRN options, :attention_type, :no_perf and :swa_full. See LlamaCppEx.Context.tuning_option_keys/0 for the authoritative list. :n_ctx, :n_batch, :n_seq_max and :kv_unified are set by the server from the options above and cannot be overridden here.
• Model loading options are forwarded to LlamaCppEx.Model.load/2 — :main_gpu, :split_mode, :tensor_split, :use_mmap, :use_mlock, :use_direct_io and :check_tensors. The three load flags collapse into llama.cpp's single load_mode: :use_direct_io wins outright, otherwise :use_mlock and :use_mmap combine; see LlamaCppEx.Model.load/2.
• GenServer options like :name.

@spec start_option_keys() :: [atom()]

The options start_link/1 accepts.

Exposed for the same reason as request_option_keys/0: callers that forward user options into a server must select them rather than guess. The one caller that guessed — LlamaCppEx.ModelManager.ModelIO — used a Keyword.drop/2 denylist, so :vocab_only reached init/1 and raised there.

@spec stream(GenServer.server(), String.t(), keyword()) :: Enumerable.t()

Returns a stream of generated text chunks.

If the request is rejected (:queue_full), fails mid-generation, or a chunk does not arrive within :timeout, the stream emits a single {:error, reason} element and halts — consumers that need to distinguish errors from text should match on it. A per-token timeout emits {:error, :timeout} and cancels the request server-side; it used to truncate the stream silently, which is indistinguishable from a completed generation.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Per-token timeout, and the budget for being admitted to a slot or the queue. Defaults to 30000.

Also accepts the per-request options documented on generate/3.

@spec stream_tokens(GenServer.server(), [integer()], keyword()) :: Enumerable.t()

Returns a stream of generated text chunks from pre-tokenized input.

Emits a single {:error, reason} element and halts on rejection, mid-generation failure, or a per-token timeout — see stream/3.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Per-token timeout, and the budget for being admitted to a slot or the queue. Defaults to 30000.