@spec cancel(GenServer.server(), reference()) :: :ok

Cancels an in-flight or queued stream request by its subscription reference.

The slot stops being scheduled immediately and is freed for other requests (its prefix cache is retained per the request's :cache_prompt). Consumer death is detected automatically via monitors — explicit cancel is for consumers that stop reading without exiting. Server.stream/3 and stream_tokens/3 call this from their cleanup, so halting those streams early (e.g. Enum.take/2) cancels generation instead of burning batch budget to max_tokens.

Returns a specification to start this module under a supervisor.

See Supervisor.

@spec complete_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok,
   %{
     text: String.t(),
     completion_tokens: non_neg_integer(),
     finish_reason: atom()
   }}
  | {:error, term()}

Like generate_tokens/3, but returns completion metadata alongside the text.

Returns {:ok, %{text: text, completion_tokens: n, finish_reason: reason}} where reason is :eog or :max_tokens. Used by the OpenAI-shaped LlamaCppEx.chat_completion/3 when routed through a server.

@spec generate(GenServer.server(), String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Generates text synchronously. Blocks until generation is complete.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Call timeout in ms. Defaults to 60_000.
• :cache_prompt - Reuse/retain this request's KV prefix on the slot. Defaults to the server-level :cache_prompt setting.
• :session - Any term identifying a conversation. Requests with the same session are routed to the same slot whenever it is free, keeping their cached prefix intact under concurrency.
• Sampling options (:temp, :top_k, :top_p, :min_p, :seed, :penalty_repeat, :penalty_freq, :penalty_present, :grammar, :grammar_root) - override the server-level defaults for this request.

@spec generate_tokens(GenServer.server(), [integer()], keyword()) ::
  {:ok, String.t()} | {:error, term()}

Generates text from pre-tokenized input. Blocks until generation is complete.

Use get_model/1 to obtain the model for tokenization outside the server.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Call timeout in ms. Defaults to 60_000.

@spec get_model(GenServer.server()) :: LlamaCppEx.Model.t()

Returns the model struct for external tokenization.

The model resource is reference-counted and thread-safe for read-only operations like tokenization. Served from a :persistent_term cache — no round-trip through the server's mailbox.

@spec get_stats(GenServer.server()) :: map()

Returns a snapshot of the server's current state.

@spec start_link(keyword()) :: GenServer.on_start()

Starts the server.

Options

• :model_path (required) - Path to the GGUF model file.
• :n_gpu_layers - GPU layers. Defaults to 99.
• :n_ctx - Total context size (shared across slots). Defaults to 8192.
• :n_parallel - Number of concurrent slots. Defaults to 4.
• :n_batch - Max tokens per forward pass. Defaults to min(n_ctx, 2048). Bounds worst-case tick latency: one huge prompt can occupy at most n_batch tokens of a tick, so decode tokens of other slots are never delayed by more than one n_batch-sized pass. Raise it for pure batch throughput (fewer, larger passes); lower it (or lower :chunk_size) for smoother streaming latency under mixed load.
• :chunk_size - Max prefill tokens per slot per tick. Defaults to 512.
• :max_queue - Max queued requests waiting for a slot. When the bound is hit, calls return {:error, :queue_full} immediately and streams emit a single {:error, :queue_full} element — no silent queueing until the call timeout. 0 for unlimited. Defaults to 0; 2 * n_parallel is a reasonable bound for latency-sensitive deployments.
• :cache_prompt - Retain KV cache between requests on the same slot for prefix reuse. Defaults to true (matching llama-server). Overridable per request via the :cache_prompt option on generate/3 and friends.
• :kv_unified - Share one KV buffer across all slots instead of splitting n_ctx evenly (n_ctx/n_parallel each). Enables cross-slot prefix sharing: a system prompt cached by any slot is adopted by every other slot via a metadata-only copy. Slots then compete for the shared n_ctx budget, and idle slots' caches are purged under KV pressure. Defaults to true. Set false for strictly isolated per-slot budgets.
• :prompt_cache_ram_mb - Byte budget (in MB) for the level-2 RAM prompt cache: when a slot's cached prefix is about to be destroyed it is serialized to RAM and can be restored later instead of re-prefilling. State blobs are KV-sized (up to hundreds of MB for long contexts) — entries larger than the budget are never stored, so a small budget degrades to "disabled" rather than OOM. Defaults to 0 (off).
• :batch_strategy - Batch building strategy module. Defaults to LlamaCppEx.Server.Strategy.DecodeMaximal. See LlamaCppEx.Server.BatchStrategy.
• Sampling options: :temp, :top_k, :top_p, :min_p, :seed, :penalty_repeat, :penalty_freq, :penalty_present, :grammar, :grammar_root.
• GenServer options like :name.

@spec stream(GenServer.server(), String.t(), keyword()) :: Enumerable.t()

Returns a stream of generated text chunks.

If the request is rejected (:queue_full) or fails mid-generation, the stream emits a single {:error, reason} element and halts — consumers that need to distinguish errors from text should match on it.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Per-token timeout. Defaults to 30_000.

Also accepts the per-request options documented on generate/3.

@spec stream_tokens(GenServer.server(), [integer()], keyword()) :: Enumerable.t()

Returns a stream of generated text chunks from pre-tokenized input.

Options

• :max_tokens - Maximum tokens to generate. Defaults to 256.
• :timeout - Per-token timeout. Defaults to 30_000.