Token sampling configuration.
Builds a sampler chain with the common sampling parameters. The samplers are applied in order: grammar -> penalties -> top_k -> top_p -> min_p -> temp -> dist/greedy.
Summary
Types
@type t() :: %LlamaCppEx.Sampler{model: LlamaCppEx.Model.t() | nil, ref: reference()}
A sampler chain.
:model holds the LlamaCppEx.Model.t/0 the chain was built from. It is
carried so the model term cannot be garbage-collected while the chain is alive:
a grammar stage keeps a raw pointer to the model's vocabulary for its whole
lifetime, and dropping the model out from under it made the next reset/1 or
accept/2 dereference freed memory. nil only for hand-constructed structs.
Functions
Accepts a token (updates sampler internal state).
@spec create( LlamaCppEx.Model.t(), keyword() ) :: {:ok, t()} | {:error, :invalid_grammar}
Creates a new sampler chain.
Requires a model reference (needed for grammar-constrained sampling).
Options
:seed- Random seed for sampling. Defaults to a random value.:temp- Temperature.0.0for greedy sampling. Defaults to0.8.:top_k- Top-K filtering.0to disable. Defaults to40.:top_p- Top-P (nucleus) filtering.1.0to disable. Defaults to0.95.:min_p- Min-P filtering.0.0to disable. Defaults to0.05.:penalty_repeat- Repetition penalty.1.0to disable. Defaults to1.0.:penalty_freq- Frequency penalty (0.0–2.0).0.0to disable. Defaults to0.0.:penalty_present- Presence penalty (0.0–2.0).0.0to disable. Defaults to0.0.:grammar- GBNF grammar string for constrained generation. Defaults to""(none).:grammar_root- Root rule name for grammar. Defaults to"root".:grammar_lazy- Leave the grammar inactive until the output matches one of:grammar_triggers. Defaults tofalse. This is how a tool-call grammar lets a model answer in prose or call a tool;LlamaCppEx.chat_completion/3sets it from the chat template when given:tools.:grammar_triggers- For a lazy grammar: a list of{:token, id}(the grammar activates when that token is sampled) and{:pattern, regex}(when the generated text matches the regex). Defaults to[].:grammar_prefill- Token ids fed to a non-lazy grammar before sampling, for a grammar that starts inside the prompt. Defaults to[].
Errors
Returns {:error, :invalid_grammar} when :grammar does not compile, exceeds
1 MiB, or nests ( groups more than 64 deep, and when a trigger pattern is not
a valid regex, a token id is out of range, or the grammar rejects the prefill.
A grammar that fails to compile is not silently dropped: a caller who
asked for JSON-constrained output would otherwise get unconstrained output and
no indication of it.
@spec option_keys() :: [atom()]
The options create/2 accepts.
This module is the single source of truth: callers that forward user sampling
options (LlamaCppEx, LlamaCppEx.Server) select them with this function
instead of keeping their own copy of the list. Every sampling option is safe
to forward, so there is no tuning/structural split here.
@spec reset(t()) :: :ok
Resets the sampler state.
@spec sample(t(), LlamaCppEx.Context.t()) :: integer()
Samples the next token from the context's logits.
@spec validate_grammar( LlamaCppEx.Model.t(), keyword() ) :: :ok | {:error, :invalid_grammar}
Checks that the :grammar in opts compiles, without building a chain.
create/2 only discovers a bad grammar as a side effect of building the
sampler, which is too late for LlamaCppEx.Server: the request is already
admitted to a slot inside the process that owns the model, so the failure
crashed the server rather than the request. This is the same check
create/2's NIF runs, callable before a request is queued.
Returns :ok when opts carries no :grammar, or an empty one.