LlamaCppEx.Sampler (LlamaCppEx v0.8.41)

Token sampling configuration.

Builds a sampler chain with the common sampling parameters. The samplers are applied in order: grammar -> penalties -> top_k -> top_p -> min_p -> temp -> dist/greedy.

Summary

Functions

accept(sampler, token)

Accepts a token (updates sampler internal state).

create(model, opts \\ [])

Creates a new sampler chain.

option_keys()

The options create/2 accepts.

reset(sampler)

Resets the sampler state.

sample(sampler, context)

Samples the next token from the context's logits.

validate_grammar(model, opts)

Checks that the :grammar in opts compiles, without building a chain.

Types

t()

@type t() :: %LlamaCppEx.Sampler{model: LlamaCppEx.Model.t() | nil, ref: reference()}

A sampler chain.

:model holds the LlamaCppEx.Model.t/0 the chain was built from. It is carried so the model term cannot be garbage-collected while the chain is alive: a grammar stage keeps a raw pointer to the model's vocabulary for its whole lifetime, and dropping the model out from under it made the next reset/1 or accept/2 dereference freed memory. nil only for hand-constructed structs.

Functions

accept(sampler, token)

@spec accept(t(), integer()) :: :ok

Accepts a token (updates sampler internal state).

create(model, opts \\ [])

@spec create(
  LlamaCppEx.Model.t(),
  keyword()
) :: {:ok, t()} | {:error, :invalid_grammar}

Creates a new sampler chain.

Requires a model reference (needed for grammar-constrained sampling).

Options

:seed - Random seed for sampling. Defaults to a random value.
:temp - Temperature. 0.0 for greedy sampling. Defaults to 0.8.
:top_k - Top-K filtering. 0 to disable. Defaults to 40.
:top_p - Top-P (nucleus) filtering. 1.0 to disable. Defaults to 0.95.
:min_p - Min-P filtering. 0.0 to disable. Defaults to 0.05.
:penalty_repeat - Repetition penalty. 1.0 to disable. Defaults to 1.0.
:penalty_freq - Frequency penalty (0.0–2.0). 0.0 to disable. Defaults to 0.0.
:penalty_present - Presence penalty (0.0–2.0). 0.0 to disable. Defaults to 0.0.
:grammar - GBNF grammar string for constrained generation. Defaults to "" (none).
:grammar_root - Root rule name for grammar. Defaults to "root".

Errors

Returns {:error, :invalid_grammar} when :grammar does not compile, exceeds 1 MiB, or nests ( groups more than 64 deep. A grammar that fails to compile is not silently dropped: a caller who asked for JSON-constrained output would otherwise get unconstrained output and no indication of it.

option_keys()

@spec option_keys() :: [atom()]

The options create/2 accepts.

This module is the single source of truth: callers that forward user sampling options (LlamaCppEx, LlamaCppEx.Server) select them with this function instead of keeping their own copy of the list. Every sampling option is safe to forward, so there is no tuning/structural split here.

reset(sampler)

@spec reset(t()) :: :ok

Resets the sampler state.

sample(sampler, context)

@spec sample(t(), LlamaCppEx.Context.t()) :: integer()

Samples the next token from the context's logits.

validate_grammar(model, opts)

@spec validate_grammar(
  LlamaCppEx.Model.t(),
  keyword()
) :: :ok | {:error, :invalid_grammar}

Checks that the :grammar in opts compiles, without building a chain.

create/2 only discovers a bad grammar as a side effect of building the sampler, which is too late for LlamaCppEx.Server: the request is already admitted to a slot inside the process that owns the model, so the failure crashed the server rather than the request. This is the same check create/2's NIF runs, callable before a request is queued.

Returns :ok when opts carries no :grammar, or an empty one.