Multi-Token Prediction (MTP) speculative decoding.
Drives a target/draft speculative loop where the draft model is an MTP head —
either embedded in the target GGUF, or shipped beside it as a sidecar file. On
Qwen 3.6 with n_draft: 3 this typically yields ~2x token-generation
throughput at ~75% draft acceptance.
Usage
:ok = LlamaCppEx.init()
{:ok, model} = LlamaCppEx.load_model("Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf",
n_gpu_layers: 999, load_mtp: true)
{:ok, mtp} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 8192)
mtp
|> LlamaCppEx.MTP.stream("Write a haiku about the sea:", max_tokens: 200)
|> Stream.each(&IO.write/1)
|> Stream.run()
stats = LlamaCppEx.MTP.stats(mtp)
IO.puts("acceptance: #{Float.round(stats.acceptance_rate * 100, 1)}%")Where the MTP head lives
The head is a set of *.nextn_predict_layers and comes in two shapes, and
either way the file carrying it must be loaded with load_mtp: true. Upstream
defaults that flag to false so non-speculative callers do not pay for the
head's tensors, and the layers cannot be attached afterwards, so init/2
refuses a model loaded without it.
In the target GGUF (e.g. ggml-org/Qwen3.6-35B-A3B-MTP-GGUF) — pass just
the model, as above.
In a sidecar GGUF — pass it as :draft_model. This is how Qwen 3.8 ships:
Qwen3.8-27B-Q4_K_M.gguf carries no head at all (n_layer_nextn == 0) and
mtp-Qwen3.8-27B-Q4_0.gguf carries nothing else. It is the binding's
equivalent of upstream's -hf <target> -hfd <draft> --spec-type draft-mtp.
{:ok, target} = LlamaCppEx.load_model("Qwen3.8-27B-Q4_K_M.gguf",
n_gpu_layers: 999, load_mtp: true)
{:ok, head} = LlamaCppEx.load_model("mtp-Qwen3.8-27B-Q4_0.gguf",
n_gpu_layers: 999, load_mtp: true)
{:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1)Gemma4 E4B sidecar — same :draft_model API: a gemma4 target plus a
gemma4-assistant head. init/2 always passes the target as :ctx_other
on the draft. gemma4-assistant keeps that pointer (shared KV and the
target's token embeddings); Qwen's constructor leaves cparams.ctx_other
as nullptr.
Hybrid models roll back on recurrent snapshots
A model that mixes recurrent (SSM) layers with attention ones — Qwen 3.8 is
48 SSM layers to 16 attention layers — cannot roll back part of a sequence
from its KV cache alone. init/2 therefore creates the target context with
one recurrent-state snapshot per draft position (n_rs_seq: n_draft), as
upstream's server does, and a rejected draft is rolled back natively. Before
that, every iteration saved the whole recurrent state (over 100 MiB at Qwen
3.8's sizes) and every partial accept restored it and re-decoded the
accepted prefix: on GB10 that held Qwen3.8-27B NVFP4 to 21 tok/s at
n_draft: 2 where the snapshot path does 27, and on an M1 Max it made MTP a
net slowdown. stats/1's timing_us.ckpt is that save/restore, and stays
zero unless n_draft exceeds the snapshots.
Upstream currently requires n_parallel = 1 for MTP. This module reflects
that — a single MTP session decodes one sequence at a time. Reuse the same
%MTP{} value across calls to stream/3 / generate/3 to avoid rebuilding
the contexts; KV caches are cleared on each call. Calls on one session run
one after another: a call made while another is still decoding, including a
stream abandoned early (Enum.take/2, an exception) whose loop has not yet
noticed its cancel flag, waits for it to finish. The wait after an abandoned
stream is at most one draft-and-verify iteration.
MTP is the only speculative type this binding exposes. Upstream llama.cpp
also implements EAGLE-3, DFlash (block-diffusion drafting), n-gram
self-speculation and combinations of them behind the same
common_speculative API; the NIF pins the MTP type, so --spec-default-style
stacking of n-gram speculation on top of MTP is not reachable from here. See
the "Speculative decoding" section of the README for the current status of
DFlash on Apple Silicon.
Summary
Functions
Synchronously generates text. Equivalent to running stream/3 and joining
the pieces into a single binary.
Initializes a speculative session: builds the target context, the draft
context, and the underlying common_speculative state.
Writes upstream's own speculative stats summary to stdout (via llama.cpp logging). Useful when cross-checking acceptance rates against the upstream llama-server benchmark output.
Returns the current MTP statistics snapshot (lock-free read of atomic counters). Safe to call at any time — including from another process while a stream is in flight.
Returns a lazy stream of generated text pieces.
Like stream/3, but yields the raw event tuples emitted by the NIF
Types
@type t() :: %LlamaCppEx.MTP{ main_ctx: LlamaCppEx.Context.t(), mtp_ctx: LlamaCppEx.Context.t(), n_draft: pos_integer(), spec_ref: reference() }
Functions
Synchronously generates text. Equivalent to running stream/3 and joining
the pieces into a single binary.
Accepts the same options as stream/3.
@spec init( LlamaCppEx.Model.t(), keyword() ) :: {:ok, t()} | {:error, term()}
Initializes a speculative session: builds the target context, the draft
context, and the underlying common_speculative state.
Options
:type- The drafter. Defaults to:mtp.:mtp- a Multi-Token Prediction head, inside the target GGUF or in a sidecar passed as:draft_model. The draft context is built withctx_type: :mtp.:dflash- a DFlash block-diffusion drafter (GGUF architecturedflash), passed as:draft_model. Upstream picks the head from the drafter's metadata, so DFlash 2 and LiliCorr drafters run under this type too. It drafts a whole block in one pass from the target's hidden states, and is the fastest option measured on GB10: 3.3× plain decode on Qwen3.8-27B NVFP4 against 2.1× for its MTP head (seedocs/dgx-spark.md).:dspark- a DSpark drafter (DFlash plus a Markov head), passed as:draft_model.init/2refuses any other DFlash-family drafter for this type;:dflashruns them all.
:draft_model- A separateLlamaCppEx.Modelholding the drafter. Required for:dflashand:dspark. A DFlash drafter computes with the target's embeddings and LM head; its context gets a backend for every device the target uses, so the drafter can sit on a subset of them (e.g.CUDA0alone beside a target split onto an RPC worker). The drafter reads the target's features from the target's output device, so a target split onto an RPC worker should list the local device last (devices: ["RPC0", "CUDA0"]); with the RPC device last, every verify ships them over the network and DFlash decodes slower than plain (init/2logs a warning). For:mtpit is the sidecar head for checkpoints that ship one rather than embedding it (Qwen 3.8:Qwen3.8-27B-Q4_K_M.ggufplusmtp-Qwen3.8-27B-Q4_0.gguf; Gemma4 E4B: agemma4target plus agemma4-assistanthead), loaded withload_mtp: true. Defaults tonil, meaning the MTP head is expected inside the target model and the draft context is built against it.:n_draft- Max draft tokens generated per iteration. Defaults to3. Larger values mean fewer model forward passes but lower per-iteration acceptance; 2–4 is the sweet spot for MTP in practice. A DFlash drafter caps it at its trained block size minus one (7 for Qwen3.8 DFlash 2), and that is also where it measured fastest.:n_ctx- Context size for both contexts. Defaults to2048.- Any
Contexttuning option (e.g.:n_threads,:flash_attn,:type_k/:type_v,:offload_kqv). Applied to both contexts.:ctx_otheris not a caller option; the draft always gets the target.
Returns {:ok, %MTP{}} or {:error, reason}.
@spec print_stats(t()) :: :ok
Writes upstream's own speculative stats summary to stdout (via llama.cpp logging). Useful when cross-checking acceptance rates against the upstream llama-server benchmark output.
Returns the current MTP statistics snapshot (lock-free read of atomic counters). Safe to call at any time — including from another process while a stream is in flight.
Returns a map with keys:
:iters- speculative loop iterations completed:drafts_generated- draft tokens proposed by the MTP head:drafts_accepted- draft tokens accepted by the target model:acceptance_rate-drafts_accepted / drafts_generated(0.0–1.0):tokens_emitted- tokens streamed back to the caller:tokens_per_sec- throughput over the active generation window:timing_us-%{draft: μs, verify: μs, sample: μs, ckpt: μs, other: μs, total: μs}.:ckptis the recurrent-state save/restore a hybrid target pays when it has fewer recurrent snapshots thann_draft(never, for a session built byinit/2), and is zero elsewhere.:otheris whatever falls outside the named buckets, dominated on Metal by GPU-sync waits from the previous iteration's async verify decode.:n_draft- max draft length configured at init
Counters are cumulative across all stream/3 / generate/3 calls on this
MTP value.
@spec stream(t(), String.t(), keyword()) :: Enumerable.t()
Returns a lazy stream of generated text pieces.
Options
:max_tokens- Maximum tokens to generate (default256).:emit_stats_every- When > 0, also emits{:stats, snapshot_map}events every Nth token via the underlying message stream. Note: these events are filtered out of thisString.t()stream — to consume them usestream_events/3instead. Default0(off).:timeout- Receive timeout in milliseconds (default60_000).- Any sampler option from
LlamaCppEx.Sampler.create/2(:temp,:top_k,:top_p,:min_p,:seed,:penalty_*,:grammar, etc.).
Each emitted element is the text piece for one accepted token. The stream ends on end-of-generation, max-tokens, or error.
@spec stream_events(t(), String.t(), keyword()) :: Enumerable.t()
Like stream/3, but yields the raw event tuples emitted by the NIF:
{:token, token_id, text_piece}- one accepted token{:stats, snapshot_map}- periodic stats (only when:emit_stats_every > 0){:done, final_stats_map}- generation completed normally{:eog, nil}- model emitted an end-of-generation token
The stream halts after :done / :eog / :error. The final stats map is
available via stats/1 on the MTP struct even after the stream ends.