LlamaCppEx.MTP (LlamaCppEx v0.8.71)

Copy Markdown View Source

Multi-Token Prediction (MTP) speculative decoding.

Drives a target/draft speculative loop where the draft model is an MTP head — either embedded in the target GGUF, or shipped beside it as a sidecar file. On Qwen 3.6 with n_draft: 3 this typically yields ~2x token-generation throughput at ~75% draft acceptance.

Usage

:ok = LlamaCppEx.init()
{:ok, model} = LlamaCppEx.load_model("Qwen3.6-35B-A3B-MTP-Q4_K_M.gguf",
                                      n_gpu_layers: 999, load_mtp: true)

{:ok, mtp} = LlamaCppEx.MTP.init(model, n_draft: 3, n_ctx: 8192)

mtp
|> LlamaCppEx.MTP.stream("Write a haiku about the sea:", max_tokens: 200)
|> Stream.each(&IO.write/1)
|> Stream.run()

stats = LlamaCppEx.MTP.stats(mtp)
IO.puts("acceptance: #{Float.round(stats.acceptance_rate * 100, 1)}%")

Where the MTP head lives

The head is a set of *.nextn_predict_layers and comes in two shapes, and either way the file carrying it must be loaded with load_mtp: true. Upstream defaults that flag to false so non-speculative callers do not pay for the head's tensors, and the layers cannot be attached afterwards, so init/2 refuses a model loaded without it.

In the target GGUF (e.g. ggml-org/Qwen3.6-35B-A3B-MTP-GGUF) — pass just the model, as above.

In a sidecar GGUF — pass it as :draft_model. This is how Qwen 3.8 ships: Qwen3.8-27B-Q4_K_M.gguf carries no head at all (n_layer_nextn == 0) and mtp-Qwen3.8-27B-Q4_0.gguf carries nothing else. It is the binding's equivalent of upstream's -hf <target> -hfd <draft> --spec-type draft-mtp.

{:ok, target} = LlamaCppEx.load_model("Qwen3.8-27B-Q4_K_M.gguf",
                                      n_gpu_layers: 999, load_mtp: true)
{:ok, head}   = LlamaCppEx.load_model("mtp-Qwen3.8-27B-Q4_0.gguf",
                                      n_gpu_layers: 999, load_mtp: true)

{:ok, mtp} = LlamaCppEx.MTP.init(target, draft_model: head, n_draft: 1)

Gemma4 E4B sidecar — same :draft_model API: a gemma4 target plus a gemma4-assistant head. init/2 always passes the target as :ctx_other on the draft. gemma4-assistant keeps that pointer (shared KV and the target's token embeddings); Qwen's constructor leaves cparams.ctx_other as nullptr.

Hybrid models roll back on recurrent snapshots

A model that mixes recurrent (SSM) layers with attention ones — Qwen 3.8 is 48 SSM layers to 16 attention layers — cannot roll back part of a sequence from its KV cache alone. init/2 therefore creates the target context with one recurrent-state snapshot per draft position (n_rs_seq: n_draft), as upstream's server does, and a rejected draft is rolled back natively. Before that, every iteration saved the whole recurrent state (over 100 MiB at Qwen 3.8's sizes) and every partial accept restored it and re-decoded the accepted prefix: on GB10 that held Qwen3.8-27B NVFP4 to 21 tok/s at n_draft: 2 where the snapshot path does 27, and on an M1 Max it made MTP a net slowdown. stats/1's timing_us.ckpt is that save/restore, and stays zero unless n_draft exceeds the snapshots.

Upstream currently requires n_parallel = 1 for MTP. This module reflects that — a single MTP session decodes one sequence at a time. Reuse the same %MTP{} value across calls to stream/3 / generate/3 to avoid rebuilding the contexts; KV caches are cleared on each call. Calls on one session run one after another: a call made while another is still decoding, including a stream abandoned early (Enum.take/2, an exception) whose loop has not yet noticed its cancel flag, waits for it to finish. The wait after an abandoned stream is at most one draft-and-verify iteration.

MTP is the only speculative type this binding exposes. Upstream llama.cpp also implements EAGLE-3, DFlash (block-diffusion drafting), n-gram self-speculation and combinations of them behind the same common_speculative API; the NIF pins the MTP type, so --spec-default-style stacking of n-gram speculation on top of MTP is not reachable from here. See the "Speculative decoding" section of the README for the current status of DFlash on Apple Silicon.

Summary

Functions

Synchronously generates text. Equivalent to running stream/3 and joining the pieces into a single binary.

Initializes a speculative session: builds the target context, the draft context, and the underlying common_speculative state.

Writes upstream's own speculative stats summary to stdout (via llama.cpp logging). Useful when cross-checking acceptance rates against the upstream llama-server benchmark output.

Returns the current MTP statistics snapshot (lock-free read of atomic counters). Safe to call at any time — including from another process while a stream is in flight.

Returns a lazy stream of generated text pieces.

Like stream/3, but yields the raw event tuples emitted by the NIF

Types

t()

@type t() :: %LlamaCppEx.MTP{
  main_ctx: LlamaCppEx.Context.t(),
  mtp_ctx: LlamaCppEx.Context.t(),
  n_draft: pos_integer(),
  spec_ref: reference()
}

Functions

generate(mtp, prompt, opts \\ [])

@spec generate(t(), String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}

Synchronously generates text. Equivalent to running stream/3 and joining the pieces into a single binary.

Accepts the same options as stream/3.

init(model, opts \\ [])

@spec init(
  LlamaCppEx.Model.t(),
  keyword()
) :: {:ok, t()} | {:error, term()}

Initializes a speculative session: builds the target context, the draft context, and the underlying common_speculative state.

Options

  • :type - The drafter. Defaults to :mtp.
    • :mtp - a Multi-Token Prediction head, inside the target GGUF or in a sidecar passed as :draft_model. The draft context is built with ctx_type: :mtp.
    • :dflash - a DFlash block-diffusion drafter (GGUF architecture dflash), passed as :draft_model. Upstream picks the head from the drafter's metadata, so DFlash 2 and LiliCorr drafters run under this type too. It drafts a whole block in one pass from the target's hidden states, and is the fastest option measured on GB10: 3.3× plain decode on Qwen3.8-27B NVFP4 against 2.1× for its MTP head (see docs/dgx-spark.md).
    • :dspark - a DSpark drafter (DFlash plus a Markov head), passed as :draft_model. init/2 refuses any other DFlash-family drafter for this type; :dflash runs them all.
  • :draft_model - A separate LlamaCppEx.Model holding the drafter. Required for :dflash and :dspark. A DFlash drafter computes with the target's embeddings and LM head; its context gets a backend for every device the target uses, so the drafter can sit on a subset of them (e.g. CUDA0 alone beside a target split onto an RPC worker). The drafter reads the target's features from the target's output device, so a target split onto an RPC worker should list the local device last (devices: ["RPC0", "CUDA0"]); with the RPC device last, every verify ships them over the network and DFlash decodes slower than plain (init/2 logs a warning). For :mtp it is the sidecar head for checkpoints that ship one rather than embedding it (Qwen 3.8: Qwen3.8-27B-Q4_K_M.gguf plus mtp-Qwen3.8-27B-Q4_0.gguf; Gemma4 E4B: a gemma4 target plus a gemma4-assistant head), loaded with load_mtp: true. Defaults to nil, meaning the MTP head is expected inside the target model and the draft context is built against it.
  • :n_draft - Max draft tokens generated per iteration. Defaults to 3. Larger values mean fewer model forward passes but lower per-iteration acceptance; 2–4 is the sweet spot for MTP in practice. A DFlash drafter caps it at its trained block size minus one (7 for Qwen3.8 DFlash 2), and that is also where it measured fastest.
  • :n_ctx - Context size for both contexts. Defaults to 2048.
  • Any Context tuning option (e.g. :n_threads, :flash_attn, :type_k/:type_v, :offload_kqv). Applied to both contexts. :ctx_other is not a caller option; the draft always gets the target.

Returns {:ok, %MTP{}} or {:error, reason}.

stats(mtp)

@spec stats(t()) :: map()

Returns the current MTP statistics snapshot (lock-free read of atomic counters). Safe to call at any time — including from another process while a stream is in flight.

Returns a map with keys:

  • :iters - speculative loop iterations completed
  • :drafts_generated - draft tokens proposed by the MTP head
  • :drafts_accepted - draft tokens accepted by the target model
  • :acceptance_rate - drafts_accepted / drafts_generated (0.0–1.0)
  • :tokens_emitted - tokens streamed back to the caller
  • :tokens_per_sec - throughput over the active generation window
  • :timing_us - %{draft: μs, verify: μs, sample: μs, ckpt: μs, other: μs, total: μs}. :ckpt is the recurrent-state save/restore a hybrid target pays when it has fewer recurrent snapshots than n_draft (never, for a session built by init/2), and is zero elsewhere. :other is whatever falls outside the named buckets, dominated on Metal by GPU-sync waits from the previous iteration's async verify decode.
  • :n_draft - max draft length configured at init

Counters are cumulative across all stream/3 / generate/3 calls on this MTP value.

stream(mtp, prompt, opts \\ [])

@spec stream(t(), String.t(), keyword()) :: Enumerable.t()

Returns a lazy stream of generated text pieces.

Options

  • :max_tokens - Maximum tokens to generate (default 256).
  • :emit_stats_every - When > 0, also emits {:stats, snapshot_map} events every Nth token via the underlying message stream. Note: these events are filtered out of this String.t() stream — to consume them use stream_events/3 instead. Default 0 (off).
  • :timeout - Receive timeout in milliseconds (default 60_000).
  • Any sampler option from LlamaCppEx.Sampler.create/2 (:temp, :top_k, :top_p, :min_p, :seed, :penalty_*, :grammar, etc.).

Each emitted element is the text piece for one accepted token. The stream ends on end-of-generation, max-tokens, or error.

stream_events(mtp, prompt, opts \\ [])

@spec stream_events(t(), String.t(), keyword()) :: Enumerable.t()

Like stream/3, but yields the raw event tuples emitted by the NIF:

  • {:token, token_id, text_piece} - one accepted token
  • {:stats, snapshot_map} - periodic stats (only when :emit_stats_every > 0)
  • {:done, final_stats_map} - generation completed normally
  • {:eog, nil} - model emitted an end-of-generation token

The stream halts after :done / :eog / :error. The final stats map is available via stats/1 on the MTP struct even after the stream ends.