LlamaCppEx.Model (LlamaCppEx v0.8.44)

Model loading and introspection.

Summary

Functions

chat_template(model)

Returns the chat template string embedded in the model, or nil if none.

desc(model)

Returns a human-readable description of the model.

load(path, opts \\ [])

Loads a GGUF model from the given file path.

n_ctx_train(model)

Returns the training context size of the model.

n_embd(model)

Returns the embedding dimension of the model.

n_embd_out(model)

Returns the output-side embedding width — the row width an MTP draft head consumes. Equal to n_embd/1 for every architecture currently in tree; it is a distinct number because LlamaCppEx.MTP matches it across the target and a separate drafter GGUF.

n_layer_nextn(model)

Returns the number of MTP / "next-N" prediction layers in the checkpoint, or 0 when it carries no MTP head. Note this reports what the file contains; the layers are only actually loaded when the model was opened with load_mtp: true.

n_params(model)

Returns the number of model parameters.

size(model)

Returns the model file size in bytes.

structural_option_keys()

Options a caller must set explicitly rather than forward blindly.

tuning_option_keys()

Options that are safe for a caller to forward from user-supplied opts.

Types

t()

@type t() :: %LlamaCppEx.Model{load_mtp: boolean(), ref: reference()}

Functions

chat_template(model)

@spec chat_template(t()) :: String.t() | nil

Returns the chat template string embedded in the model, or nil if none.

desc(model)

@spec desc(t()) :: String.t()

Returns a human-readable description of the model.

load(path, opts \\ [])

@spec load(
  String.t(),
  keyword()
) :: {:ok, t()} | {:error, String.t()}

Loads a GGUF model from the given file path.

Options

:n_gpu_layers - Number of layers to offload to GPU. Use -1 for all layers. Defaults to 99 (offload all layers).
:use_mmap - Whether to memory-map the model file. Defaults to true.
:main_gpu - GPU device index for single-GPU mode. Defaults to 0.
:split_mode - How to split the model across devices: :none, :layer, :row or :tensor. Defaults to :none. Only :none and :layer are generally usable at this llama.cpp version — see the note below.
:tensor_split - List of floats specifying the proportion of work per GPU (e.g. [0.5, 0.5] for two GPUs). Defaults to [].
:use_mlock - Pin model memory in RAM to prevent swapping. Implies :use_mmap. Defaults to false.
:use_direct_io - Bypass page cache when loading (takes precedence over mmap). Defaults to false.
:vocab_only - Load vocabulary and metadata only, skip weights. Defaults to false.
:check_tensors - Validate model tensor data on load. Defaults to false, because the check walks every tensor and costs real time on a large model.
:load_mtp - Load the Multi-Token Prediction head's layers, for use with LlamaCppEx.MTP. Defaults to false, matching upstream, so that callers who are not doing speculative decoding do not pay for the extra tensors. Required for LlamaCppEx.MTP.init/2, which refuses a model loaded without it — the layers cannot be added after the fact.
:rpc_servers - Endpoints ("host:port") to register before loading, so their remote devices can hold part of the model. Defaults to []. Requires a build with LLAMA_RPC=1. See LlamaCppEx.RPC. Note that llama.cpp puts remote devices first in its automatic placement list — which is not the order LlamaCppEx.devices/0 reports — so tensor_split: [0.25, 0.75] gives 25% to the first remote endpoint. Pass :devices to avoid guessing.
:devices - Device names, e.g. ["CUDA0", "RPC0"], used verbatim as the placement list: no reordering, no dedup, no CPU filtering. Defaults to [], which lets llama.cpp build the list itself. Set this whenever more than one device is in play, because the automatic list is not the order LlamaCppEx.devices/0 reports — it puts RPC devices first — so :tensor_split and :main_gpu would index a list you never saw. With :devices set, they index this one.

Split modes at llama.cpp b10362

:layer splits contiguous layer ranges across devices, one KV cache per device, and is the only mode that works across hosts.

:row throws at load time on CUDA: ggml-cuda no longer exports ggml_backend_split_buffer_type, so llama_model_load raises device CUDA0 does not support split buffers. It is kept mapped to its upstream value rather than removed, because the enum is upstream's, but do not build on it. Only SYCL still declares a split buffer type.

:tensor is real tensor parallelism via a Meta device, added in llama.cpp #19378. It forces flash attention on, refuses some architectures, disables backend sampling, and its CUDA all-reduce is ncclCommInitAll — a single-process, all-local-GPUs API. It cannot span hosts, so it is not "tp=2 across two machines". See docs/dgx-spark.md for the measurements.

Load mode

llama.cpp collapsed its three loading booleans into one load_mode enum, so these options resolve to a single mode. :use_direct_io takes precedence over everything and selects dio; otherwise :use_mlock and :use_mmap combine — both true selects mmap_mlock, :use_mlock alone selects mlock (read into anonymous memory, no mapping), :use_mmap alone selects mmap, and all false selects none.

Untrusted models

GGUF parsing happens in llama.cpp's C++ loader, and :check_tensors defaults to false for every source — including files fetched by LlamaCppEx.Hub.download/3, which verifies a download against the SHA-256 HuggingFace publishes but cannot vouch for what the repository owner uploaded. load/2 receives a bare path and has no notion of provenance, so it cannot raise that default on its own: pass check_tensors: true explicitly for any model whose publisher you do not trust.

Examples

{:ok, model} = LlamaCppEx.Model.load("path/to/model.gguf", n_gpu_layers: -1)
{:ok, model} = LlamaCppEx.Model.load("path/to/model.gguf", split_mode: :layer, tensor_split: [0.5, 0.5])
{:ok, model} = LlamaCppEx.Model.load("path/to/model.gguf", vocab_only: true)