Release Guide

How to upgrade the llama.cpp submodule and publish a new release.

Prerequisites

Elixir 1.18+, Erlang/OTP 26+ (OTP 26/27/28 report NIF 2.17, OTP 29 reports 2.18 — those are the two artifact flavours the release builds)
cmake and git
A GGUF model file for testing (e.g. Qwen3.5-0.8B)
An embedding model file for embedding tests (e.g. Qwen3-Embedding-0.6B)

1. Update the submodule

# Fetch latest upstream commits
git -C vendor/llama.cpp fetch origin

# Check what's new since the current pin
git -C vendor/llama.cpp log --oneline HEAD..origin/master

# Checkout the target commit
git -C vendor/llama.cpp checkout <commit-hash>

Then update the Makefile's LLAMA_COMMIT to the same SHA. It is what a Hex source build clones when vendor/llama.cpp is absent, so leaving it behind means source builds get the old llama.cpp while git checkouts get the new one:

git -C vendor/llama.cpp rev-parse HEAD
# paste into LLAMA_COMMIT in Makefile

2. Check API compatibility

Before building, verify the llama.cpp APIs used by the NIF haven't changed:

Re-derive the header list if you add an include -- a hand-kept list drifts, and that is exactly how the common/speculative.h break below reached a build:

grep -hoE '^#include [<"](llama|ggml|chat|json-schema|speculative)[^">]*' \
  c_src/llama_cpp_ex/*.cpp c_src/llama_cpp_ex/*.h | sort -u

for h in include/llama.h \
         ggml/include/ggml-backend.h ggml/include/ggml-rpc.h \
         common/chat.h common/json-schema-to-grammar.h common/speculative.h; do
  echo "##### $h"
  git -C vendor/llama.cpp diff <old-commit>..<new-commit> -- "$h"
done

A signature change is the easy case -- it fails to compile with a clear message. Watch for two harder ones:

A function removed outright. The compiler's "did you mean" is actively misleading: when common_speculative_need_embd was deleted in f785fc9ea, both GCC and clang suggested the unrelated common_speculative_n_max, whose first parameter happens to be a different pointer type. Check the header diff for - lines before believing the suggestion.
A default value changed with the signature intact. Nothing fails to compile. llama_model_default_params() moved load_mode from LLAMA_LOAD_MODE_MMAP to LLAMA_LOAD_MODE_AUTO; because the NIF always sets that field explicitly, behaviour did not change -- but a field we left at its default would have shifted silently. Diff llama_model_default_params / llama_context_default_params on every bump.

The NIF uses these key APIs (grep llama_nif.cpp for the full list):

llama_model_*, llama_context_*, llama_vocab_* — model/context/vocab management
llama_tokenize, llama_detokenize, llama_token_to_piece — tokenization
llama_batch_*, llama_decode — inference
llama_sampler_* — sampling chain
llama_memory_* — KV cache / memory management
llama_get_embeddings_*, llama_pooling_type — embeddings
llama_chat_apply_template — legacy chat templates
common_chat_templates_init, common_chat_templates_apply — Jinja chat templates
json_schema_to_grammar — grammar generation
common_speculative_* — speculative decoding and MTP draft models
ggml_backend_dev_*, ggml_backend_reg_* — device enumeration for :devices
ggml_backend_rpc_add_server, ggml_backend_rpc_start_server — RPC backend

If any signatures changed, update c_src/llama_cpp_ex/llama_nif.cpp and/or llama_nif.h.

Upstream defects we work around

Three known llama.cpp defects have workarounds in this repo. A bump is the only time anyone looks at them, so check each one here — if upstream has fixed it, the workaround should come out rather than quietly accumulate.

Each was measured against 4801e3c567d5 (b10362) on NVIDIA DGX Spark (GB10, aarch64, GCC 13.3, CUDA 13.0). Full reports, with reproductions and suggested upstream fixes, are drafted in .claude/plans/dgx-spark-2node/upstream-issues.md — not yet filed, so there are no issue URLs to link. When they are filed, put the URLs in this table.

Re-checked at e85caa81e (b10582): all three still stand. Every re-check so far has been a source diff, not a re-measurement. At a94d563ed801 the three files involved (ggml/src/ggml-cpu/CMakeLists.txt, ggml_backend_cuda_comm_init, ggml_backend_rpc_start_server) were untouched. At e85caa81e the CUDA and RPC ones are still untouched — #26502 moved the tensor-split meta backend and was reverted in f20395dae — while ggml-cpu/CMakeLists.txt did change: OpenMP target variables, KleidiAI SME2 GEMV sources, and IntelLLVM fast-math gating, none of it near the -mcpu=native probe. A source diff is enough to say a defect is still there; it is not enough to say it is gone, so if a diff ever touches the probe itself, run the command in the last column.

#	Upstream defect	Our workaround	Still needed?
1	`GGML_NATIVE=ON` makes ggml's `-mcpu=native` probe resolve to base ARMv8-A on Cortex-X925/A725 with GCC 13.3 — silently, with a soft CMake warning and exit 0. Costs every `sdot`/`smmla`/SVE kernel.	`LLAMA_CPU_ARM_ARCH` + `LLAMA_CUDA_ARCH` in the `Makefile`, which must be set together. See DGX Spark and Cross-Platform Builds.	`scripts/spark/verify-build-flags.sh` on an aarch64 host. If a default build (no `LLAMA_CPU_ARM_ARCH`) now reports non-zero `sdot`/`smmla`, upstream fixed the probe.
2	`-sm tensor` with a non-CUDA device in the set runs and is correct but ~2.7× slower on decode: `ggml_backend_cuda_comm_init` returns `nullptr` on any non-CUDA member, so the generic meta-backend butterfly runs instead, and the RPC backend's `NULL` 2-D tensor hooks degrade it to a loop of 1-D transfers.	Documented, not coded around: `Model.load/2` maps `:tensor` to its upstream value and the docs say to use `:layer` across hosts. See the tp=2 verdict in DGX Spark.	`mix run bench/spark_tensor_split.exs remote`. If `:tensor` comes within range of `:layer`, upstream implemented the 2-D hooks or the all-reduce — update the verdict section.
3	`ggml_backend_rpc_start_server` returns `void`, never returns on success, and prints failures to stderr, so an embedded caller cannot tell "listening" from "port in use".	`rpc_start_server` in `llama_nif.cpp` pre-`bind()`s the endpoint for a real `errno`, then polls `connect()` until something accepts. A TOCTOU window and one wasted connection per start.	Check whether the signature gained a return value or a listening callback. If so, delete `rpc_preflight_bind` and `rpc_wait_until_listening` and drop the poll.

Not a defect and not going away: RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30), so a peer failure terminates the client process — the BEAM included. That is upstream's deliberate design. LlamaCppEx.RPC documents it; see also the :row split mode, which throws on CUDA at this version and which we deliberately do not work around.

3. Build and test

# Setting LLAMA_BACKEND forces a source build, so no version bump is needed to
# stop the precompiler downloading the old binary. The build stamp is keyed on
# the llama.cpp commit, so the bump from step 1 already forces a rebuild.
LLAMA_BACKEND=cpu mix compile

# Run the suite. The default run needs no model; each opt-in tag names the env
# var for the model it loads (see test/test_helper.exs). GGML_METAL_NO_RESIDENCY
# is Metal-only, and only stops a post-suite assert in llama.cpp's Metal device
# destructor from aborting the VM after a green run.
mix test

GGML_METAL_NO_RESIDENCY=1 \
LLAMA_SMOKE_GEN_MODEL=~/Downloads/Qwen3.5-0.8B-UD-Q4_K_XL.gguf \
LLAMA_SMOKE_EMB_MODEL=~/Downloads/Qwen3-Embedding-0.6B-f16.gguf \
LLAMA_SMOKE_MTP_MODEL=~/Downloads/Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \
  mix test --include smoke --include embeddings --include slow --include mtp

GGML_METAL_NO_RESIDENCY=1 \
LLAMA_SMOKE_MTP_MODEL=~/Downloads/Qwen3.8-27B-Q4_K_M.gguf \
LLAMA_SMOKE_MTP_DRAFT_MODEL=~/Downloads/mtp-Qwen3.8-27B-Q4_0.gguf \
  mix test --include mtp_sidecar

# :rpc_live needs an RPC build AND a reachable worker, and must run with no
# model tag beside it — see test/test_helper.exs for why combining them aborts.
# The worker can be local: another BEAM running LlamaCppEx.RPC.Server, or
# `LLAMA_RPC=1 make rpc-server` and upstream's ggml-rpc-server binary.
LLAMA_RPC=1 LLAMA_BACKEND=metal MIX_ENV=test mix run --no-halt -e \
  'LlamaCppEx.RPC.Server.start_link(endpoint: "127.0.0.1:50052")' &
GGML_METAL_NO_RESIDENCY=1 LLAMA_RPC=1 LLAMA_RPC_ENDPOINT=127.0.0.1:50052 \
  mix test --include rpc_live

# :mtp_cancel is the one tag that is not expected to pass; run it to confirm
# how it fails, and update test/mtp_model_test.exs if the failure mode moved.
GGML_METAL_NO_RESIDENCY=1 \
LLAMA_SMOKE_MTP_MODEL=~/Downloads/Qwen3.6-35B-A3B-MTP-UD-Q4_K_XL.gguf \
  mix test --only mtp_cancel

# Verify formatting and types
mix format --check-formatted
mix dialyzer

Then check that a Hex source build still works, which is the path every LLAMA_BACKEND user and every unlisted target takes. It exercises the Makefile's llama.cpp clone, so it catches a LLAMA_COMMIT that drifted from the submodule:

mix hex.build
d=$(mktemp -d) && tar xf llama_cpp_ex-*.tar -C "$d" && tar xzf "$d"/contents.tar.gz -C "$d"
(cd "$d" && mix deps.get && LLAMA_BACKEND=cpu mix compile)
git -C "$d"/vendor/llama.cpp rev-parse HEAD   # must equal the submodule SHA

4. Update version and changelog

mix.exs: bump @version on LlamaCppEx.MixProject (e.g. "0.8.42" → "0.8.43")
CHANGELOG.md: add a new ## vX.Y.Z section at the top with:
- The submodule commit range and count
- Notable changes categorized by subsystem (follow existing format)

To list commits for the changelog:

git -C vendor/llama.cpp log --oneline <old-commit>..<new-commit>

5. Commit

git add vendor/llama.cpp mix.exs CHANGELOG.md
git commit -m "Bump llama.cpp to <short-hash>, release vX.Y.Z"

6. Tag and push

git tag vX.Y.Z
git push origin master
git push origin vX.Y.Z

The tag push triggers the precompile workflow (.github/workflows/precompile.yml), which does everything including the Hex publish. The jobs run in this order:

prepare_release creates the GitHub Release as a draft, so nothing is visible while assets are still arriving.
precompile (4 legs: macOS/Metal and Linux/CPU × OTP 27 and OTP 29) builds each NIF with LLAMA_PORTABLE=1 and uploads its .tar.gz into the draft. Only the tarballs are uploaded — the .sha256 sidecars stay on the runner so the next job hashes the bytes it actually downloads.
checksum verifies every artifact mix.exs declares is present, flips the release out of draft, runs mix elixir_make.checksum --all, verifies the resulting checksum.exs has an entry for each of them, and commits it to master.
publish checks out the tag (not master), takes only checksum.exs from master, compiles once to verify the published artifact against those checksums, and runs mix hex.publish --yes.

So there is nothing to do by hand after the tag push. Watch the run; if a leg fails, the release stays a draft and nothing reaches Hex.

If you ever need to publish manually — a workflow outage, say — reproduce what publish does rather than publishing from master:

git checkout vX.Y.Z
git fetch origin master
git checkout origin/master -- checksum.exs
mix hex.publish

Troubleshooting

Compilation errors after upgrade

Missing function: check if the API was renamed or removed in include/llama.h
Struct field changes: check llama_model_params, llama_context_params, llama_batch structs
Common library changes: common/chat.h is the most volatile dependency — check common_chat_templates_inputs and common_chat_msg

Build downloads precompiled binary instead of compiling from source

Set LLAMA_BACKEND (to cpu if you do not care which). Any value flips make_force_build in mix.exs and skips the download entirely. Bumping @version also works, but only because no artifact exists for the new version yet.

CI precompile fails

Check .github/workflows/precompile.yml. Common issues:

New llama.cpp dependencies not available in CI runners
CMake flag changes requiring updates to the Makefile
The tag is not strict semver. Every job re-derives the version from GITHUB_REF and refuses anything that is not X.Y.Z[-pre][+build], because that value is interpolated into a sed script. vX.Y.Z-rc1 is fine, v1.2 and vlatest are not.
A matrix leg failed. The release then stays a draft and nothing is published to Hex. Fix the leg and re-run the workflow; prepare_release reuses the existing draft and the uploads use --clobber.
checksum.exs came back incomplete. mix elixir_make.checksum prints an error but still exits 0 when an artifact download fails, so the workflow re-checks the file against the artifact list derived from mix.exs and fails the release itself. Re-running is usually enough.

← Previous Page Performance Guide

Next Page → ADR 001: C++ NIF Over Rustler