Release Guide

How to upgrade the llama.cpp submodule and publish a new release.

Prerequisites

• Elixir 1.18+, Erlang/OTP 26+ (OTP 26/27/28 report NIF 2.17, OTP 29 reports 2.18 — those are the two artifact flavours the release builds)
• cmake and git
• A GGUF model file for testing (e.g. Qwen3.5-0.8B)
• An embedding model file for embedding tests (e.g. Qwen3-Embedding-0.6B)

1. Update the submodule

# Fetch latest upstream commits
git -C vendor/llama.cpp fetch origin

# Check what's new since the current pin
git -C vendor/llama.cpp log --oneline HEAD..origin/master

# Checkout the target commit
git -C vendor/llama.cpp checkout <commit-hash>

Then update the Makefile's LLAMA_COMMIT to the same SHA. It is what a Hex source build clones when vendor/llama.cpp is absent, so leaving it behind means source builds get the old llama.cpp while git checkouts get the new one:

git -C vendor/llama.cpp rev-parse HEAD
# paste into LLAMA_COMMIT in Makefile

2. Check API compatibility

Before building, verify the llama.cpp APIs used by the NIF haven't changed:

Re-derive the header list if you add an include -- a hand-kept list drifts, and that is exactly how the common/speculative.h break below reached a build:

grep -hoE '^#include [<"](llama|ggml|chat|json-schema|speculative)[^">]*' \
  c_src/llama_cpp_ex/*.cpp c_src/llama_cpp_ex/*.h | sort -u

for h in include/llama.h \
         ggml/include/ggml-backend.h ggml/include/ggml-rpc.h \
         common/chat.h common/json-schema-to-grammar.h common/speculative.h; do
  echo "##### $h"
  git -C vendor/llama.cpp diff <old-commit>..<new-commit> -- "$h"
done

A signature change is the easy case -- it fails to compile with a clear message. Watch for two harder ones:

• A function removed outright. The compiler's "did you mean" is actively misleading: when common_speculative_need_embd was deleted in f785fc9ea, both GCC and clang suggested the unrelated common_speculative_n_max, whose first parameter happens to be a different pointer type. Check the header diff for - lines before believing the suggestion.
• A default value changed with the signature intact. Nothing fails to compile. llama_model_default_params() moved load_mode from LLAMA_LOAD_MODE_MMAP to LLAMA_LOAD_MODE_AUTO; because the NIF always sets that field explicitly, behaviour did not change -- but a field we left at its default would have shifted silently. Diff llama_model_default_params / llama_context_default_params on every bump.

The NIF uses these key APIs (grep llama_nif.cpp for the full list):

• llama_model_*, llama_context_*, llama_vocab_* — model/context/vocab management
• llama_tokenize, llama_detokenize, llama_token_to_piece — tokenization
• llama_batch_*, llama_decode — inference
• llama_sampler_* — sampling chain
• llama_memory_* — KV cache / memory management
• llama_get_embeddings_*, llama_pooling_type — embeddings
• llama_chat_apply_template — legacy chat templates
• common_chat_templates_init, common_chat_templates_apply — Jinja chat templates
• json_schema_to_grammar — grammar generation
• common_speculative_* — speculative decoding and MTP draft models
• ggml_backend_dev_*, ggml_backend_reg_* — device enumeration for :devices
• ggml_backend_rpc_add_server, ggml_backend_rpc_start_server — RPC backend

If any signatures changed, update c_src/llama_cpp_ex/llama_nif.cpp and/or llama_nif.h.

Upstream defects we work around

Three known llama.cpp defects have workarounds in this repo. A bump is the only time anyone looks at them, so check each one here — if upstream has fixed it, the workaround should come out rather than quietly accumulate.

Each was measured against 4801e3c567d5 (b10362) on NVIDIA DGX Spark (GB10, aarch64, GCC 13.3, CUDA 13.0). Full reports, with reproductions and suggested upstream fixes, are drafted in .claude/plans/dgx-spark-2node/upstream-issues.md — not yet filed, so there are no issue URLs to link. When they are filed, put the URLs in this table.

Re-checked at a94d563ed801 (61 commits later): all three still stand. That check was a source diff, not a re-measurement — the files each defect lives in (ggml/src/ggml-cpu/CMakeLists.txt, ggml_backend_cuda_comm_init, and ggml_backend_rpc_start_server) were untouched by the bump. A source diff is enough to say a defect is still there; it is not enough to say it is gone, so if a diff ever shows movement, run the command in the last column.

Not a defect and not going away: RPC_STATUS_ASSERT is GGML_ABORT (ggml-rpc.cpp:30), so a peer failure terminates the client process — the BEAM included. That is upstream's deliberate design. LlamaCppEx.RPC documents it; see also the :row split mode, which throws on CUDA at this version and which we deliberately do not work around.

3. Build and test

# Setting LLAMA_BACKEND forces a source build, so no version bump is needed to
# stop the precompiler downloading the old binary. The build stamp is keyed on
# the llama.cpp commit, so the bump from step 1 already forces a rebuild.
LLAMA_BACKEND=cpu mix compile

# Run full test suite
LLAMA_MODEL_PATH=~/Downloads/Qwen3.5-0.8B-UD-Q4_K_XL.gguf \
LLAMA_EMBEDDING_MODEL_PATH=~/Downloads/Qwen3-Embedding-0.6B-f16.gguf \
mix test

# Verify formatting and types
mix format --check-formatted
mix dialyzer

Then check that a Hex source build still works, which is the path every LLAMA_BACKEND user and every unlisted target takes. It exercises the Makefile's llama.cpp clone, so it catches a LLAMA_COMMIT that drifted from the submodule:

mix hex.build
d=$(mktemp -d) && tar xf llama_cpp_ex-*.tar -C "$d" && tar xzf "$d"/contents.tar.gz -C "$d"
(cd "$d" && mix deps.get && LLAMA_BACKEND=cpu mix compile)
git -C "$d"/vendor/llama.cpp rev-parse HEAD   # must equal the submodule SHA

4. Update version and changelog

1. mix.exs: bump @version on LlamaCppEx.MixProject (e.g. "0.8.42" → "0.8.43")
2. CHANGELOG.md: add a new ## vX.Y.Z section at the top with:
   • The submodule commit range and count
   • Notable changes categorized by subsystem (follow existing format)

To list commits for the changelog:

git -C vendor/llama.cpp log --oneline <old-commit>..<new-commit>

5. Commit

git add vendor/llama.cpp mix.exs CHANGELOG.md
git commit -m "Bump llama.cpp to <short-hash>, release vX.Y.Z"

6. Tag and push

git tag vX.Y.Z
git push origin master
git push origin vX.Y.Z

The tag push triggers the precompile workflow (.github/workflows/precompile.yml), which does everything including the Hex publish. The jobs run in this order:

1. prepare_release creates the GitHub Release as a draft, so nothing is visible while assets are still arriving.
2. precompile (4 legs: macOS/Metal and Linux/CPU × OTP 27 and OTP 29) builds each NIF with LLAMA_PORTABLE=1 and uploads its .tar.gz into the draft. Only the tarballs are uploaded — the .sha256 sidecars stay on the runner so the next job hashes the bytes it actually downloads.
3. checksum verifies every artifact mix.exs declares is present, flips the release out of draft, runs mix elixir_make.checksum --all, verifies the resulting checksum.exs has an entry for each of them, and commits it to master.
4. publish checks out the tag (not master), takes only checksum.exs from master, compiles once to verify the published artifact against those checksums, and runs mix hex.publish --yes.

So there is nothing to do by hand after the tag push. Watch the run; if a leg fails, the release stays a draft and nothing reaches Hex.

If you ever need to publish manually — a workflow outage, say — reproduce what publish does rather than publishing from master:

git checkout vX.Y.Z
git fetch origin master
git checkout origin/master -- checksum.exs
mix hex.publish

Troubleshooting

Compilation errors after upgrade

• Missing function: check if the API was renamed or removed in include/llama.h
• Struct field changes: check llama_model_params, llama_context_params, llama_batch structs
• Common library changes: common/chat.h is the most volatile dependency — check common_chat_templates_inputs and common_chat_msg

Build downloads precompiled binary instead of compiling from source

Set LLAMA_BACKEND (to cpu if you do not care which). Any value flips make_force_build in mix.exs and skips the download entirely. Bumping @version also works, but only because no artifact exists for the new version yet.

CI precompile fails

Check .github/workflows/precompile.yml. Common issues:

• New llama.cpp dependencies not available in CI runners
• CMake flag changes requiring updates to the Makefile
• The tag is not strict semver. Every job re-derives the version from GITHUB_REF and refuses anything that is not X.Y.Z[-pre][+build], because that value is interpolated into a sed script. vX.Y.Z-rc1 is fine, v1.2 and vlatest are not.
• A matrix leg failed. The release then stays a draft and nothing is published to Hex. Fix the leg and re-run the workflow; prepare_release reuses the existing draft and the uploads use --clobber.
• checksum.exs came back incomplete. mix elixir_make.checksum prints an error but still exits 0 when an artifact download fails, so the workflow re-checks the file against the artifact list derived from mix.exs and fails the release itself. Re-running is usually enough.