Elixir bindings for llama.cpp.
Provides a high-level API for loading GGUF models and generating text.
Quick Start
# Initialize the backend (once per application)
:ok = LlamaCppEx.init()
# Load a model
{:ok, model} = LlamaCppEx.load_model("model.gguf", n_gpu_layers: -1)
# Generate text
{:ok, text} = LlamaCppEx.generate(model, "Once upon a time", max_tokens: 200)Lower-level API
For fine-grained control, use the individual modules:
LlamaCppEx.Model- Model loading and introspectionLlamaCppEx.Context- Inference context with KV cacheLlamaCppEx.Sampler- Token sampling configurationLlamaCppEx.Tokenizer- Text tokenization and detokenizationLlamaCppEx.Embedding- Embedding generationLlamaCppEx.Decision- Typed decisions with a decision model
Summary
Functions
Applies the chat template and generates a response.
Generates an OpenAI-compatible chat completion response.
Answers typed questions about state with a decision model.
Lists the available ggml backend devices (GPUs, integrated GPUs, accelerators, and the CPU).
Computes an embedding for a single text.
Computes embeddings for multiple texts.
Generates text from a prompt.
Initializes the llama.cpp backend. Call once at application start.
Loads a GGUF model from the given file path.
Downloads a GGUF model from HuggingFace Hub and loads it.
Returns a lazy stream of generated text chunks (tokens).
Returns a lazy stream of chat response chunks.
Returns a lazy stream of OpenAI-compatible chat completion chunks.
Functions
@spec chat(LlamaCppEx.Model.t(), [LlamaCppEx.Chat.message()], keyword()) :: {:ok, String.t()} | {:error, term()}
Applies the chat template and generates a response.
Options
Accepts all options from generate/3 plus the chat-templating options from
LlamaCppEx.Chat.apply_template/3 (:add_assistant, :enable_thinking,
:chat_template_kwargs).
Special-token text in message content (<|im_start|>, [INST], ...) is
tokenized as text, so a message cannot forge a turn of the conversation; see
LlamaCppEx.Chat.
Examples
{:ok, reply} = LlamaCppEx.chat(model, [
%{role: "system", content: "You are helpful."},
%{role: "user", content: "What is Elixir?"}
], max_tokens: 200)
@spec chat_completion( LlamaCppEx.Model.t() | GenServer.server(), [LlamaCppEx.Chat.message()], keyword() ) :: {:ok, LlamaCppEx.ChatCompletion.t()} | {:error, term()}
Generates an OpenAI-compatible chat completion response.
Applies the chat template, runs generation, and returns a %ChatCompletion{}
struct with choices, usage counts, and finish reason.
Options
Accepts all options from generate/3 plus the chat-templating options from
LlamaCppEx.Chat.apply_template/3 (:add_assistant, :enable_thinking,
:chat_template_kwargs), and:
:tools- Tools the model may call, in the OpenAI shape:[%{type: "function", function: %{name: "get_weather", description: "...", parameters: json_schema}}]. The model's chat template renders them and supplies the grammar and parser for its tool-call format; a template without tool support returns{:error, reason}saying so.:tool_choice-:auto(default: the model decides),:none(the tools are left out of the prompt, so the reply is text),:required(the grammar does not let the reply end without a call; the model may write text before it), or%{type: "function", function: %{name: name}}to offer only that tool and require a call to it.:parallel_tool_calls- Allow several calls in one reply. Defaults tofalse.
With tools, a reply that calls one has finish_reason: "tool_calls" and its
message carries tool_calls: [%{id: "call_…", type: "function", function: %{name: name, arguments: json_text}}], with content: nil when the model
wrote nothing else. Answer each call with a message %{role: "tool", tool_call_id: id, content: result}, after the assistant message that made
it, and call again. reasoning_content is split out by the template's own
parser, whatever :enable_thinking says. :grammar and :json_schema
cannot be combined with tools (ArgumentError) unless tool_choice: :none.
Examples
{:ok, completion} = LlamaCppEx.chat_completion(model, [
%{role: "user", content: "What is Elixir?"}
], max_tokens: 200)
completion.choices |> hd() |> Map.get(:message) |> Map.get(:content)Errors
Returns {:error, reason} when templating, tokenization, or generation fails,
{:error, :timeout} when generation does not complete within :timeout,
{:error, {:invalid_json_schema, message}} when :json_schema cannot be
converted to a grammar, and {:error, {:chat_parse_failed, message}} when a
reply does not parse in the template's tool-call format.
Breaking change in 0.8.40
Both cases previously returned {:ok, %ChatCompletion{}} — a NIF error was
reported as finish_reason: "stop" and a timeout as "length", so a caller
matching only {:ok, _} silently used truncated or empty output. The error
branch of this function's @spec was unreachable.
Server routing
When given a running LlamaCppEx.Server (pid or name) instead of a
%Model{}, the request is served by the batching server: chat templating
and tokenization happen in the caller, and the multi-turn prompt benefits
from the server's prefix cache. Server-only request options like
:session and :cache_prompt are forwarded.
@spec decide(LlamaCppEx.Model.t(), term(), LlamaCppEx.Decision.questions(), keyword()) :: {:ok, LlamaCppEx.Decision.result()} | {:error, String.t()}
Answers typed questions about state with a decision model.
Creates a LlamaCppEx.Decision (and its context) for this one call. To send
several requests, create it once with LlamaCppEx.Decision.new/2 and call
LlamaCppEx.Decision.decide/3 on it. See LlamaCppEx.Decision for the shape
of state, questions and the answers, and LlamaCppEx.Decision.new/2 for
options.
@spec devices() :: [map()]
Lists the available ggml backend devices (GPUs, integrated GPUs, accelerators, and the CPU).
Each entry is a map with :index (ggml device order), :gpu_index (0-based
among GPU/IGPU devices, matching :tensor_split's index space, or nil for
non-GPU devices), :name, :description, :type (:gpu, :igpu, :cpu,
:accel, or :other), :backend (e.g. "CUDA", "Metal"), :memory_total,
and :memory_free (bytes). On Metal, unified memory means a single device.
Examples
:ok = LlamaCppEx.init()
LlamaCppEx.devices()
#=> [%{gpu_index: 0, type: :gpu, name: "NVIDIA RTX 4090",
# memory_total: 24_000_000_000, memory_free: 23_500_000_000, ...}, ...]
@spec embed(LlamaCppEx.Model.t(), String.t(), keyword()) :: {:ok, LlamaCppEx.Embedding.t()} | {:error, String.t()}
Computes an embedding for a single text.
See LlamaCppEx.Embedding.embed/3 for options.
@spec embed_batch(LlamaCppEx.Model.t(), [String.t()], keyword()) :: {:ok, [LlamaCppEx.Embedding.t()]} | {:error, String.t()}
Computes embeddings for multiple texts.
See LlamaCppEx.Embedding.embed_batch/3 for options.
@spec generate(LlamaCppEx.Model.t(), String.t(), keyword()) :: {:ok, String.t()} | {:error, term()}
Generates text from a prompt.
Creates a temporary context and sampler, tokenizes the prompt, runs generation, and returns the generated text.
Options
:max_tokens- Maximum tokens to generate. Defaults to256.:n_ctx- Context size. Defaults to2048.:temp- Sampling temperature.0.0for greedy. Defaults to0.8.:top_k- Top-K filtering. Defaults to40.:top_p- Top-P (nucleus) filtering. Defaults to0.95.:min_p- Min-P filtering. Defaults to0.05.:seed- Random seed. Defaults to random.:penalty_repeat- Repetition penalty. Defaults to1.0.:penalty_freq- Frequency penalty (0.0–2.0). Defaults to0.0.:penalty_present- Presence penalty (0.0–2.0). Defaults to0.0.:grammar- GBNF grammar string for constrained generation.:grammar_root- Root rule name for grammar. Defaults to"root".:json_schema- JSON Schema map for structured output. Automatically converted to a GBNF grammar. Cannot be used together with:grammar. Tip: set"additionalProperties" => falsefor tighter grammars.
Errors
Returns {:error, {:invalid_json_schema, message}} when llama.cpp cannot
convert :json_schema to a grammar (an unresolvable $ref, say), with the
converter's message. Passing both :grammar and :json_schema raises
ArgumentError.
@spec init() :: :ok
Initializes the llama.cpp backend. Call once at application start.
@spec load_model( String.t(), keyword() ) :: {:ok, LlamaCppEx.Model.t()} | {:error, String.t()}
Loads a GGUF model from the given file path.
See LlamaCppEx.Model.load/2 for options.
@spec load_model_from_hub(String.t(), String.t(), keyword()) :: {:ok, LlamaCppEx.Model.t()} | {:error, String.t()}
Downloads a GGUF model from HuggingFace Hub and loads it.
Requires the optional :req dependency.
Examples
:ok = LlamaCppEx.init()
{:ok, model} = LlamaCppEx.load_model_from_hub(
"Qwen/Qwen3-4B-GGUF",
"qwen3-4b-q4_k_m.gguf",
n_gpu_layers: -1
)Options
Accepts all options from load_model/2 plus:
:cache_dir- Local cache directory for downloaded models.:token- HuggingFace API token for private repos.:progress- Download progress callback.:revision- Git revision (branch, tag, commit). Defaults to"main".
@spec stream(LlamaCppEx.Model.t(), String.t(), keyword()) :: Enumerable.t()
Returns a lazy stream of generated text chunks (tokens).
Each element is a string (the text piece for one token). The stream ends
when an end-of-generation token is produced or max_tokens is reached.
Accepts the same options as generate/3, except that :timeout bounds the
wait for the next chunk rather than the whole generation and therefore
defaults to 30000 ms.
Errors
A generation failure is emitted as a final {:error, reason} element rather
than truncating the stream silently, matching LlamaCppEx.Server.stream/3.
Elements are otherwise always strings, so Enum.join/1 on a stream that may
fail is unsafe — match on the element shape if you care.
A :json_schema that cannot be converted is reported the same way, before
anything is generated: the stream's only element is
{:error, {:invalid_json_schema, message}}.
Examples
model
|> LlamaCppEx.stream("Tell me a story", max_tokens: 500)
|> Enum.each(&IO.write/1)
@spec stream_chat(LlamaCppEx.Model.t(), [LlamaCppEx.Chat.message()], keyword()) :: Enumerable.t()
Returns a lazy stream of chat response chunks.
Applies the chat template and streams the generated response token by token.
Accepts same options as chat/3.
Like stream/3, a failure is emitted as a final {:error, reason} element.
@spec stream_chat_completion( LlamaCppEx.Model.t() | GenServer.server(), [LlamaCppEx.Chat.message()], keyword() ) :: Enumerable.t()
Returns a lazy stream of OpenAI-compatible chat completion chunks.
Each element is a %ChatCompletionChunk{} struct. The first chunk contains
delta: %{role: "assistant", content: ""}. Subsequent chunks contain
delta: %{content: "token_text"}. The final chunk contains the finish_reason.
All chunks share the same id and created timestamp.
With :tools, deltas come from the template's parser instead: %{content: _},
%{reasoning_content: _}, and %{tool_calls: [%{index: i, id: id, type: "function", function: %{name: name, arguments: ""}}]} when a call starts,
then %{tool_calls: [%{index: i, function: %{arguments: fragment}}]} as its
arguments arrive, as OpenAI streams them. A reply that called a tool ends with
finish_reason: "tool_calls". Concatenating a call's argument fragments gives
the arguments chat_completion/3 returns.
Options
Accepts same options as chat_completion/3. :timeout bounds the wait for the
next chunk rather than the whole generation, so it defaults to
30000 ms for both the model- and
server-routed forms. It previously flipped between 60s and 30s depending on the
type of the first argument.
Errors
A failure is emitted as a final {:error, reason} element, and a chunk timeout
as {:error, :timeout}, rather than truncating the stream silently. Every other
element is a %ChatCompletionChunk{}. A :json_schema that cannot be
converted makes {:error, {:invalid_json_schema, message}} the only element.
Examples
model
|> LlamaCppEx.stream_chat_completion(messages, max_tokens: 200)
|> Enum.each(fn
%LlamaCppEx.ChatCompletionChunk{} = chunk ->
chunk.choices |> hd() |> get_in([:delta, :content]) |> IO.write()
{:error, reason} ->
IO.warn("generation failed: #{inspect(reason)}")
end)