LlamaCppEx (LlamaCppEx v0.8.63)

Copy Markdown View Source

Elixir bindings for llama.cpp.

Provides a high-level API for loading GGUF models and generating text.

Quick Start

# Initialize the backend (once per application)
:ok = LlamaCppEx.init()

# Load a model
{:ok, model} = LlamaCppEx.load_model("model.gguf", n_gpu_layers: -1)

# Generate text
{:ok, text} = LlamaCppEx.generate(model, "Once upon a time", max_tokens: 200)

Lower-level API

For fine-grained control, use the individual modules:

Summary

Functions

Applies the chat template and generates a response.

Generates an OpenAI-compatible chat completion response.

Answers typed questions about state with a decision model.

Lists the available ggml backend devices (GPUs, integrated GPUs, accelerators, and the CPU).

Computes an embedding for a single text.

Computes embeddings for multiple texts.

Generates text from a prompt.

Initializes the llama.cpp backend. Call once at application start.

Loads a GGUF model from the given file path.

Downloads a GGUF model from HuggingFace Hub and loads it.

Returns a lazy stream of generated text chunks (tokens).

Returns a lazy stream of chat response chunks.

Returns a lazy stream of OpenAI-compatible chat completion chunks.

Functions

chat(model, messages, opts \\ [])

@spec chat(LlamaCppEx.Model.t(), [LlamaCppEx.Chat.message()], keyword()) ::
  {:ok, String.t()} | {:error, term()}

Applies the chat template and generates a response.

Options

Accepts all options from generate/3 plus the chat-templating options from LlamaCppEx.Chat.apply_template/3 (:add_assistant, :enable_thinking, :chat_template_kwargs).

Special-token text in message content (<|im_start|>, [INST], ...) is tokenized as text, so a message cannot forge a turn of the conversation; see LlamaCppEx.Chat.

Examples

{:ok, reply} = LlamaCppEx.chat(model, [
  %{role: "system", content: "You are helpful."},
  %{role: "user", content: "What is Elixir?"}
], max_tokens: 200)

chat_completion(model_or_server, messages, opts \\ [])

@spec chat_completion(
  LlamaCppEx.Model.t() | GenServer.server(),
  [LlamaCppEx.Chat.message()],
  keyword()
) ::
  {:ok, LlamaCppEx.ChatCompletion.t()} | {:error, term()}

Generates an OpenAI-compatible chat completion response.

Applies the chat template, runs generation, and returns a %ChatCompletion{} struct with choices, usage counts, and finish reason.

Options

Accepts all options from generate/3 plus the chat-templating options from LlamaCppEx.Chat.apply_template/3 (:add_assistant, :enable_thinking, :chat_template_kwargs), and:

  • :tools - Tools the model may call, in the OpenAI shape: [%{type: "function", function: %{name: "get_weather", description: "...", parameters: json_schema}}]. The model's chat template renders them and supplies the grammar and parser for its tool-call format; a template without tool support returns {:error, reason} saying so.
  • :tool_choice - :auto (default: the model decides), :none (the tools are left out of the prompt, so the reply is text), :required (the grammar does not let the reply end without a call; the model may write text before it), or %{type: "function", function: %{name: name}} to offer only that tool and require a call to it.
  • :parallel_tool_calls - Allow several calls in one reply. Defaults to false.

With tools, a reply that calls one has finish_reason: "tool_calls" and its message carries tool_calls: [%{id: "call_…", type: "function", function: %{name: name, arguments: json_text}}], with content: nil when the model wrote nothing else. Answer each call with a message %{role: "tool", tool_call_id: id, content: result}, after the assistant message that made it, and call again. reasoning_content is split out by the template's own parser, whatever :enable_thinking says. :grammar and :json_schema cannot be combined with tools (ArgumentError) unless tool_choice: :none.

Examples

{:ok, completion} = LlamaCppEx.chat_completion(model, [
  %{role: "user", content: "What is Elixir?"}
], max_tokens: 200)

completion.choices |> hd() |> Map.get(:message) |> Map.get(:content)

Errors

Returns {:error, reason} when templating, tokenization, or generation fails, {:error, :timeout} when generation does not complete within :timeout, {:error, {:invalid_json_schema, message}} when :json_schema cannot be converted to a grammar, and {:error, {:chat_parse_failed, message}} when a reply does not parse in the template's tool-call format.

Breaking change in 0.8.40

Both cases previously returned {:ok, %ChatCompletion{}} — a NIF error was reported as finish_reason: "stop" and a timeout as "length", so a caller matching only {:ok, _} silently used truncated or empty output. The error branch of this function's @spec was unreachable.

Server routing

When given a running LlamaCppEx.Server (pid or name) instead of a %Model{}, the request is served by the batching server: chat templating and tokenization happen in the caller, and the multi-turn prompt benefits from the server's prefix cache. Server-only request options like :session and :cache_prompt are forwarded.

decide(model, state, questions, opts \\ [])

Answers typed questions about state with a decision model.

Creates a LlamaCppEx.Decision (and its context) for this one call. To send several requests, create it once with LlamaCppEx.Decision.new/2 and call LlamaCppEx.Decision.decide/3 on it. See LlamaCppEx.Decision for the shape of state, questions and the answers, and LlamaCppEx.Decision.new/2 for options.

devices()

@spec devices() :: [map()]

Lists the available ggml backend devices (GPUs, integrated GPUs, accelerators, and the CPU).

Each entry is a map with :index (ggml device order), :gpu_index (0-based among GPU/IGPU devices, matching :tensor_split's index space, or nil for non-GPU devices), :name, :description, :type (:gpu, :igpu, :cpu, :accel, or :other), :backend (e.g. "CUDA", "Metal"), :memory_total, and :memory_free (bytes). On Metal, unified memory means a single device.

Examples

:ok = LlamaCppEx.init()
LlamaCppEx.devices()
#=> [%{gpu_index: 0, type: :gpu, name: "NVIDIA RTX 4090",
#      memory_total: 24_000_000_000, memory_free: 23_500_000_000, ...}, ...]

embed(model, text, opts \\ [])

@spec embed(LlamaCppEx.Model.t(), String.t(), keyword()) ::
  {:ok, LlamaCppEx.Embedding.t()} | {:error, String.t()}

Computes an embedding for a single text.

See LlamaCppEx.Embedding.embed/3 for options.

embed_batch(model, texts, opts \\ [])

@spec embed_batch(LlamaCppEx.Model.t(), [String.t()], keyword()) ::
  {:ok, [LlamaCppEx.Embedding.t()]} | {:error, String.t()}

Computes embeddings for multiple texts.

See LlamaCppEx.Embedding.embed_batch/3 for options.

generate(model, prompt, opts \\ [])

@spec generate(LlamaCppEx.Model.t(), String.t(), keyword()) ::
  {:ok, String.t()} | {:error, term()}

Generates text from a prompt.

Creates a temporary context and sampler, tokenizes the prompt, runs generation, and returns the generated text.

Options

  • :max_tokens - Maximum tokens to generate. Defaults to 256.
  • :n_ctx - Context size. Defaults to 2048.
  • :temp - Sampling temperature. 0.0 for greedy. Defaults to 0.8.
  • :top_k - Top-K filtering. Defaults to 40.
  • :top_p - Top-P (nucleus) filtering. Defaults to 0.95.
  • :min_p - Min-P filtering. Defaults to 0.05.
  • :seed - Random seed. Defaults to random.
  • :penalty_repeat - Repetition penalty. Defaults to 1.0.
  • :penalty_freq - Frequency penalty (0.0–2.0). Defaults to 0.0.
  • :penalty_present - Presence penalty (0.0–2.0). Defaults to 0.0.
  • :grammar - GBNF grammar string for constrained generation.
  • :grammar_root - Root rule name for grammar. Defaults to "root".
  • :json_schema - JSON Schema map for structured output. Automatically converted to a GBNF grammar. Cannot be used together with :grammar. Tip: set "additionalProperties" => false for tighter grammars.

Errors

Returns {:error, {:invalid_json_schema, message}} when llama.cpp cannot convert :json_schema to a grammar (an unresolvable $ref, say), with the converter's message. Passing both :grammar and :json_schema raises ArgumentError.

init()

@spec init() :: :ok

Initializes the llama.cpp backend. Call once at application start.

load_model(path, opts \\ [])

@spec load_model(
  String.t(),
  keyword()
) :: {:ok, LlamaCppEx.Model.t()} | {:error, String.t()}

Loads a GGUF model from the given file path.

See LlamaCppEx.Model.load/2 for options.

load_model_from_hub(repo_id, filename, opts \\ [])

@spec load_model_from_hub(String.t(), String.t(), keyword()) ::
  {:ok, LlamaCppEx.Model.t()} | {:error, String.t()}

Downloads a GGUF model from HuggingFace Hub and loads it.

Requires the optional :req dependency.

Examples

:ok = LlamaCppEx.init()
{:ok, model} = LlamaCppEx.load_model_from_hub(
  "Qwen/Qwen3-4B-GGUF",
  "qwen3-4b-q4_k_m.gguf",
  n_gpu_layers: -1
)

Options

Accepts all options from load_model/2 plus:

  • :cache_dir - Local cache directory for downloaded models.
  • :token - HuggingFace API token for private repos.
  • :progress - Download progress callback.
  • :revision - Git revision (branch, tag, commit). Defaults to "main".

stream(model, prompt, opts \\ [])

@spec stream(LlamaCppEx.Model.t(), String.t(), keyword()) :: Enumerable.t()

Returns a lazy stream of generated text chunks (tokens).

Each element is a string (the text piece for one token). The stream ends when an end-of-generation token is produced or max_tokens is reached.

Accepts the same options as generate/3, except that :timeout bounds the wait for the next chunk rather than the whole generation and therefore defaults to 30000 ms.

Errors

A generation failure is emitted as a final {:error, reason} element rather than truncating the stream silently, matching LlamaCppEx.Server.stream/3. Elements are otherwise always strings, so Enum.join/1 on a stream that may fail is unsafe — match on the element shape if you care.

A :json_schema that cannot be converted is reported the same way, before anything is generated: the stream's only element is {:error, {:invalid_json_schema, message}}.

Examples

model
|> LlamaCppEx.stream("Tell me a story", max_tokens: 500)
|> Enum.each(&IO.write/1)

stream_chat(model, messages, opts \\ [])

Returns a lazy stream of chat response chunks.

Applies the chat template and streams the generated response token by token. Accepts same options as chat/3.

Like stream/3, a failure is emitted as a final {:error, reason} element.

stream_chat_completion(model_or_server, messages, opts \\ [])

@spec stream_chat_completion(
  LlamaCppEx.Model.t() | GenServer.server(),
  [LlamaCppEx.Chat.message()],
  keyword()
) :: Enumerable.t()

Returns a lazy stream of OpenAI-compatible chat completion chunks.

Each element is a %ChatCompletionChunk{} struct. The first chunk contains delta: %{role: "assistant", content: ""}. Subsequent chunks contain delta: %{content: "token_text"}. The final chunk contains the finish_reason.

All chunks share the same id and created timestamp.

With :tools, deltas come from the template's parser instead: %{content: _}, %{reasoning_content: _}, and %{tool_calls: [%{index: i, id: id, type: "function", function: %{name: name, arguments: ""}}]} when a call starts, then %{tool_calls: [%{index: i, function: %{arguments: fragment}}]} as its arguments arrive, as OpenAI streams them. A reply that called a tool ends with finish_reason: "tool_calls". Concatenating a call's argument fragments gives the arguments chat_completion/3 returns.

Options

Accepts same options as chat_completion/3. :timeout bounds the wait for the next chunk rather than the whole generation, so it defaults to 30000 ms for both the model- and server-routed forms. It previously flipped between 60s and 30s depending on the type of the first argument.

Errors

A failure is emitted as a final {:error, reason} element, and a chunk timeout as {:error, :timeout}, rather than truncating the stream silently. Every other element is a %ChatCompletionChunk{}. A :json_schema that cannot be converted makes {:error, {:invalid_json_schema, message}} the only element.

Examples

model
|> LlamaCppEx.stream_chat_completion(messages, max_tokens: 200)
|> Enum.each(fn
  %LlamaCppEx.ChatCompletionChunk{} = chunk ->
    chunk.choices |> hd() |> get_in([:delta, :content]) |> IO.write()

  {:error, reason} ->
    IO.warn("generation failed: #{inspect(reason)}")
end)