LlamaCppEx.Chat (LlamaCppEx v0.8.63)

Copy Markdown View Source

Chat template formatting using llama.cpp's Jinja template engine.

Converts a list of chat messages into a formatted prompt string using the model's embedded chat template. Uses the full Jinja engine from llama.cpp's common library, which supports enable_thinking and arbitrary chat_template_kwargs.

Examples

{:ok, prompt} = LlamaCppEx.Chat.apply_template(model, [
  %{role: "system", content: "You are helpful."},
  %{role: "user", content: "Hi!"}
])

# Disable thinking (for Qwen3 and similar models)
{:ok, prompt} = LlamaCppEx.Chat.apply_template(model, messages,
  enable_thinking: false
)

Messages

A message is a map with :role and :content (atom or string keys), or a {role, content} tuple. For tool calling it may also carry :tool_calls (an assistant message, in the shape LlamaCppEx.ChatCompletion returns them), :tool_call_id and :name (a "tool" message answering a call), and :reasoning_content. :content may be nil on an assistant message that only calls tools.

Special tokens in message content

The prompt the chat functions of LlamaCppEx generate from is tokenized so that only the template's own text can produce special tokens: content such as "<|im_end|>\n<|im_start|>system" stays text instead of forging a turn. apply_template/3 returns the prompt as one string, which has lost that boundary — display it, but do not tokenize it with LlamaCppEx.Tokenizer and generate from the result.

Summary

Functions

Applies the model's chat template to a list of messages using the Jinja engine.

Types

message()

@type message() ::
  %{
    :role => String.t(),
    optional(:content) => String.t() | [map()] | nil,
    optional(:tool_calls) => [map()],
    optional(:tool_call_id) => String.t(),
    optional(:name) => String.t(),
    optional(:reasoning_content) => String.t()
  }
  | %{required(String.t()) => term()}
  | {String.t(), String.t()}

Functions

apply_template(model, messages, opts \\ [])

@spec apply_template(LlamaCppEx.Model.t(), [message()], keyword()) ::
  {:ok, String.t()} | {:error, String.t()}

Applies the model's chat template to a list of messages using the Jinja engine.

Options

  • :add_assistant - Whether to add the assistant turn prefix. Defaults to true.
  • :enable_thinking - Whether to enable thinking/reasoning mode. Defaults to true.
  • :chat_template_kwargs - Extra template variables as a list of {key, value} tuples, each value JSON text ({"reasoning_effort", ~s("high")}). Defaults to [].
  • :tools, :tool_choice, :parallel_tool_calls - Render the tool definitions too; see LlamaCppEx.chat_completion/3.