Typed decisions with a decision model: llama.cpp's /v1/systemone API, in-process.

A decision model answers typed questions about a state in one forward pass per prompt; no token is generated. Each answer is a probability distribution, so it comes with a confidence instead of free text. This is llama-server's TypeSafe-compatible System One API, ported into the NIF: same request shape, same prompts, same answers.

Models

The model file says what kind of decision model it is ("<arch>.decision.type"); model_type/1 reads it. Supported types:

• :openjev, :lev, :nimble - read the logits of one label token per option (:nimble lists every question of the request in each prompt, :lev asks each choice twice with the options reversed).
• :kev - scores each option by a dot product of hidden states.
• :laya - a ModernBERT encoder; one score per [MASK] marker.
• :clef - reads every question in one prompt and decides them jointly.

Decision GGUFs are published in ggml-org's "Decision models" collection on Hugging Face, for example ggml-org/Laya-GGUF, ggml-org/Clef-Flash-GGUF and ggml-org/OpenJev-GGUF.

Questions

questions maps a question id to a question. Each question has:

• :type - :choice, :score or :noul (strings work too).
• :instructions - the question. A string, or any JSON-encodable term, which the model is given as JSON text.
• :criteria - depends on :type:
  • :choice - the options, mapping each option key to its description (nil for none). The number of options is capped by the model: 52 for openjev, 255 for laya and clef.
  • :score - a list of 2 to 10 level descriptions, lowest first.
  • :noul - optional; %{"true" => ..., "false" => ...} descriptions.

The questions of a request are answered independently, except with clef, which decides them jointly.

Order matters to the model: the options of a choice are shown in the order given, and with clef and nimble so are the questions. A map is taken in Elixir's map order (sorted keys for up to 32 entries); pass a keyword list or a list of {key, value} tuples to fix the order yourself. Ids and option keys come back exactly as given, atoms included.

State

A string, or any JSON-encodable term (a map, a list), which the model is given as JSON text. A list of chat messages, or a map with a "messages" list, works the same way.

Answers

• :choice - %{type: :choice, choice: key, probabilities: %{key => p}, confidence: c}
• :score - %{type: :score, score: s, legend: %{0 => desc, ...}, probabilities: %{0 => p, ...}, confidence: c}, where score is the probability-weighted level index and can fall between two levels.
• :noul - %{type: :noul, noul: p}, the probability that the answer is true.

confidence runs from 0 (every option equally likely) to 1. Probabilities are scaled with the temperatures stored in the model file; they are not guaranteed to be calibrated for your data.

Example

{:ok, model} = LlamaCppEx.load_model("Laya-Q8_0.gguf", n_gpu_layers: -1)
{:ok, decision} = LlamaCppEx.Decision.new(model)

{:ok, %{answers: answers}} =
  LlamaCppEx.Decision.decide(
    decision,
    "Customer message: I was charged twice for my order last week.",
    route: [
      type: :choice,
      instructions: "Which team should handle this?",
      criteria: [billing: "payments and refunds", shipping: nil, technical: nil]
    ],
    angry: [type: :noul, instructions: "Is the customer angry?"],
    urgency: [
      type: :score,
      instructions: "How urgent is this?",
      criteria: ["can wait", "this week", "today", "right now"]
    ]
  )

answers.route.choice
#=> :billing (0.992 with Laya-Q8_0)

Limits

• Text only. Upstream also takes images for openjev and clef through libmtmd, which this build does not link; a request with images is rejected.
• laya and clef evaluate the whole prompt in one batch, so it has to fit in :n_batch (2048 tokens by default, see new/2).
• A %Decision{} drives one context and is not safe to use from two processes at once, like LlamaCppEx.Context. Give each process its own, or serialize calls through one process.