Typed decisions with a decision model: llama.cpp's /v1/systemone API, in-process.

A decision model answers typed questions about a state in one forward pass per prompt; no token is generated. Each answer is a probability distribution, so it comes with a confidence instead of free text. This is llama-server's TypeSafe-compatible System One API, ported into the NIF: same request shape, same prompts, same answers.

Models

The model file says what kind of decision model it is ("<arch>.decision.type"); model_type/1 reads it. Supported types:

• :openjev, :lev, :nimble, :pplx_decider, :vllm_d3, :lfm2_d1, :granite_switch, :h2o_lightning - read the logits of one label token per option (:nimble lists every question of the request in each prompt, :lev asks each choice twice with the options reversed, :pplx_decider, :vllm_d3 and :granite_switch use one- or two-letter label codes like :lev, :lfm2_d1 picks label codes per question and accepts a nil state). :vllm_d3 is non-causal, so each prompt is evaluated in one batch; it accepts a nil state and a question without :instructions (its prompt has a default question). :granite_switch is IBM's Granite Switch, whose decision adapter is built into the model; the same model also generates text through the rest of the API. :h2o_lightning reads the labels its model file lists, after an answer prefill: :instructions and :type may be left out (:choice by default), a :score takes 1 to 255 levels, :noul descriptions are also read under "yes" and "no", and a state longer than 32000 tokens keeps its first and last 16000.
• :kev - scores each option by a dot product of hidden states.
• :laya - a ModernBERT encoder; one score per [MASK] marker.
• :lfm2_d1_omni - like :laya with another prompt layout: a non-causal LFM2 trunk with a decision head, one score per <|mask|> marker. It has no memory, so every prompt is evaluated in one batch. Accepts a nil state, and also reads :noul descriptions under "yes" and "no".
• :clef - reads every question in one prompt and decides them jointly.
• :vllm_decision2 - clef's joint head, one question per prompt: a Qwen3 trunk scores each option span against the question span. Strict input: the state, instructions and descriptions must be text, a map or a list, a choice needs at least 2 options, and :noul criteria may only describe "true" and "false".

Decision GGUFs are published in ggml-org's "Decision models" collection on Hugging Face, for example ggml-org/Laya-GGUF, ggml-org/Clef-Flash-GGUF and ggml-org/OpenJev-GGUF.

Questions

questions maps a question id to a question. Each question has:

• :type - :choice, :score or :noul (strings work too).
• :instructions - the question. A string, or any JSON-encodable term, which the model is given as JSON text.
• :criteria - depends on :type:
  • :choice - the options, mapping each option key to its description (nil for none). The number of options is capped by the model: 52 for openjev, 255 for laya, clef, pplx-decider, vllm-decision2, vllm-d3, lfm2-d1, lfm2-d1-omni, granite-switch and h2o-lightning.
  • :score - a list of 2 to 10 level descriptions, lowest first (1 to 255 for h2o-lightning).
  • :noul - optional; %{"true" => ..., "false" => ...} descriptions.

The questions of a request are answered independently, except with clef, which decides them jointly.

Order matters to the model: the options of a choice are shown in the order given, and with clef and nimble so are the questions. A map is taken in Elixir's map order (sorted keys for up to 32 entries); pass a keyword list or a list of {key, value} tuples to fix the order yourself. Ids and option keys come back exactly as given, atoms included.

State

A string, or any JSON-encodable term (a map, a list), which the model is given as JSON text. A list of chat messages, or a map with a "messages" list, works the same way.

Answers

• :choice - %{type: :choice, choice: key, probabilities: %{key => p}, confidence: c}
• :score - %{type: :score, score: s, legend: %{0 => desc, ...}, probabilities: %{0 => p, ...}, confidence: c}, where score is the probability-weighted level index and can fall between two levels.
• :noul - %{type: :noul, noul: p}, the probability that the answer is true.

confidence runs from 0 (every option equally likely) to 1; vllm-decision2 and vllm-d3 compute it from the normalized entropy of the probabilities. Probabilities are scaled with the temperatures (and, for scores, the per-level biases) stored in the model file; they are not guaranteed to be calibrated for your data. vllm-decision2 and vllm-d3 return a legend entry that is not a string as its JSON text, with sorted keys.

Example

{:ok, model} = LlamaCppEx.load_model("Laya-Q8_0.gguf", n_gpu_layers: -1)
{:ok, decision} = LlamaCppEx.Decision.new(model)

{:ok, %{answers: answers}} =
  LlamaCppEx.Decision.decide(
    decision,
    "Customer message: I was charged twice for my order last week.",
    route: [
      type: :choice,
      instructions: "Which team should handle this?",
      criteria: [billing: "payments and refunds", shipping: nil, technical: nil]
    ],
    angry: [type: :noul, instructions: "Is the customer angry?"],
    urgency: [
      type: :score,
      instructions: "How urgent is this?",
      criteria: ["can wait", "this week", "today", "right now"]
    ]
  )

answers.route.choice
#=> :billing (0.992 with Laya-Q8_0)

Limits

• Text only. Upstream also takes images (openjev, clef, pplx-decider, vllm-d3, lfm2-d1, lfm2-d1-omni) and audio (lfm2-d1-omni) through libmtmd, which this build does not link; a request with images or audio is rejected.
• laya, clef, vllm-d3 and lfm2-d1-omni evaluate the whole prompt in one batch, so it has to fit in :n_batch (2048 tokens by default, see new/2); vllm-decision2 needs the question, its options and the end of the prompt to fit in one.
• A %Decision{} drives one context and is not safe to use from two processes at once, like LlamaCppEx.Context. Give each process its own, or serialize calls through one process.