Deterministic, scripted adapter for typed-classification testing.
Implements ALLM.ClassificationAdapter.
Layer B — runtime. FakeClassification is the canonical testing
classification adapter; it ships in lib/ (not test/support/) because
users need it for their own application tests, like ALLM.Providers.Fake
and ALLM.Providers.FakeModeration.
What FakeClassification is (and isn't)
FakeClassification never looks at :state. It reads :questions to answer
each question id with an answer of the right type and shape, :model and
:metadata to round-trip them onto the response, and nothing else. It
does not re-validate the request: a question that
ALLM.Validate.classification_request/1 would reject may still be
answered, or may raise.
Its confidence convention is its own and is NOT any provider's formula.
A default or scripted choice answer reports confidence: 1.0; a scripted
fractional score reports the larger of its two level probabilities. A real
adapter reports whatever its provider returns and never computes
confidence.
Default answers (no script)
With no script — adapter_opts[:classification_script] absent or
[] — every question gets a deterministic default:
| Type | Default |
|---|---|
:choice | the lexicographically first option, probability 1.0 on it and 0.0 on every other option, confidence: 1.0 |
:score | score: 0.0, probabilities [1.0, 0.0, …] (one per level), legend = the question's levels, confidence: 1.0 |
:yes_no | yes_probability: 0.0, confidence: nil |
iex> q = ALLM.ClassificationQuestion.choice("Which team?", ["technical", "billing"])
iex> req = ALLM.ClassificationRequest.new(state: "My card was charged twice.", questions: %{"team" => q})
iex> {:ok, resp} = ALLM.Providers.FakeClassification.classify(req, [])
iex> ALLM.ClassificationResponse.answer(resp, "team").choice
"billing"Spent script
A non-empty script whose cursor has run off the end does NOT fall back to the defaults — it returns
{:error, %ALLM.Error.ClassificationAdapterError{
reason: :unknown,
metadata: %{cause: :classification_script_exhausted}}}"I didn't script anything, give me a default" is a convenience; "my script
ran out" is almost always an off-by-one in the caller's expectation of how
many times classify/2 gets invoked, and answering it with defaults would
make that bug pass green.
Script shapes
opts[:adapter_opts][:classification_script] accepts a list of entries.
See script/1 for the grammar.
adapter_opts: [
classification_script: [
{:answers, %{"team" => "billing", "frustration" => 1.25, "refund" => 0.9}},
{:error, %ALLM.Error.ClassificationAdapterError{reason: :rate_limited}},
{:retry_until_call, 3}
]
]{:answers, map} overrides the answer for each listed question id; every
id not listed gets its default. A value is read according to the
question's type:
| Value | :choice | :score (n levels) | :yes_no |
|---|---|---|---|
| a string | the chosen option; must be one of the options | raises | raises |
| a number | raises | the position, between 0 and n - 1 | the probability of yes, between 0 and 1 |
%ALLM.ClassificationAnswer{} | used verbatim | used verbatim | used verbatim |
A fractional score x puts probability ceil(x) - x on level floor(x)
and x - floor(x) on level ceil(x); a whole-number score puts 1.0 on
that level. Every "raises" cell, an out-of-range number, and an id that is
not one of the request's questions raise ArgumentError naming the
question id: those are test-author mistakes and should not pass green.
{:retry_until_call, n} returns a synthetic
%ALLM.Error.ClassificationAdapterError{reason: :rate_limited, retry_after_ms: 0}
for the first n - 1 calls against this entry, then advances to the next
entry on call n. Consecutive {:retry_until_call, _} entries chain into a
layered budget.
Cursor behaviour
Multi-call scripts advance a per-process cursor on every call. The cursor
lives in the process dictionary at
{:allm_fake_classification_cursor, key_id}, isolated per ExUnit test
process (async: true), GC'd on pid-down, zero-setup for the common case.
The key_id is chosen by this precedence:
adapter_opts[:script_cursor]— an explicit Agent pid (handled separately; seestart_script_cursor/0).adapter_opts[:cursor_key]— the engine's stable:id, injected by the façade dispatch chokepoint viaALLM.Engine.put_cursor_key/2.:erlang.phash2(script)— the content-hash fallback for direct adapter calls with no engine.
The {:retry_until_call, n} budget keys on the same identity, so two
engines built with content-equal scripts each get their own cursor and
their own retry budget. The content-hash footgun remains only for DIRECT
adapter calls — ALLM.Providers.FakeClassification.classify(req, opts)
invoked without an engine receives no :cursor_key. Workaround for that
path: pass distinct adapter_opts[:script_cursor] Agent pids from
start_script_cursor/0, or distinct :cursor_key values.
Test-only capture seam
Pass adapter_opts[:capture_pid] with a pid to receive a side-channel
message every time classify/2 is invoked, BEFORE the empty-questions
gate runs and before the script is consulted:
{ALLM.Providers.FakeClassification, :call, %{request: request, opts: opts}}It does not affect the response.
Summary
Types
One scripted classification call.
Functions
Execute a scripted classification request.
Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many script entries have been consumed.
Document and validate the script grammar for
adapter_opts[:classification_script].
Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.
Types
@type script_entry() :: {:answers, %{ required(String.t()) => String.t() | number() | ALLM.ClassificationAnswer.t() }} | {:error, ALLM.Error.ClassificationAdapterError.t()} | {:retry_until_call, pos_integer()}
One scripted classification call.
Functions
@spec classify( ALLM.ClassificationRequest.t(), keyword() ) :: {:ok, ALLM.ClassificationResponse.t()} | {:error, ALLM.Error.ClassificationAdapterError.t()}
Execute a scripted classification request.
Order, all before the script is consulted:
adapter_opts[:capture_pid]side-channel (fires even for rejected calls).questions: %{}→{:error, %ClassificationAdapterError{reason: :invalid_request}}.
Otherwise reads the script from opts[:adapter_opts][:classification_script],
advances the process-local cursor, and interprets the entry. An absent
script produces the default answers; a spent non-empty script returns
:classification_script_exhausted.
The response carries model: request.model || "fake-classification",
provider: :fake, zero token usage, request_id: opts[:request_id],
metadata: request.metadata, and id: nil.
Examples
iex> q = ALLM.ClassificationQuestion.yes_no("Is a refund requested?")
iex> req = ALLM.ClassificationRequest.new(state: "Please refund me.", questions: %{"refund" => q})
iex> opts = [adapter_opts: [classification_script: [{:answers, %{"refund" => 0.9}}]]]
iex> {:ok, resp} = ALLM.Providers.FakeClassification.classify(req, opts)
iex> ALLM.ClassificationResponse.answer(resp, "refund").yes_probability
0.9
iex> req = ALLM.ClassificationRequest.new(state: "x", questions: %{})
iex> {:error, err} = ALLM.Providers.FakeClassification.classify(req, [])
iex> err.reason
:invalid_request
@spec cursor_index(pid()) :: non_neg_integer()
Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many script entries have been consumed.
Examples
iex> pid = ALLM.Providers.FakeClassification.start_script_cursor()
iex> q = ALLM.ClassificationQuestion.yes_no("Spam?")
iex> req = ALLM.ClassificationRequest.new(state: "x", questions: %{"spam" => q})
iex> opts = [adapter_opts: [classification_script: [{:answers, %{}}], script_cursor: pid]]
iex> {:ok, _} = ALLM.Providers.FakeClassification.classify(req, opts)
iex> ALLM.Providers.FakeClassification.cursor_index(pid)
1
@spec script([script_entry()]) :: :ok
Document and validate the script grammar for
adapter_opts[:classification_script].
Each entry is one of:
{:answers, %{id => value}}— per-id overrides; see the module docs for how each value is read.{:error, %ALLM.Error.ClassificationAdapterError{}}— return the struct verbatim.{:retry_until_call, n}— synthetic:rate_limitedfor the firstn - 1calls against this entry.
Returns :ok when the script is well-formed and raises ArgumentError on
the first invalid entry. Validation is opt-in: classify/2 does not call
it. It checks entry shapes only; whether an {:answers, map} value fits its
question is checked when classify/2 reads it.
Examples
iex> ALLM.Providers.FakeClassification.script([{:answers, %{"team" => "billing"}}, {:retry_until_call, 2}])
:ok
@spec start_script_cursor() :: pid()
Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.
Pass the returned pid as adapter_opts[:script_cursor].
Examples
iex> pid = ALLM.Providers.FakeClassification.start_script_cursor()
iex> ALLM.Providers.FakeClassification.cursor_index(pid)
0