ALLM.Providers.FakeClassification (allm v0.6.1)

Copy Markdown View Source

Deterministic, scripted adapter for typed-classification testing. Implements ALLM.ClassificationAdapter.

Layer B — runtime. FakeClassification is the canonical testing classification adapter; it ships in lib/ (not test/support/) because users need it for their own application tests, like ALLM.Providers.Fake and ALLM.Providers.FakeModeration.

What FakeClassification is (and isn't)

FakeClassification never looks at :state. It reads :questions to answer each question id with an answer of the right type and shape, :model and :metadata to round-trip them onto the response, and nothing else. It does not re-validate the request: a question that ALLM.Validate.classification_request/1 would reject may still be answered, or may raise.

Its confidence convention is its own and is NOT any provider's formula. A default or scripted choice answer reports confidence: 1.0; a scripted fractional score reports the larger of its two level probabilities. A real adapter reports whatever its provider returns and never computes confidence.

Default answers (no script)

With no script — adapter_opts[:classification_script] absent or [] — every question gets a deterministic default:

TypeDefault
:choicethe lexicographically first option, probability 1.0 on it and 0.0 on every other option, confidence: 1.0
:scorescore: 0.0, probabilities [1.0, 0.0, …] (one per level), legend = the question's levels, confidence: 1.0
:yes_noyes_probability: 0.0, confidence: nil
iex> q = ALLM.ClassificationQuestion.choice("Which team?", ["technical", "billing"])
iex> req = ALLM.ClassificationRequest.new(state: "My card was charged twice.", questions: %{"team" => q})
iex> {:ok, resp} = ALLM.Providers.FakeClassification.classify(req, [])
iex> ALLM.ClassificationResponse.answer(resp, "team").choice
"billing"

Spent script

A non-empty script whose cursor has run off the end does NOT fall back to the defaults — it returns

{:error, %ALLM.Error.ClassificationAdapterError{
   reason: :unknown,
   metadata: %{cause: :classification_script_exhausted}}}

"I didn't script anything, give me a default" is a convenience; "my script ran out" is almost always an off-by-one in the caller's expectation of how many times classify/2 gets invoked, and answering it with defaults would make that bug pass green.

Script shapes

opts[:adapter_opts][:classification_script] accepts a list of entries. See script/1 for the grammar.

adapter_opts: [
  classification_script: [
    {:answers, %{"team" => "billing", "frustration" => 1.25, "refund" => 0.9}},
    {:error, %ALLM.Error.ClassificationAdapterError{reason: :rate_limited}},
    {:retry_until_call, 3}
  ]
]

{:answers, map} overrides the answer for each listed question id; every id not listed gets its default. A value is read according to the question's type:

Value:choice:score (n levels):yes_no
a stringthe chosen option; must be one of the optionsraisesraises
a numberraisesthe position, between 0 and n - 1the probability of yes, between 0 and 1
%ALLM.ClassificationAnswer{}used verbatimused verbatimused verbatim

A fractional score x puts probability ceil(x) - x on level floor(x) and x - floor(x) on level ceil(x); a whole-number score puts 1.0 on that level. Every "raises" cell, an out-of-range number, and an id that is not one of the request's questions raise ArgumentError naming the question id: those are test-author mistakes and should not pass green.

{:retry_until_call, n} returns a synthetic %ALLM.Error.ClassificationAdapterError{reason: :rate_limited, retry_after_ms: 0} for the first n - 1 calls against this entry, then advances to the next entry on call n. Consecutive {:retry_until_call, _} entries chain into a layered budget.

Cursor behaviour

Multi-call scripts advance a per-process cursor on every call. The cursor lives in the process dictionary at {:allm_fake_classification_cursor, key_id}, isolated per ExUnit test process (async: true), GC'd on pid-down, zero-setup for the common case. The key_id is chosen by this precedence:

  1. adapter_opts[:script_cursor] — an explicit Agent pid (handled separately; see start_script_cursor/0).
  2. adapter_opts[:cursor_key] — the engine's stable :id, injected by the façade dispatch chokepoint via ALLM.Engine.put_cursor_key/2.
  3. :erlang.phash2(script) — the content-hash fallback for direct adapter calls with no engine.

The {:retry_until_call, n} budget keys on the same identity, so two engines built with content-equal scripts each get their own cursor and their own retry budget. The content-hash footgun remains only for DIRECT adapter calls — ALLM.Providers.FakeClassification.classify(req, opts) invoked without an engine receives no :cursor_key. Workaround for that path: pass distinct adapter_opts[:script_cursor] Agent pids from start_script_cursor/0, or distinct :cursor_key values.

Test-only capture seam

Pass adapter_opts[:capture_pid] with a pid to receive a side-channel message every time classify/2 is invoked, BEFORE the empty-questions gate runs and before the script is consulted:

{ALLM.Providers.FakeClassification, :call, %{request: request, opts: opts}}

It does not affect the response.

Summary

Types

One scripted classification call.

Functions

Execute a scripted classification request.

Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many script entries have been consumed.

Document and validate the script grammar for adapter_opts[:classification_script].

Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.

Types

script_entry()

@type script_entry() ::
  {:answers,
   %{
     required(String.t()) =>
       String.t() | number() | ALLM.ClassificationAnswer.t()
   }}
  | {:error, ALLM.Error.ClassificationAdapterError.t()}
  | {:retry_until_call, pos_integer()}

One scripted classification call.

Functions

classify(request, opts)

Execute a scripted classification request.

Order, all before the script is consulted:

  1. adapter_opts[:capture_pid] side-channel (fires even for rejected calls).
  2. questions: %{} → {:error, %ClassificationAdapterError{reason: :invalid_request}}.

Otherwise reads the script from opts[:adapter_opts][:classification_script], advances the process-local cursor, and interprets the entry. An absent script produces the default answers; a spent non-empty script returns :classification_script_exhausted.

The response carries model: request.model || "fake-classification", provider: :fake, zero token usage, request_id: opts[:request_id], metadata: request.metadata, and id: nil.

Examples

iex> q = ALLM.ClassificationQuestion.yes_no("Is a refund requested?")
iex> req = ALLM.ClassificationRequest.new(state: "Please refund me.", questions: %{"refund" => q})
iex> opts = [adapter_opts: [classification_script: [{:answers, %{"refund" => 0.9}}]]]
iex> {:ok, resp} = ALLM.Providers.FakeClassification.classify(req, opts)
iex> ALLM.ClassificationResponse.answer(resp, "refund").yes_probability
0.9

iex> req = ALLM.ClassificationRequest.new(state: "x", questions: %{})
iex> {:error, err} = ALLM.Providers.FakeClassification.classify(req, [])
iex> err.reason
:invalid_request

cursor_index(pid)

@spec cursor_index(pid()) :: non_neg_integer()

Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many script entries have been consumed.

Examples

iex> pid = ALLM.Providers.FakeClassification.start_script_cursor()
iex> q = ALLM.ClassificationQuestion.yes_no("Spam?")
iex> req = ALLM.ClassificationRequest.new(state: "x", questions: %{"spam" => q})
iex> opts = [adapter_opts: [classification_script: [{:answers, %{}}], script_cursor: pid]]
iex> {:ok, _} = ALLM.Providers.FakeClassification.classify(req, opts)
iex> ALLM.Providers.FakeClassification.cursor_index(pid)
1

script(entries)

@spec script([script_entry()]) :: :ok

Document and validate the script grammar for adapter_opts[:classification_script].

Each entry is one of:

  • {:answers, %{id => value}} — per-id overrides; see the module docs for how each value is read.
  • {:error, %ALLM.Error.ClassificationAdapterError{}} — return the struct verbatim.
  • {:retry_until_call, n} — synthetic :rate_limited for the first n - 1 calls against this entry.

Returns :ok when the script is well-formed and raises ArgumentError on the first invalid entry. Validation is opt-in: classify/2 does not call it. It checks entry shapes only; whether an {:answers, map} value fits its question is checked when classify/2 reads it.

Examples

iex> ALLM.Providers.FakeClassification.script([{:answers, %{"team" => "billing"}}, {:retry_until_call, 2}])
:ok

start_script_cursor()

@spec start_script_cursor() :: pid()

Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.

Pass the returned pid as adapter_opts[:script_cursor].

Examples

iex> pid = ALLM.Providers.FakeClassification.start_script_cursor()
iex> ALLM.Providers.FakeClassification.cursor_index(pid)
0