ALLM.Providers.FakeModeration (allm v0.6.1)

Copy Markdown View Source

Deterministic, scripted adapter for content-moderation testing. Implements ALLM.ModerationAdapter.

Layer B — runtime. FakeModeration is the canonical testing moderation adapter; it ships in lib/ (not test/support/) because users need it for their own application tests, mirroring the ALLM.Providers.Fake, ALLM.Providers.FakeImages, and ALLM.Providers.FakeEmbeddings precedent.

What FakeModeration is (and isn't)

FakeModeration mostly ignores the %ALLM.ModerationRequest{} passed to moderate/2. It inspects :input to enforce the empty-input and batch-size gates the ALLM.ModerationAdapter contract requires of every implementation, and to size the default verdict; it reads :model and :metadata to round-trip them onto the response. Otherwise the scripted response is produced irrespective of the request — in particular no scripted verdict is derived from the input text, and a {:ok, results} entry decides how many results come back, not length(input).

Default verdict (no script)

With no script — adapter_opts[:moderation_script] absent or [] — every input yields one unflagged ALLM.ModerationResult carrying all 13 omni-moderation category names, every value false and every score 0.0. Cardinality follows ALLM.ModerationRequest's rule: length(input) results for an all-strings input, exactly one result for a multimodal one.

This is a deliberate divergence from ALLM.Providers.FakeEmbeddings, whose no-script call surfaces an error: a clean verdict is a meaningful default that costs a caller nothing, whereas a synthesized embedding vector is not.

Spent script

A non-empty script whose cursor has run off the end is a different case and does NOT fall back to the clean verdict — it returns

{:error, %ALLM.Error.ModerationAdapterError{
   reason: :unknown,
   metadata: %{cause: :moderation_script_exhausted}}}

"I didn't script anything, give me a benign default" is a convenience; "my script ran out" is almost always an off-by-one in the caller's expectation of how many times moderate/2 gets invoked, and a test vehicle that answers it with an unflagged pass hides that bug. In particular a truncated [{:retry_until_call, 1}] script would otherwise report success on call 1 with no retry ever exercised.

Script shapes

opts[:adapter_opts][:moderation_script] accepts a list of script entries. See script/1 for the full grammar.

adapter_opts: [
  moderation_script: [
    {:ok, [%ALLM.ModerationResult{...}]},
    {:flagged, ["violence"]},
    {:error, %ALLM.Error.ModerationAdapterError{reason: :rate_limited}},
    {:retry_until_call, 3}
  ]
]

{:flagged, categories} is the shorthand for the overwhelmingly common test — "assert my app rejects flagged content" — and synthesizes a single flagged result with those category names true at score 1.0 and every other omni category false at 0.0.

{:retry_until_call, n} returns a synthetic %ALLM.Error.ModerationAdapterError{reason: :rate_limited, retry_after_ms: 0} for the first n - 1 calls against this entry, then advances the cursor to the next entry on call n. Vehicle for testing ALLM.Retry.run/3 integration. Consecutive {:retry_until_call, _} entries chain: the call that exhausts one entry's budget lands on the next entry and opens its budget, which is how a layered retry budget is scripted.

Cursor behaviour

Multi-call scripts (:moderation_script) advance a per-process cursor on every call. The cursor lives in the process dictionary at {:allm_fake_moderation_cursor, key_id}, isolated per ExUnit test process (async: true), GC'd on pid-down, zero-setup for the common case. The key_id is chosen by this precedence:

  1. adapter_opts[:script_cursor] — an explicit Agent pid (handled separately; see start_script_cursor/0).
  2. adapter_opts[:cursor_key] — the engine's stable :id, injected by the façade dispatch chokepoint via ALLM.Engine.put_cursor_key/2.
  3. :erlang.phash2(script) — the content-hash fallback for direct adapter calls with no engine.

At the façade the cursor keys on engine identity, so two engines built with content-equal :moderation_script values each read index 0 on their first call, even in the same process. The content-hash footgun remains only for DIRECT adapter calls — ALLM.Providers.FakeModeration.moderate(req, opts) invoked without an engine receives no :cursor_key. Workaround for that path: pass distinct adapter_opts[:script_cursor] Agent pids from start_script_cursor/0.

Test-only capture seam

Pass adapter_opts[:capture_pid] with a pid to receive a side-channel message every time moderate/2 is invoked, BEFORE any gate runs and before the script is consulted. The message has the form:

{ALLM.Providers.FakeModeration, :call, %{request: request, opts: opts}}

This is purely a side-channel — it does NOT affect the response. It exists so test files can assert on what the adapter received without the Process.register/2 + named-pid pattern (which forces async: false).

Examples

iex> req = ALLM.ModerationRequest.new(input: ["is this ok?"])
iex> opts = [adapter_opts: [moderation_script: [{:flagged, ["violence"]}]]]
iex> {:ok, resp} = ALLM.Providers.FakeModeration.moderate(req, opts)
iex> ALLM.ModerationResponse.flagged_categories(resp)
["violence"]

Summary

Types

One scripted moderation result.

Functions

The 13 omni-moderation category names FakeModeration reports, sorted.

Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many calls have been consumed.

Return the maximum number of inputs FakeModeration accepts per call.

Execute a scripted moderation request.

Document and validate the script grammar for adapter_opts[:moderation_script].

Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.

Types

script_entry()

@type script_entry() ::
  {:ok, [ALLM.ModerationResult.t()]}
  | {:flagged, [String.t()]}
  | {:error, ALLM.Error.ModerationAdapterError.t()}
  | {:retry_until_call, pos_integer()}

One scripted moderation result.

Functions

categories()

@spec categories() :: [String.t()]

The 13 omni-moderation category names FakeModeration reports, sorted.

Exposed so tests can assert against the same vocabulary the adapter synthesizes rather than hand-copying a 13-element list.

Examples

iex> ALLM.Providers.FakeModeration.categories |> length()
13

iex> "self-harm/intent" in ALLM.Providers.FakeModeration.categories
true

cursor_index(pid)

@spec cursor_index(pid()) :: non_neg_integer()

Read the current cursor index for an Agent-backed cursor. Used in tests to assert how many calls have been consumed.

Examples

iex> pid = ALLM.Providers.FakeModeration.start_script_cursor
iex> req = ALLM.ModerationRequest.new(input: ["x"])
iex> opts = [adapter_opts: [moderation_script: [{:flagged, ["hate"]}], script_cursor: pid]]
iex> {:ok, _} = ALLM.Providers.FakeModeration.moderate(req, opts)
iex> ALLM.Providers.FakeModeration.cursor_index(pid)
1

max_batch_size()

@spec max_batch_size() :: pos_integer()

Return the maximum number of inputs FakeModeration accepts per call.

Deliberately not a provider-shaped number. The real moderation cap is undocumented, and a mirrored constant would force every conformance run to build a list of that size just to cross the :batch_too_large boundary. 32 is large enough to be plausible and small enough to be cheap.

Examples

iex> ALLM.Providers.FakeModeration.max_batch_size
32

moderate(request, opts)

Execute a scripted moderation request.

Gate order, all before the script is consulted:

  1. adapter_opts[:capture_pid] side-channel (fires even for rejected calls).
  2. input: [] → {:error, %ModerationAdapterError{reason: :invalid_request}}.
  3. item count > max_batch_size() → {:error, %ModerationAdapterError{reason: :batch_too_large, metadata: %{count: n, max: max}}}. Per contract invariant 5 the count is the item count invariant 3 defines — length(input) for an all-strings input, 1 for a multimodal one — not the raw list length.

Otherwise reads the script from opts[:adapter_opts][:moderation_script], advances the process-local cursor, and interprets the entry. An absent script produces the default clean verdict; a spent non-empty script returns :moderation_script_exhausted. Both are described in the module docs.

Propagates opts[:request_id] onto response.request_id, request.model onto response.model, request.metadata onto response.metadata, and stamps provider: :fake.

Examples

iex> req = ALLM.ModerationRequest.new(input: ["hello"], metadata: %{trace: "t1"})
iex> {:ok, resp} = ALLM.Providers.FakeModeration.moderate(req, request_id: "rid-1")
iex> {resp.request_id, resp.metadata, ALLM.ModerationResponse.flagged?(resp)}
{"rid-1", %{trace: "t1"}, false}

iex> req = ALLM.ModerationRequest.new(input: [])
iex> {:error, err} = ALLM.Providers.FakeModeration.moderate(req, [])
iex> err.reason
:invalid_request

script(entries)

@spec script([script_entry()]) :: :ok

Document and validate the script grammar for adapter_opts[:moderation_script].

Each entry is one of:

  • {:ok, [%ALLM.ModerationResult{}, ...]} — return the listed results verbatim.
  • {:flagged, ["violence", ...]} — synthesize ONE flagged result with those category names true at score 1.0, every other omni category false at 0.0.
  • {:error, %ALLM.Error.ModerationAdapterError{}} — return the struct verbatim.
  • {:retry_until_call, n} — synthetic :rate_limited for the first n - 1 calls against this entry. Consecutive entries of this shape chain into a layered budget (the call that exhausts one entry opens the next entry's budget).

Returns :ok when the script is well-formed; raises ArgumentError on the first invalid entry. (Validation is opt-in — the runtime moderate/2 path tolerates a mix of legal entries and surfaces metadata: %{cause: :moderation_script_exhausted} when the cursor runs off the end of a non-empty script.)

Examples

iex> ALLM.Providers.FakeModeration.script([{:flagged, ["violence"]}])
:ok

start_script_cursor()

@spec start_script_cursor() :: pid()

Start an Agent-backed script cursor for cross-process multi-call scripting and for disambiguating content-equal scripts in the same process.

Pass the returned pid as adapter_opts[:script_cursor]; subsequent calls increment the cursor on the Agent rather than on the process dictionary.

Examples

iex> pid = ALLM.Providers.FakeModeration.start_script_cursor
iex> ALLM.Providers.FakeModeration.cursor_index(pid)
0