ALLM.Pipeline.Schema.JsonSchema (allm_pipeline v0.1.0)

Copy Markdown View Source

Derives an OpenAI strict-mode JSON schema from an ALLM.Pipeline.Schema declaration.

This module exists so ALLM.Pipeline.Schema does not grow a second responsibility: the macro owns the struct, the type and the introspection clauses; this owns the wire contract.

When it runs

At the using module's compile time, from ALLM.Pipeline.Schema.__before_compile__/1, and only when that module declares use ALLM.Pipeline.Schema, json_schema: true. The derived map is a compile-time constant spliced into __allm_schema__(:json_schema) — there is no runtime work and no API call needed to inspect it, which is what lets the project rule "unit-test the NORMALIZED schema, never the raw map" apply to a generated schema at all.

A field whose type cannot be mapped is a compile error, not a silent {} — the failure surfaces at build time rather than as a production 400.

The type mapping

Declared type (from :generated_types)JSON type
String.t()"string"
String.t() with values:"string" + enum
[String.t()] with values:"array"; the enum lands on items
integer(), non_neg_integer(), pos_integer(), neg_integer()"integer"
float()"number"
boolean()"boolean"
atom() with values:"string" + enum (the parse path coerces back)
atom() without values:compile error — an open atom is not a closed schema
Date.t()"string"
[T]"array", items derived from T
a module exporting __allm_schema__/1"object" — recurses
map(), term(), bare list()compile error

values: is accepted only on a String.t() or atom() field, or on the element type of a list of them — those are the two rows above that emit an enum. Declaring it on any other mapped type (integer(), float(), boolean(), Date.t(), a nested schema) is a compile error: JSON-schema enum on a non-string property would be silently meaningless to the parse path, which keys atom coercion off the declared type. field/3's own values: validation checks the list's shape (non-empty, homogeneous, no nil) and cannot see the field's type; this is where the type half is enforced.

Nullability

A | nil tail on the generated type — the narrow nilability rule's output, not the declared AST — makes the emitted type a union ("string" becomes ["string", "null"]). Every property still appears in required: strict mode expresses optionality by the null union alone, so the DSL's required: and the JSON schema's required are deliberately different sets.

When a nullable field also carries values:, nil is appended to the emitted enum — a "null"-permitting type whose enum has no null member is rejected by the strict validator. nil is derived, never declared: a literal nil inside a hand-written values: list is a compile error, since is_atom(nil) is true and a bare list-of-atoms check would wave it through.

Field inclusion is OPT-OUT, and redact: true must say which it means

Every declared field reaches the derived schema — and therefore the model — unless it declares wire: false. That is the opposite of the hand-written schemas this replaces, which were implicit allowlists: a field was on the wire because somebody typed it there. The opt-out default is deliberate (a derivation whose default is "exclude" derives nothing, and every omission would be silent), but it moves the failure mode from "a field is missing" to "a field is present that nobody decided to send".

One flag makes that disagreement likely enough to be worth catching: redact: true declares a value too sensitive to appear in a step log, and says nothing about whether it may be sent to a third-party model. The two are independent — a redacted field can legitimately be model-produced — so neither implication is safe to assume. A redact: true field in a module that derives a JSON schema must therefore declare wire: explicitly, one way or the other, or it is a compile error. wire: false keeps it off the wire; wire: true (or a rename) says the author looked and meant it.

This costs a json_schema: true module one word per redacted field and buys the guarantee that no secret-bearing property reaches OpenAI by omission. Modules that do not derive a schema are untouched.

Key casing

The derived map is string-keyed at every level, matching the host normalizer's output. required is built as Map.keys/1 of the property map for the same reason: it is exactly what a strict-mode normalizer would compute, so a derived schema is a fixed point of one — with exactly one exception, immediately below.

The one exception: a json_schema: literal

That fixed-point property holds for every mapped property. It does not extend to a per-field json_schema: literal, which is spliced in verbatim by design (an author reaching for the hatch has a shape the mapping cannot express, so second-guessing it would defeat it). A literal declaring its own "properties" without "additionalProperties" => false and a full "required" — or one using atom keys — makes the enclosing derived map a non-fixed-point of a strict-mode normalizer. Inside a literal, strict-mode compliance is the author's responsibility, and the compiler will not check it.

The consequence is a divergence, not a production defect: the host's LLMEngine.normalize_schema/1 runs unconditionally inside generate_structured/4, so the model still receives a repaired, compliant schema. What is lost is this module's headline guarantee — that a malformed wire contract is a compile error rather than something a downstream normalizer quietly fixes. Both halves are pinned: the package test asserts an object-shaped literal is emitted untouched, and the host-side derived_schema_normalization_test.exs asserts the normalizer is what repairs it.

Nested schemas

A field typed by a module that itself uses this DSL recurses into that module's own :json_schema. The nested module must therefore also declare json_schema: true; it is a named compile error when it does not.

Resolution expands the alias against the using module's environment. Module.concat/1 on the raw AST parts is not sufficient: the stored type AST is unexpanded, and nested schemas are declared inside their parent and referenced by short alias (field(:key_provisions, [KeyProvision.t()])), so naive resolution yields Elixir.KeyProvision — a module that does not exist, whose absence would turn a perfectly good nested schema into an unmappable type.

Code.ensure_compiled/1not ensure_loaded?/1 — is called before function_exported?/3. The latter answers false for a merely-unloaded module, and ensure_loaded?/1 additionally answers false for a module being compiled concurrently in the same mix compile batch, because it does not participate in Kernel.ParallelCompiler's module handshake. That is exactly the cross-file case — <step>/input.ex beside <step>/output.ex, this tree's own convention — and it produced a bogus "no strict-mode JSON rendering" error pointing at a nested module that was perfectly correct.

Summary

Functions

Derives the strict-mode JSON schema for module from its field list.

Functions

derive!(module, fields, env)

@spec derive!(module(), [ALLM.Pipeline.Schema.field_spec()], Macro.Env.t()) :: map()

Derives the strict-mode JSON schema for module from its field list.

fields carries the generated type AST (__allm_schema__(:generated_types)), not the declared one — the | nil tail is the nullable-union decision.

Raises ArgumentError naming the module and field when a type cannot be mapped, or when two fields resolve to the same wire property name.