A structured contract for LLM workflow steps: you describe the request and how to handle the decoded response — the engine owns everything in between.

Baton.LLMWorker gives LLM steps the right Oban posture (queue, backoff, idempotency, usage stats). But every step body still hand-writes the same transport loop: time the call, invoke the client, map max_tokens truncation, map 429 to a snooze, decode the (possibly fence-wrapped) JSON, and remember to merge llm_usage into the result. That loop is identical across steps and easy to get subtly wrong — a missed error clause either burns tokens retrying a non-retryable request or drops a retryable one.

Baton.LLMStep moves the loop into the library. A step implements:

• request/1 (required) — build the messages + client options from the job (read upstream deps, render prompts). Return {:ok, messages, opts} or {:ok, messages, opts, context} to thread a value through to handle_response/3. Return {:done, result} to short-circuit with a stored result and no model call (e.g. a fan-out step whose claim has no limitations to check) — no usage is attached, since nothing was spent.
• handle_response/3 (usually) — turn the decoded payload into the step's stored result. Defaults to {:ok, data}.
• decode/1 (optional) — defaults to decode_json/1, a tolerant JSON decoder (raw → fenced block → brace slice). Override with {:ok, text} passthrough for free-text steps.
• output_schema/0 (optional) — a JSON Schema map. When defined it is injected into the client options as :output_schema automatically, so the schema is declared once and both the API call and any introspection tooling read the same one.
• classify_error/1 (optional) — override the transport-error taxonomy.

What the engine does per call

1. request/1 → messages + opts (schema injected, context captured)
2. Gates on the configured Baton.RateLimiter: estimates the call's input tokens from the prompt and calls acquire/3. {:snooze, n} parks the job attempt-free (nothing is sent), tagged :rate_budget so the "snoozed" broadcast names the wait; {:ok, opts} proceeds with the possibly rewritten opts. After a successful response it calls reconcile/3 with the actual input and output tokens (paired with the input estimate and the call's requested :max_tokens as the output estimate). The default limiter is a no-op, so this is invisible until a host configures one.
3. Times the call and invokes Baton.Debug.call_llm/3 (so context-window capture keeps working unchanged)
4. stop_reason: "max_tokens" → {:error, :max_tokens_truncated} (retryable — an adaptive-thinking retry may fit; override classify_error/1 to discard instead)
5. Transport errors → classify_error/1, whose snooze verdicts the engine tags :provider_limit for the broadcast. Default taxonomy:
   • HTTP 429 (rate limit) and 529 (overloaded) → {:snooze, n} — no retry attempt consumed
   • other HTTP 4xx (except 408) → {:cancel, reason} — the request itself is invalid; retrying can never succeed
   • anything else → {:error, reason} — retried per max_attempts
6. decode/1 on response.text; a decode error is a job error (retryable)
7. handle_response/3 with the decoded data, your context, and the job
8. If the result is {:ok, map} and carries no llm_usage, the engine attaches one built from the client's normalized response.usage plus measured latency — so stats/cost recording needs nothing from the step. A rejected sample — truncation, a decode failure, a handle_response that returns an error to draw a fresh one — is recorded directly into stats at the point of rejection instead: the call was paid for whether or not the answer was kept, so every attempt's spend is visible in workflow_step_stats.

Example

defmodule MyApp.Steps.AssessQuality do
  use Baton.LLMStep

  @impl true
  def output_schema, do: %{"type" => "object", ...}

  @impl true
  def request(%Oban.Job{} = job) do
    {:ok, %{"parsed" => parsed}} = Baton.Results.get_result(job, :parse_claims)

    messages = [%{role: "user", content: prompt(parsed)}]
    {:ok, messages, [model: "claude-sonnet-4-20250514", system: @system]}
  end

  @impl true
  def handle_response(%{"score" => _} = quality, _ctx, _job) do
    {:ok, %{"quality" => quality}}
  end
end

request/1 may also return {:error, _}, {:snooze, _}, or {:cancel, _} directly (e.g. a dependency read failed) — they pass through untouched.

The client contract

Baton.Debug.call_llm/3 calls the configured :llm_client's complete/2, which must return {:ok, response} with text, model, stop_reason, and a usage map, or {:error, reason} (an HTTP failure as %{status: status} so the taxonomy can read it). Usage keys follow Anthropic naming (cache_read_input_tokens / cache_creation_input_tokens); the engine renames them to the workflow_step_stats columns and passes any extra keys (e.g. web_search_requests) through to the configured pricing module.

Batch mode

A step can trade latency for cost by switching transports:

use Baton.LLMStep, mode: :batch

It then goes through the provider's Message Batches API — around half the token price, hours instead of seconds — while every callback above stays exactly as written. The engine submits a one-request batch, parks the job on Oban snoozes until the batch ends (snoozes cost no retry attempts, so a step can wait a day without touching its budget), then feeds the result through the same decode → handle_response → attach-usage pipeline. Downstream steps, completion, retries, and stats can't tell the difference; the attached usage carries "service_tier" => "batch" so pricing can.

This requires a client implementing Baton.LLMClient's batch callbacks — a step whose client doesn't cancels with {:batch_unsupported, client} on its first attempt. Chaining batch steps stacks their latency, so prefer them for DAG tails or where the downstream work is cheap and local.

Options

All Baton.LLMWorker options are accepted and forwarded, plus:

• :rate_limit_snooze — seconds to snooze on a rate limit when the provider doesn't advertise its own Retry-After (default: 30; see default_classify/2)
• :mode — :live (default) or :batch
• :poll_interval — seconds between batch polls (default: 300)
• :batch_deadline — seconds before a batch is abandoned (default: 90_000, i.e. 25h — a backstop above the provider's own 24h expiry)