Baton.LLMClient behaviour (Baton v0.27.4)

The contract between Baton and whatever actually talks to a model provider.

Baton never speaks HTTP. A host wires its own client:

config :baton, llm_client: MyApp.LLM.Client

and Baton calls complete/2 for every live LLM step. Adopting this behaviour is optional for that path — the call is resolved at runtime, so a module with a matching complete/2 has always been enough and still is.

Batch mode

A use Baton.LLMStep, mode: :batch step uses the provider's Message Batches API instead: roughly half the token cost, hours-scale latency. That transport needs three more functions — submit, poll, collect — which are optional callbacks here because most clients never implement them.

The engine checks function_exported?(client, :submit_batch, 2) before its first submit and returns {:cancel, {:batch_unsupported, client}} if the client can't do batches, rather than failing the step repeatedly against a gap no retry can close.

submit_batch/2 takes a list of requests even though a batch step submits exactly one today. The provider APIs are list-shaped, and a future coalescer that packs many steps' requests into one provider batch reuses this callback unchanged.

The trailing keyword() on each callback carries the step's client opts (:model, :provider, …), so a multi-provider host routes batch calls the same way it routes live ones.

Polling, not webhooks

Baton polls (via Oban snoozes, which cost no retry attempts) rather than receiving completion callbacks. Polling is provider-agnostic and needs no public endpoint; a provider that offers webhooks can still be driven this way.

Summary

Types

batch_request()

One request within a batch. custom_id is how the result is matched back to the step that submitted it; Baton uses "job-<oban job id>", which is stable across attempts and inside Anthropic's 64-character limit.

batch_result()

One result within a finished batch. response is present when type is :succeeded; error when it is :errored. A :canceled or :expired request carries neither.

message()

A chat message, as Baton.LLMStep builds it.

response()

A normalized completion. usage follows Anthropic naming (cache_read_input_tokens / cache_creation_input_tokens); Baton.LLMStep renames those to the workflow_step_stats columns and passes any extra keys through to the pricing module.

Callbacks

batch_results(batch_id, keyword)

Every result in an ended batch, in any order. Baton finds its own by custom_id.

complete(list, keyword)

Send a completion request and return the normalized response.

poll_batch(batch_id, keyword)

Whether a batch has finished processing. {:ok, :pending} re-snoozes the step; {:ok, :ended} moves it to collecting results. "Ended" means the provider is done with the batch, not that every request in it succeeded.

submit_batch(list, keyword)

Submit a batch and return the provider's batch id. Required for mode: :batch.

Types

batch_request()

@type batch_request() :: %{
  custom_id: String.t(),
  messages: [message()],
  opts: keyword()
}

batch_result()

@type batch_result() :: %{
  custom_id: String.t(),
  type: :succeeded | :errored | :canceled | :expired,
  response: response() | nil,
  error: map() | nil
}

One result within a finished batch. response is present when type is :succeeded; error when it is :errored. A :canceled or :expired request carries neither.

message()

@type message() :: Baton.LLMStep.message()

A chat message, as Baton.LLMStep builds it.

response()

@type response() :: %{
  text: String.t(),
  model: String.t() | nil,
  stop_reason: String.t() | nil,
  usage: map()
}

Callbacks

batch_results(batch_id, keyword)

(optional)

@callback batch_results(
  batch_id :: String.t(),
  keyword()
) :: {:ok, [batch_result()]} | {:error, term()}

Every result in an ended batch, in any order. Baton finds its own by custom_id.

complete(list, keyword)

@callback complete(
  [message()],
  keyword()
) :: {:ok, response()} | {:error, term()}

Send a completion request and return the normalized response.

An HTTP failure should come back as {:error, %{status: status}} so Baton.LLMStep's error taxonomy can tell a rate limit from a bad request.

poll_batch(batch_id, keyword)

(optional)

@callback poll_batch(
  batch_id :: String.t(),
  keyword()
) :: {:ok, :pending} | {:ok, :ended} | {:error, term()}

submit_batch(list, keyword)

(optional)

@callback submit_batch(
  [batch_request()],
  keyword()
) :: {:ok, batch_id :: String.t()} | {:error, term()}

Submit a batch and return the provider's batch id. Required for mode: :batch.