Imp.Clients.ReqLLM (Imp v0.12.1)

Copy Markdown View Source

Imp LM client backed by the Elixir req_llm ecosystem.

Imp owns signatures, adapters, optimizers, traces, and evaluation. req_llm owns provider/model resolution, Req/Finch transport, streaming, provider option translation, and canonical response structs.

For OpenRouter, an unspecified output-token limit stays absent on the wire, leaving the endpoint to choose it. Explicit output limits are passed through ReqLLM unchanged; Imp does not reserve the catalog's full output maximum.

:input_envelope is an Imp-owned safety option. It accepts a positive :max_bytes guard and an optional positive :reservation_tokens value. Imp measures the rendered message content before cache lookup or transport, raises Imp.OperationalSafetyError when the byte guard is exceeded, and removes the envelope before calling ReqLLM. The token value records a pricing or capacity reservation; without a model tokenizer it is not treated as an exact token counter.

A failed request returns {:error, %Imp.LMError{}}, whether the provider answered with an error status, the connection failed, or ReqLLM raised; the struct carries the status, whether a retry may succeed, and whether the input was longer than the model's context window, with ReqLLM's own error under :reason. Only an HTTP 400 with the structured code error.code = "context_length_exceeded" counts as the context window; prose and generic HTTP 400 responses do not trigger context recovery. A successful HTTP response whose body carries a provider error, which is how OpenRouter relays an upstream refusal, is returned as that error, never as an empty completion. So is a completion that finished with reason :content_filter or :error (ReqLLM's reason for any finish reason it does not recognise): a non-retryable Imp.LMError whose message ends with the response's text, with content_filtered: true for the filter. The provider billed such a response, so through Imp.LM its usage and cost are still recorded, on the call's failed :model_response event and in Imp.Usage.

stream/3 ends with exactly one terminal event. A provider stream that runs to its end closes with done: true and the metadata the provider reported along the way: usage (with any "cost" the provider charged), model and finish reason. One that raises, carries a provider error, or finishes with reason :error or :cancelled closes with {:error, %Imp.LMError{}} instead, never a completion, and that event carries the metadata that arrived before it stopped. A consumer that stops early receives no terminal event, and the provider stream is cancelled.

:reasoning_effort is the one reasoning option, on the client or on a call. It takes any value of ReqLLM's own reasoning_effort option, such as high, xhigh, max or default, as an atom or a string, on every provider. ReqLLM's provider translates it where it has a translation (Anthropic and Google turn it into a thinking budget); OpenAI, OpenRouter, Groq and xAI receive the effort as written, and it is the provider's to accept. A call naming nil spends no reasoning on that call whatever the client is configured with. Native reasoning fields (Imp.Predict) set the same option, so a client configured with an effort and a program that asks for one never disagree.

OpenRouter accepts the effort in two wire fields, and its endpoint catalog says which one an endpoint supports: ReqLLM's top-level reasoning_effort (the default here) or the nested reasoning object, which openrouter_reasoning_wire: :nested selects. The switch names an encoding only; the value is always :reasoning_effort.

Summary

Types

t()

@type t() :: %Imp.Clients.ReqLLM{
  model: ReqLLM.model_input(),
  opts: keyword(),
  req_module: module(),
  tool_calling: {ReqLLM.model_input(), boolean()} | nil
}

Functions

cache_key(lm, messages, opts)

dump(lm)

generate_async(lm, messages, opts \\ [])

new(model_spec, opts \\ [])