Behaviour for language model clients.
Inside an Imp.Run context, request/2 emits one :model_request and one
:model_response event per call. The request event carries the messages as
its input and the rest of the request in its metadata: :options, the
request options with the tool definitions removed, and :tools_hash, the
SHA-256 of the canonical JSON of those definitions, or nil when the request
sent no tools. The definitions themselves are emitted once per run per
distinct hash, as a :tools_sent event whose input is the tool list as
sent. Between the two, a recorded request can be reproduced without repeating
a roster on every call. Both are redacted like every other event. A request
made with generate/3's :purpose carries it in the request event's
metadata as :purpose; it is never part of what is sent. A request streamed
through the client's stream/3 (Imp.stream/3 with provider_stream: true)
is recorded the same way, with the usage the provider reported at the end of
the stream. A streamed call that fails after the provider reported usage or
cost records them on its failed :model_response and in Imp.Usage, since
the provider may have charged for it.
The response event's metadata carries the money for that call as two
numbers, each a non-negative float in USD or nil, as on
Imp.Core.LMResponse. :cost is what the provider reported charging, and
nil when it reported no charge, which means the charge is unknown rather
than zero. :estimated_cost is ReqLLM's catalog price for the reported
tokens, and nil when the catalog has no price for the model or the call
was streamed. A host summing
money spent sums :cost; one that falls back on :estimated_cost for calls
with no reported charge is counting an estimate, and should know it.
When ReqLLM priced the call, the breakdown behind :estimated_cost is also
on the event as :billing, unchanged; otherwise there is no :billing key.
A breakdown's shape is ReqLLM's, so treat it as evidence to inspect, not as
a contract.
Summary
Types
An LM: a struct whose module implements this behaviour, or such a module itself for a client that holds no configuration. Every callback receives the LM as its first argument, the struct or the module as it was given.
Callbacks
Sends messages and returns the model's output: a map of fields, a string,
an Imp.Prediction, or, when opts asks for several completions (:n), a
list of them.
Executes one provider-neutral request. A client that implements it gets
the request's configuration and metadata whole; one that does not is called
through generate/3.
Streams the output of one request, for Imp.stream/3.
Functions
Sends one request to lm and returns its output.
Executes one provider-neutral LM request and returns a normalized response.
Types
Callbacks
@callback generate(lm :: t(), messages :: [map()], opts :: keyword()) :: {:ok, map() | binary() | Imp.Prediction.t() | list()} | {:error, term()}
Sends messages and returns the model's output: a map of fields, a string,
an Imp.Prediction, or, when opts asks for several completions (:n), a
list of them.
@callback request(lm :: t(), request :: Imp.Core.LMRequest.t()) :: {:ok, Imp.Core.LMResponse.t()} | {:error, term()}
Executes one provider-neutral request. A client that implements it gets
the request's configuration and metadata whole; one that does not is called
through generate/3.
@callback stream(lm :: t(), messages :: [map()], opts :: keyword()) :: Enumerable.t()
Streams the output of one request, for Imp.stream/3.
Functions
Sends one request to lm and returns its output.
:purpose names what kind of call this is, for a caller that makes more than
one kind -- a program's own loop and a second model that writes its replies,
say. It is recorded on the :model_request event's metadata as :purpose
and is never sent to the provider: it is the record's, so a reader can tell
the calls apart without guessing from their options. A request without it
has no :purpose key.
Executes one provider-neutral LM request and returns a normalized response.