Behaviour for language model clients.
Inside an Imp.Run context, request/2 emits one :model_request and one
:model_response event per call. The request event carries the messages as
its input and the rest of the request in its metadata: :options, the
request options with the tool definitions removed, and :tools_hash, the
SHA-256 of the canonical JSON of those definitions, or nil when the request
sent no tools. The definitions themselves are emitted once per run per
distinct hash, as a :tools_sent event whose input is the tool list as
sent. Between the two, a recorded request can be reproduced without repeating
a roster on every call. Both are redacted like every other event. A request
made with generate/3's :purpose carries it in the request event's
metadata as :purpose; it is never part of what is sent.
The response event's metadata carries the
money for that call in :cost: the provider's reported total in USD as a
non-negative float, or nil when the provider reported nothing Imp can read
as a number. A host summing spend reads that number and nothing else.
Providers report the total as a bare number, a string, a Decimal or a cost
breakdown map, and Imp reads the number out of all four. When the provider
reported a breakdown, that map is also on the event as :billing, unchanged;
when it reported none, there is no :billing key. A breakdown's shape is the
provider's, so treat it as evidence to inspect, not as a contract.
Summary
Types
An LM: a struct whose module implements this behaviour, or such a module itself for a client that holds no configuration. Every callback receives the LM as its first argument, the struct or the module as it was given.
Callbacks
Sends messages and returns the model's output: a map of fields, a string,
an Imp.Prediction, or, when opts asks for several completions (:n), a
list of them.
Executes one provider-neutral request. A client that implements it gets
the request's configuration and metadata whole; one that does not is called
through generate/3.
Streams the output of one request, for Imp.stream/3.
Functions
Sends one request to lm and returns its output.
Executes one provider-neutral LM request and returns a normalized response.
Types
Callbacks
@callback generate(lm :: t(), messages :: [map()], opts :: keyword()) :: {:ok, map() | binary() | Imp.Prediction.t() | list()} | {:error, term()}
Sends messages and returns the model's output: a map of fields, a string,
an Imp.Prediction, or, when opts asks for several completions (:n), a
list of them.
@callback request(lm :: t(), request :: Imp.Core.LMRequest.t()) :: {:ok, Imp.Core.LMResponse.t()} | {:error, term()}
Executes one provider-neutral request. A client that implements it gets
the request's configuration and metadata whole; one that does not is called
through generate/3.
@callback stream(lm :: t(), messages :: [map()], opts :: keyword()) :: Enumerable.t()
Streams the output of one request, for Imp.stream/3.
Functions
Sends one request to lm and returns its output.
:purpose names what kind of call this is, for a caller that makes more than
one kind -- a program's own loop and a second model that writes its replies,
say. It is recorded on the :model_request event's metadata as :purpose
and is never sent to the provider: it is the record's, so a reader can tell
the calls apart without guessing from their options. A request without it
has no :purpose key.
Executes one provider-neutral LM request and returns a normalized response.