erllama_middleware (erllama v0.10.1)

Middleware around the erllama API calls, in the style of hackney's middleware: a middleware is a plain fun that receives the request and a Next fun, and returns the response. It can observe, rewrite, short-circuit or wrap a call. No behaviour, no registry, no dependencies.

-type request() :: #{op := atom(), model := erllama:model() | undefined, args := map()}.
-type next() :: fun((request()) -> term()).
-type middleware() :: fun((request(), next()) -> term()).

Chain order is outermost first: [A, B] means A wraps B, the request flows A -> B -> erllama and the response unwinds erllama -> B -> A.

Install a global chain for every call:

application:set_env(erllama, middleware, [Log, Metrics]).

or a per-call chain, which replaces the global one for that call:

erllama:complete(Model, Prompt, #{middleware => [Log]}).

Wrapped operations and their args:

op	args
`load_model`	`#{config := map()}` (`model` is the id)
`unload`	`#{}`
`complete`	`#{prompt := binary(), opts := map()}`
`prefill_only`	`#{tokens := [token_id()], opts := map()}`
`stream`	`#{prompt := binary() \\| [token_id()], opts := map()}`
`continue`	`#{tokens := [token_id()], opts := map()}`
`chat`	`#{messages := [map()], opts := map()}`
`chat_apply`	`#{messages := [map()], opts := map()}`
`chat_parse`	`#{params := term(), input := binary(), partial := boolean()}`
`embed`, `embed_batch`	`#{input := term()}`
`tokenize`	`#{text := binary(), opts := map()}`
`detokenize`	`#{tokens := [token_id()]}`

Streaming ops (stream, continue) return {ok, Ref} to the middleware; the events are delivered later to the to process. To observe completion, set to to a proxy process of your own.

A middleware that raises propagates to the caller; erllama does not catch. Recipes (logging, Prometheus, telemetry, caching) are in the middleware guide.

Summary

Types

middleware()

next()

request()

Functions

global()

The global chain (application:get_env(erllama, middleware, [])).

run(Request, Next)

run/3 with the global chain from the middleware application environment key.

run/3

Run Request through Chain and then Next. An empty chain calls Next directly.

take(Opts)

Split the middleware key out of an option map: returns the chain to use (the per-call list when present, else the global chain) and the options without the key.