View Source Agens.Serving (agens v0.1.1)

The Serving module provides functions for starting, stopping and running Servings.

Agens.Serving accepts a GenServer module or Nx.Serving struct for processing messages.

Agens.Serving is decoupled from Agens.Agent in order to reuse a single LM across multiple agents. In most cases, however, you will only need to start one text generation serving to be used by most, if not all, agents.

In some cases, you may have additional servings for more specific use cases such as image generation, speech recognition, etc.

Servings were built with the Bumblebee library in mind, as well as Nx.Serving. GenServer is supported for working with LM APIs instead, which may be more cost effective and easier to get started with.

Summary

Functions

@spec run(Agens.Message.t()) :: String.t() | {:error, :serving_not_running}

Executes an Agens.Message against an Agens.Serving

@spec start(Agens.Serving.Config.t()) :: {:ok, pid()}

Starts an Agens.Serving process

@spec stop(atom()) :: :ok | {:error, :serving_not_found}

Stops an Agens.Serving process