View Source Agens.Serving (agens v0.1.2)

The Serving module provides functions for starting, stopping and running Servings.

Agens.Serving accepts a GenServer module or Nx.Serving struct for processing messages.

Agens.Serving is decoupled from Agens.Agent in order to reuse a single LM across multiple agents. In most cases, however, you will only need to start one text generation serving to be used by most, if not all, agents.

In some cases, you may have additional servings for more specific use cases such as image generation, speech recognition, etc.

Servings were built with the Bumblebee library in mind, as well as Nx.Serving. GenServer is supported for working with LM APIs instead, which may be more cost effective and easier to get started with.

Summary

Functions

Retrieves the Serving configuration by Serving name or pid.

Executes an Agens.Message against an Agens.Serving

Starts an Agens.Serving process

Stops an Agens.Serving process

Functions

@spec get_config(atom() | pid()) ::
  {:ok, Agens.Serving.Config.t()} | {:error, :serving_not_found}

Retrieves the Serving configuration by Serving name or pid.

@spec run(Agens.Message.t()) :: String.t() | {:error, :serving_not_found}

Executes an Agens.Message against an Agens.Serving

@spec start(Agens.Serving.Config.t()) :: {:ok, pid()} | {:error, term()}

Starts an Agens.Serving process

@spec stop(atom()) :: :ok | {:error, :serving_not_found}

Stops an Agens.Serving process