View Source Omni (Omni v0.1.1)
Omni focusses on one thing only - being a chat interface to any LLM provider. If you want a full featured client for a specific provider, supporting all available API endpoints, this is probably not it. If you want a single client to generate chat completions with literally any LLM backend, Omni is for you.
- π§©
Omni.Provider
behaviour to create integrations with any LLM provider. Built-in providers for: - π Streaming API requests
- Stream to an Enumerable
- Or stream messages to any Elixir process
- π« Simple to use and easily customisable
Installation
The package can be installed by adding omni
to your list of dependencies
in mix.exs
.
def deps do
[
{:omni, "0.1.1"}
]
end
Quickstart
To chat with an LLM, initialize a t:provider/0
with
init/2
, and then send a t:request/0
, using
one of generate/2
, async/2
or stream/2
. Refer to the schema
documentation for each provider to ensure you construct a valid request.
iex> provider = Omni.init(:openai)
iex> Omni.generate(provider, model: "gpt-4o", messages: [
...> %{role: "user", content: "Write a haiku about the Greek Gods"}
...> ])
{:ok, %{"object" => "chat.completion", "choices" => [...]}}
Streaming
Omni supports streaming request through async/2
or stream/2
.
Calling async/2
returns a Task.t/0
, which asynchronously sends text
delta messages to the calling process. Using the :stream_to
request option
allows you to control the receiving process.
The example below demonstrates making a streaming request in a LiveView event, and sends each of the streaming messages back to the same LiveView process.
defmodule MyApp.ChatLive do
use Phoenix.LiveView
# When the client invokes the "prompt" event, create a streaming request and
# asynchronously send messages back to self.
def handle_event("prompt", %{"message" => prompt}, socket) do
{:ok, task} = Omni.async(Omni.init(:openai), [
model: "gpt-4o",
messages: [
%{role: "user", content: "Write a haiku about the Greek Gods"}
]
])
{:noreply, assign(socket, current_request: task)}
end
# The streaming request sends messages back to the LiveView process.
def handle_info({_request_pid, {:data, _data}} = message, socket) do
pid = socket.assigns.current_request.pid
case message do
{:omni, ^pid, {:chunk, %{"choices" => choices, "finish_reason" => nil}}} ->
# handle each streaming chunk
{:omni, ^pid, {:chunk, %{"choices" => choices}}} ->
# handle the final streaming chunk
end
end
# Tidy up when the request is finished
def handle_info({ref, {:ok, _response}}, socket) do
Process.demonitor(ref, [:flush])
{:noreply, assign(socket, current_request: nil)}
end
end
Alternatively, use stream/2
to collect the streaming responses into an
Enumerable.t/0
that can be used with Elixir's Stream
functions.
iex> provider = Omni.init(:openai)
iex> {:ok, stream} = Omni.stream(provider, model: "gpt-4o", messages: [
...> %{role: "user", content: "Write a haiku about the Greek Gods"}
...> ])
iex> stream
...> |> Stream.each(&IO.inspect/1)
...> |> Stream.run()
Because this function builds the Enumerable.t/0
by calling receive/1
,
take care using stream/2
inside GenServer
callbacks as it may cause the
GenServer to misbehave.
Summary
Functions
Asynchronously generates a chat completion using the given t:provider/0
and t:request/0
. Returns a Task.t/0
.
As async/2
but raises in the case of an error.
Generates a chat completion using the given t:provider/0
and t:request/0
. Synchronously returns a
t:response/0
.
As generate/2
but raises in the case of an error.
Asynchronously generates a chat completion using the given t:provider/0
and t:request/0
. Returns an Enumerable.t/0
.
As stream/2
but raises in the case of an error.
Functions
@spec async(Omni.Provider.t(), Omni.Provider.request()) :: {:ok, Task.t()} | {:error, term()}
Asynchronously generates a chat completion using the given t:provider/0
and t:request/0
. Returns a Task.t/0
.
Within your code, you should manually define a receive/1
block (or setup
GenServer.handle_info/2
callbacks) to receive the message stream.
Additional request options
In addition to the t:request/0
options for
the given t:provider/0
, this function accepts the
following options:
:stream-to
- Pass apid/0
to control the receiving process.
Example
iex> provider = Omni.init(:openai)
iex> Omni.async(provider, model: "gpt-4o", messages: [
%{role: "user", content: "Write a haiku about the Greek Gods"}
])
{:ok, %Task{pid: pid, ref: ref}}
# Somewhere in your code
receive do
{:omni, ^pid, {:chunk, chunk}} -> # handle chunk
{^ref, {:ok, res}} -> # handle final response
{^ref, {:error, error}} -> # handle error
{:DOWN, _ref, _, _pid, _reason} -> # handle DOWN signal
end
@spec async!(Omni.Provider.t(), Omni.Provider.request()) :: Task.t()
As async/2
but raises in the case of an error.
@spec generate(Omni.Provider.t(), Omni.Provider.request()) :: {:ok, Omni.Provider.response()} | {:error, term()}
Generates a chat completion using the given t:provider/0
and t:request/0
. Synchronously returns a
t:response/0
.
Example
iex> provider = Omni.init(:openai)
iex> Omni.generate(provider, model: "gpt-4o", messages: [
%{role: "user", content: "Write a haiku about the Greek Gods"}
])
{:ok, %{"message" => %{
"content" => "Mount Olympus stands,\nImmortal whispers echoβ\nZeus reigns, thunder roars."
}}}
@spec generate!(Omni.Provider.t(), Omni.Provider.request()) :: Omni.Provider.response()
As generate/2
but raises in the case of an error.
See Omni.Provider.init/2
.
@spec stream(Omni.Provider.t(), Omni.Provider.request()) :: {:ok, Enumerable.t()} | {:error, term()}
Asynchronously generates a chat completion using the given t:provider/0
and t:request/0
. Returns an Enumerable.t/0
.
Because this function builds the Enumerable.t/0
by calling receive/1
,
using this function inside GenServer
callbacks may cause the GenServer to
misbehave. In such cases, use async/2
instead.
Example
iex> provider = Omni.init(:openai)
iex> {:ok, stream} = Omni.stream(provider, model: "gpt-4o", messages: [
%{role: "user", content: "Write a haiku about the Greek Gods"}
])
iex> stream
...> |> Stream.each(&IO.inspect/1)
...> |> Stream.run()
@spec stream!(Omni.Provider.t(), Omni.Provider.request()) :: Enum.t()
As stream/2
but raises in the case of an error.