View Source Omni (Omni v0.1.0)

Omni

License

Omni focusses on one thing only - being a chat interface to any LLM provider. If you want a full featured client for a specific provider, supporting all available API endpoints, this is probably not it. If you want a single client to generate chat completions with literally any LLM backend, Omni is for you.

  • 🧩 Omni.Provider behaviour to create integrations with any LLM provider. Built-in providers for:
    • Anthropic - chat with any of of the Claude models.
    • Google - chat with any of of the Gemini models.
    • Ollama - use Ollama to chat with any local model.
    • OpenAI - chat with ChatGPT or any other OpenAI compatible API.
  • πŸ›œ Streaming API requests
    • Stream to an Enumerable
    • Or stream messages to any Elixir process
  • πŸ’« Simple to use and easily customisable

Installation

The package can be installed by adding omni to your list of dependencies in mix.exs.

def deps do
  [
    {:omni, "0.1.0"}
  ]
end

Quickstart

To chat with an LLM, initialize a t:provider/0 with init/2, and then send a t:request/0, using one of generate/2, async/2 or stream/2. Refer to the schema documentation for each provider to ensure you construct a valid request.

iex> provider = Omni.init(:openai)
iex> Omni.generate(provider, model: "gpt-4o", messages: [
...>   %{role: "user", content: "Write a haiku about the Greek Gods"}
...> ])
{:ok, %{"object" => "chat.completion", "choices" => [...]}}

Streaming

Omni supports streaming request through async/2 or stream/2.

Calling async/2 returns a Task.t/0, which asynchronously sends text delta messages to the calling process. Using the :stream_to request option allows you to control the receiving process.

The example below demonstrates making a streaming request in a LiveView event, and sends each of the streaming messages back to the same LiveView process.

defmodule MyApp.ChatLive do
  use Phoenix.LiveView

  # When the client invokes the "prompt" event, create a streaming request and
  # asynchronously send messages back to self.
  def handle_event("prompt", %{"message" => prompt}, socket) do
    {:ok, task} = Omni.async(Omni.init(:openai), [
      model: "gpt-4o",
      messages: [
        %{role: "user", content: "Write a haiku about the Greek Gods"}
      ]
    ])

    {:noreply, assign(socket, current_request: task)}
  end

  # The streaming request sends messages back to the LiveView process.
  def handle_info({_request_pid, {:data, _data}} = message, socket) do
    pid = socket.assigns.current_request.pid
    case message do
      {:omni, ^pid, {:chunk, %{"choices" => choices, "finish_reason" => nil}}} ->
        # handle each streaming chunk

      {:omni, ^pid, {:chunk, %{"choices" => choices}}} ->
        # handle the final streaming chunk
    end
  end

  # Tidy up when the request is finished
  def handle_info({ref, {:ok, _response}}, socket) do
    Process.demonitor(ref, [:flush])
    {:noreply, assign(socket, current_request: nil)}
  end
end

Alternatively, use stream/2 to collect the streaming responses into an Enumerable.t/0 that can be used with Elixir's Stream functions.

iex> provider = Omni.init(:openai)
iex> {:ok, stream} = Omni.stream(provider, model: "gpt-4o", messages: [
...>   %{role: "user", content: "Write a haiku about the Greek Gods"}
...> ])

iex> stream
...> |> Stream.each(&IO.inspect/1)
...> |> Stream.run()

Because this function builds the Enumerable.t/0 by calling receive/1, take care using stream/2 inside GenServer callbacks as it may cause the GenServer to misbehave.

Summary

Functions

Asynchronously generates a chat completion using the given t:provider/0 and t:request/0. Returns a Task.t/0.

As async/2 but raises in the case of an error.

Generates a chat completion using the given t:provider/0 and t:request/0. Synchronously returns a t:response/0.

As generate/2 but raises in the case of an error.

Asynchronously generates a chat completion using the given t:provider/0 and t:request/0. Returns an Enumerable.t/0.

As stream/2 but raises in the case of an error.

Functions

@spec async(Omni.Provider.t(), Omni.Provider.request()) ::
  {:ok, Task.t()} | {:error, term()}

Asynchronously generates a chat completion using the given t:provider/0 and t:request/0. Returns a Task.t/0.

Within your code, you should manually define a receive/1 block (or setup GenServer.handle_info/2 callbacks) to receive the message stream.

Additional request options

In addition to the t:request/0 options for the given t:provider/0, this function accepts the following options:

  • :stream-to - Pass a pid/0 to control the receiving process.

Example

iex> provider = Omni.init(:openai)
iex> Omni.async(provider, model: "gpt-4o", messages: [
  %{role: "user", content: "Write a haiku about the Greek Gods"}
])
{:ok, %Task{pid: pid, ref: ref}}

# Somewhere in your code
receive do
  {:omni, ^pid, {:chunk, chunk}} ->  # handle chunk
  {^ref, {:ok, res}} ->              # handle final response
  {^ref, {:error, error}} ->         # handle error
  {:DOWN, _ref, _, _pid, _reason} -> # handle DOWN signal
end
@spec async!(Omni.Provider.t(), Omni.Provider.request()) :: Task.t()

As async/2 but raises in the case of an error.

Link to this function

generate(provider, opts)

View Source
@spec generate(Omni.Provider.t(), Omni.Provider.request()) ::
  {:ok, Omni.Provider.response()} | {:error, term()}

Generates a chat completion using the given t:provider/0 and t:request/0. Synchronously returns a t:response/0.

Example

iex> provider = Omni.init(:openai)
iex> Omni.generate(provider, model: "gpt-4o", messages: [
  %{role: "user", content: "Write a haiku about the Greek Gods"}
])
{:ok, %{"message" => %{
  "content" => "Mount Olympus stands,\nImmortal whispers echoβ€”\nZeus reigns, thunder roars."
}}}
Link to this function

generate!(provider, opts)

View Source

As generate/2 but raises in the case of an error.

Link to this function

init(provider, opts \\ [])

View Source

See Omni.Provider.init/2.

@spec stream(Omni.Provider.t(), Omni.Provider.request()) ::
  {:ok, Enumerable.t()} | {:error, term()}

Asynchronously generates a chat completion using the given t:provider/0 and t:request/0. Returns an Enumerable.t/0.

Because this function builds the Enumerable.t/0 by calling receive/1, using this function inside GenServer callbacks may cause the GenServer to misbehave. In such cases, use async/2 instead.

Example

iex> provider = Omni.init(:openai)
iex> {:ok, stream} = Omni.stream(provider, model: "gpt-4o", messages: [
  %{role: "user", content: "Write a haiku about the Greek Gods"}
])

iex> stream
...> |> Stream.each(&IO.inspect/1)
...> |> Stream.run()
@spec stream!(Omni.Provider.t(), Omni.Provider.request()) :: Enum.t()

As stream/2 but raises in the case of an error.