defmodule Omni do @moduledoc """ ![Omni](https://raw.githubusercontent.com/aaronrussell/omni/main/media/poster.webp) ![License](https://img.shields.io/github/license/aaronrussell/omni?color=informational) Universal Elixir client for LLM APIs. Streaming text generation, tool use, and structured output. Omni provides a single API for text generation that works with many LLM providers. All requests are streaming-first — `generate_text/3` is built on top of `stream_text/3`. ## Setup Add Omni to your dependencies: {:omni, "~> 1.3"} Each built-in provider reads its API key from a standard environment variable by default — if your keys are set, no configuration is needed: | Provider | Environment variable | | --- | --- | | Alibaba | `DASHSCOPE_API_KEY` | | Anthropic | `ANTHROPIC_API_KEY` | | Google | `GEMINI_API_KEY` | | Groq | `GROQ_API_KEY` | | Moonshot AI | `MOONSHOT_API_KEY` | | Ollama Cloud | `OLLAMA_API_KEY` | | OpenAI | `OPENAI_API_KEY` | | OpenCode | `OPENCODE_API_KEY` | | OpenRouter | `OPENROUTER_API_KEY` | | Z.ai | `ZAI_API_KEY` | To override a key — for example, to use a different variable name or load from a vault at runtime — use per-provider application config: # config/runtime.exs config :omni, Omni.Providers.Anthropic, api_key: {:system, "MY_ANTHROPIC_KEY"} config :omni, Omni.Providers.OpenAI, api_key: {MyApp.Secrets, :fetch, ["openai"]} You can also pass `:api_key` directly to `generate_text/3` or `stream_text/3`. Anthropic, OpenAI, and Google are loaded by default. To add others or limit what loads at startup: config :omni, :providers, [:anthropic, :openai, :openrouter] ## Text generation The simplest way to use Omni — pass a model tuple and a string: {:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, "Hello!") response.message #=> %Omni.Message{role: :assistant, content: [%Omni.Content.Text{text: "Hello! How can..."}]} response.usage #=> %Omni.Usage{input_tokens: 10, output_tokens: 25, total_cost: 0.0003, ...} For multi-turn conversations, build a context with a system prompt and messages: context = Omni.context( system: "You are a helpful assistant.", messages: [ Omni.message(role: :user, content: "What is Elixir?"), Omni.message(role: :assistant, content: "Elixir is a functional programming language..."), Omni.message(role: :user, content: "How does it handle concurrency?") ] ) {:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, context) ## Streaming `stream_text/3` returns a `StreamingResponse` that you can consume with event handlers: {:ok, stream} = Omni.stream_text({:anthropic, "claude-sonnet-4-5-20250514"}, "Tell me a story") {:ok, response} = stream |> StreamingResponse.on(:text_delta, fn %{delta: text} -> IO.write(text) end) |> StreamingResponse.complete() For simple cases where you just need the text chunks: stream |> StreamingResponse.text_stream() |> Enum.each(&IO.write/1) See `Omni.StreamingResponse` for the full event taxonomy. ## Structured output Pass an `Omni.Schema` map via the `:output` option to get validated, decoded output: schema = Omni.Schema.object(%{ name: Omni.Schema.string(description: "The capital city"), population: Omni.Schema.integer(description: "Approximate population") }, required: [:name, :population]) {:ok, response} = Omni.generate_text( {:anthropic, "claude-sonnet-4-5-20250514"}, "What is the capital of France?", output: schema ) response.output #=> %{name: "Paris", population: 2161000} The schema is sent to the provider for constrained decoding. The response text is validated against the schema and decoded into `response.output`, with automatic retries on validation failure. ## Tool use Define tools with schemas and handlers — the loop automatically executes tool uses and feeds results back to the model: weather_tool = Omni.tool( name: "get_weather", description: "Gets the current weather for a city", input_schema: Omni.Schema.object( %{city: Omni.Schema.string(description: "City name")}, required: [:city] ), handler: fn input -> "72°F and sunny in \#{input.city}" end ) context = Omni.context( messages: [Omni.message("What's the weather in London?")], tools: [weather_tool] ) {:ok, response} = Omni.generate_text({:anthropic, "claude-sonnet-4-5-20250514"}, context) When tools have handlers, the loop automatically executes tool uses and feeds results back to the model until it produces a final text response. For reusable tools with validation, define a tool module — see `Omni.Tool`. ## Providers Omni ships with built-in providers for all of the services listed in the table above. Each provider is paired with one or more dialects that handle wire format translation — most providers use a single dialect, while multi-model gateways like OpenCode Zen use different dialects per model. Models are referenced as `{provider_id, model_id}` tuples. To list available models for a provider: {:ok, models} = Omni.list_models(:anthropic) To add a custom provider, implement the `Omni.Provider` behaviour and load it at runtime — see `Omni.Provider` for details. """ alias Omni.{Context, Loop, Model, Response, StreamingResponse} @doc group: :generation @doc """ Streams a text generation request, returning a `%StreamingResponse{}`. The model can be a `%Model{}` struct or a `{provider_id, model_id}` tuple. The context can be a string, list of messages, or `%Context{}` struct. When tools with handlers are present in the context, automatically executes tool uses and loops until the model stops calling tools. Between rounds, synthetic `:tool_result` events are emitted for observability. ## Options * `:max_tokens` — maximum output tokens (Anthropic defaults to 4096) * `:temperature` — sampling temperature (number) * `:thinking` — enable extended thinking. Pass an effort level (`:low`, `:medium`, `:high`, `:xhigh`, `:max`), `%{budget: tokens}`, or `false` to disable * `:output` — a JSON Schema map for structured output (see the [Structured output](#module-structured-output) section above) * `:max_steps` — maximum tool execution rounds (default `:infinity`). Pass `1` to disable auto-looping for manual tool handling * `:tool_timeout` — per-tool execution timeout in milliseconds (default `30_000`) * `:cache` — prompt caching strategy (`:short` or `:long`) * `:timeout` — request timeout in milliseconds (default `300_000`) * `:api_key` — API key, overriding provider/app config * `:raw` — when `true`, attaches raw `{%Req.Request{}, %Req.Response{}}` tuples to the response (one per round) * `:metadata` — arbitrary metadata map passed to the provider * `:plug` — a Req test plug for stubbing HTTP responses in tests """ @spec stream_text(Model.ref() | Model.t(), term(), keyword()) :: {:ok, StreamingResponse.t()} | {:error, term()} def stream_text(model, context, opts \\ []) def stream_text({provider_id, model_id}, context, opts) do with {:ok, model} <- Model.get(provider_id, model_id) do stream_text(model, context, opts) end end def stream_text(%Model{} = model, context, opts) do Loop.stream(model, Context.new(context), opts) end @doc group: :generation @doc """ Generates text by consuming a streaming response to completion. Accepts the same arguments as `stream_text/3`. Equivalent to calling `stream_text/3` and then `StreamingResponse.complete/1`. Returns `{:ok, %Response{}}` with the assistant's message, token usage, stop reason, and structured output (when `:output` is set). """ @spec generate_text(Model.ref() | Model.t(), term(), keyword()) :: {:ok, Response.t()} | {:error, term()} def generate_text(model, context, opts \\ []) do with {:ok, stream} <- stream_text(model, context, opts) do StreamingResponse.complete(stream) end end # -- Delegates -- @doc group: :models @doc """ Looks up a model by provider and model ID. Returns `{:ok, model}` if found, `{:error, {:unknown_provider, id}}` if the provider isn't loaded, or `{:error, {:unknown_model, provider_id, model_id}}` if the model doesn't exist. {:ok, model} = Omni.get_model(:anthropic, "claude-sonnet-4-5-20250514") model.context_size #=> 200000 """ @spec get_model(atom(), String.t()) :: {:ok, Model.t()} | {:error, term()} defdelegate get_model(provider_id, model_id), to: Omni.Model, as: :get @doc group: :models @doc """ Registers a model for lookup by `get_model/2` and `list_models/1`. See `Omni.Model.put/2` for details. model = Omni.Model.new(...) Omni.put_model(:openai, model) """ @spec put_model(atom(), Model.t()) :: :ok defdelegate put_model(provider_id, model), to: Omni.Model, as: :put @doc group: :models @doc """ Lists all models for a provider. {:ok, models} = Omni.list_models(:anthropic) Enum.map(models, & &1.id) #=> ["claude-sonnet-4-5-20250514", "claude-haiku-4-5", ...] """ @spec list_models(atom()) :: {:ok, [Model.t()]} | {:error, term()} defdelegate list_models(provider_id), to: Omni.Model, as: :list @doc group: :context @doc """ Creates a `%Tool{}` struct from a keyword list or map. For inline tools with a handler function. For reusable tool modules with validation, see `Omni.Tool`. Omni.tool( name: "search", description: "Searches the web", input_schema: Omni.Schema.object(%{query: Omni.Schema.string()}, required: [:query]), handler: fn input -> do_search(input.query) end ) """ @spec tool(Enumerable.t()) :: Omni.Tool.t() defdelegate tool(attrs), to: Omni.Tool, as: :new @doc group: :context @doc """ Creates a `%Context{}` from a string, list of messages, keyword list, or map. A string is treated as a single user message. A list of `%Message{}` structs is treated as the message history. A keyword list or map can set `:system`, `:messages`, and `:tools`. Omni.context("Hello!") Omni.context(system: "You are helpful.", messages: [...], tools: [...]) """ @spec context(String.t() | [Omni.Message.t()] | Context.t() | Enumerable.t()) :: Context.t() defdelegate context(input), to: Omni.Context, as: :new @doc group: :context @doc """ Creates a `%Message{}` from a string, keyword list, or map. A string is treated as a user message. A keyword list or map can set `:role` and `:content`. Omni.message("What is Elixir?") Omni.message(role: :assistant, content: "Elixir is...") """ @spec message(String.t() | Enumerable.t()) :: Omni.Message.t() defdelegate message(input), to: Omni.Message, as: :new end