# lemon_ai A provider-agnostic LLM client for Elixir. Call **27 model providers** — Anthropic, OpenAI, Google, AWS Bedrock, Azure, Groq, Mistral, xAI, DeepSeek, OpenRouter, and more — through **one streaming API**, with a built-in model registry, per-provider circuit breaking and rate limiting, and cost accounting. No web framework, no umbrella, no runtime dependencies beyond [`req`](https://hex.pm/packages/req), [`jason`](https://hex.pm/packages/jason), and [`nimble_options`](https://hex.pm/packages/nimble_options). The whole surface is `LemonAi.stream/3` and `LemonAi.complete/3` plus a handful of helpers. Point a `%LemonAi.Types.Model{}` at any provider and the rest of your code stays identical. ## Install `lemon_ai` is not yet on Hex. Add it as a git dependency for now: ```elixir def deps do [ {:lemon_ai, github: "z80dev/lemon", sparse: "apps/lemon_ai"} ] end ``` Once published, this becomes: ```elixir {:lemon_ai, "~> 0.1"} ``` The OTP application is named `:lemon_ai`, so you call it as `LemonAi.*`. It starts its own supervision tree (circuit breakers, rate limiters, the model cache) automatically. ## Quickstart Set the API key for whichever provider you want (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GOOGLE_GENERATIVE_AI_API_KEY`, …) and make a blocking call: ```elixir model = LemonAi.Models.get_model(:anthropic, "claude-haiku-4-5") context = LemonAi.new_context(system_prompt: "You are a helpful assistant.") |> LemonAi.Types.Context.add_user_message("Explain OTP supervision in two sentences.") {:ok, message} = LemonAi.complete(model, context) IO.puts(LemonAi.get_text(message)) ``` Switch providers by changing one line — the context, options, and result handling are identical: ```elixir model = LemonAi.Models.get_model(:openai, "gpt-4o") model = LemonAi.Models.get_model(:google, "gemini-2.5-pro") model = LemonAi.Models.get_model(:groq, "llama-3.3-70b-versatile") ``` The API key resolves in this order: `opts.api_key` you pass in → provider-specific env var (e.g. `ANTHROPIC_API_KEY`) → generic fallback. So you can pass keys explicitly or rely on the environment. ## Streaming `LemonAi.stream/3` returns an `LemonAi.EventStream` you consume as a lazy stream of events: ```elixir {:ok, stream} = LemonAi.stream(model, context, %{temperature: 0.7, reasoning: :medium}) stream |> LemonAi.EventStream.events() |> Enum.each(fn {:text_delta, _idx, delta, _partial} -> IO.write(delta) {:thinking_delta, _idx, delta, _partial} -> IO.write([IO.ANSI.faint(), delta, IO.ANSI.reset()]) {:done, _reason, _message} -> IO.puts("\n-- done --") {:error, _reason, message} -> IO.puts("Error: #{message.error_message}") _ -> :ok end) ``` Convenience helpers on the stream: ```elixir text = LemonAi.EventStream.collect_text(stream) # blocking, returns full text {:ok, message} = LemonAi.EventStream.result(stream) # blocking, returns final message %{queue_size: _, dropped: _} = LemonAi.EventStream.stats(stream) LemonAi.EventStream.cancel(stream, :user_requested) ``` `LemonAi.complete/3` is just `stream/3` + `EventStream.result/1` collected for you. ## Tool use ```elixir tools = [ %LemonAi.Types.Tool{ name: "get_weather", description: "Get weather for a location", parameters: %{ type: "object", properties: %{location: %{type: "string"}}, required: ["location"] } } ] context = LemonAi.new_context(system_prompt: "You can check weather.", tools: tools) |> LemonAi.Types.Context.add_user_message("Weather in Paris?") {:ok, message} = LemonAi.complete(model, context) for tc <- LemonAi.get_tool_calls(message) do result = %LemonAi.Types.ToolResultMessage{ tool_call_id: tc.id, tool_name: tc.name, content: [%LemonAi.Types.TextContent{text: "Sunny, 22C"}], is_error: false } context = context |> LemonAi.Types.Context.add_assistant_message(message) |> LemonAi.Types.Context.add_tool_result(result) {:ok, final} = LemonAi.complete(model, context) IO.puts(LemonAi.get_text(final)) end ``` ## Provider capability matrix Every provider is reached through one of **10 wire-protocol modules**. The 27 provider catalogs in `LemonAi.Models` route through these — for example Groq, xAI, DeepSeek, Qwen, Cerebras, OpenRouter, Vercel AI Gateway, HuggingFace, Fireworks, and Mistral's catalog all speak the OpenAI Chat Completions format, so they inherit its capabilities. | Wire module (`api_id`) | Streaming | Tool calls | Vision (image input) | Reasoning / thinking | Cost data | |------------------------|:---------:|:----------:|:--------------------:|:--------------------:|:---------:| | Anthropic `:anthropic_messages` | ✅ | ✅ | ✅ | ✅ | ✅ | | OpenAI Chat Completions `:openai_completions` | ✅ | ✅ | ✅ | ✅ | ✅ | | OpenAI Responses `:openai_responses` | ✅ | ✅ | ✅ | ✅ | ✅ | | OpenAI Codex `:openai_codex_responses` | ✅ | ✅ | ✅ | ✅ | ✅ | | Azure OpenAI `:azure_openai_responses` | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Generative AI `:google_generative_ai` | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Vertex `:google_vertex` | ✅ | ✅ | ✅ | ✅ | ✅ | | Google Gemini CLI `:google_gemini_cli` | ✅ | ✅ | ✅ | ✅ | ✅ | | AWS Bedrock `:bedrock_converse_stream` | ✅ | ✅ | ✅ | ✅ | ✅ | | Mistral Conversations `:mistral_conversations` | ✅ | — | — | — | ✅ | Vision and reasoning are additionally gated **per model** by the model's `input` and `reasoning` fields — a wire module supporting vision doesn't make a text-only model accept images. Query the specific model: ```elixir LemonAi.Models.supports_vision?(model) LemonAi.Models.supports_reasoning?(model) LemonAi.Models.supports_xhigh(model) ``` The 27 provider catalogs: `:anthropic`, `:openai`, `:"openai-codex"`, `:amazon_bedrock`, `:google`, `:google_antigravity`, `:kimi`, `:kimi_coding`, `:opencode`, `:opencode_go`, `:xai`, `:mistral`, `:cerebras`, `:deepseek`, `:qwen`, `:minimax`, `:zai`, `:azure_openai_responses`, `:github_copilot`, `:google_gemini_cli`, `:google_vertex`, `:groq`, `:huggingface`, `:minimax_cn`, `:fireworks`, `:openrouter`, `:vercel_ai_gateway`. ## What you get beyond raw HTTP ### Circuit breaking Each provider gets its own circuit breaker (closed → open → half-open), lazily started the first time you call it. After a run of failures the breaker opens and fails fast instead of hammering a down provider. ```elixir config :lemon_ai, :circuit_breaker, failure_threshold: 5, # failures before opening (default: 5) recovery_timeout: 30_000 # ms before half-open recovery (default: 30_000) LemonAi.CircuitBreaker.open?(:anthropic) LemonAi.CircuitBreaker.reset(:anthropic) LemonAi.CircuitBreaker.get_state(:anthropic) # state, failure count, last failure reason ``` ### Rate limiting and concurrency caps A per-provider token bucket plus an in-flight concurrency cap, both enforced by the dispatcher before the provider is ever called. ```elixir config :lemon_ai, :rate_limiter, tokens_per_second: 10, # refill rate (default: 10) max_tokens: 20 # bucket capacity (default: 20) LemonAi.CallDispatcher.set_concurrency_cap(:anthropic, 20) LemonAi.CallDispatcher.get_active_requests(:anthropic) ``` ### Cost tracking Every model carries pricing, so you can price any response: ```elixir {:ok, message} = LemonAi.complete(model, context) cost = LemonAi.calculate_cost(model, message.usage) # cost.total, cost.input, cost.output, cost.cache_read, cost.cache_write (US dollars) ``` ### Automatic retries Providers retry transient failures (429, 5xx, connection resets, TLS hiccups) on the call's own async task with exponential backoff + jitter, honoring `retry-after` headers when present. Retries are bounded — 2 for Anthropic, 3 for the OpenAI family — and each request retries in isolation, so one slow retry never blocks another caller. ### Context compaction `LemonAi.CompactingClient` wraps a call and, on a `ContextLengthExceeded` error, automatically compacts the conversation and retries instead of failing outright. ### Token estimation `LemonAi.Tokens` gives fast token *estimates* for budgeting and thresholds: ```elixir LemonAi.Tokens.estimate_chars("some prompt text") # ~ String.length / 4 LemonAi.Tokens.estimate_bytes(payload) # ~ byte_size / 4 ``` > **Caveat:** this is a rough **4-characters-per-token heuristic, not a real > tokenizer.** It will diverge from any model's actual token count and must not be > trusted for billing or hard context-window limits. Use it for quick thresholds and > diagnostics only; use `message.usage` (the provider's reported counts) for anything > that needs to be accurate. ### Actionable, classified errors `LemonAi.Error` parses provider error bodies into a normalized category and tells you whether to retry: ```elixir parsed = LemonAi.Error.parse_http_error(429, response_body, headers) # parsed.category => :rate_limit | :auth | :client | :server | :transient # parsed.retryable => true # parsed.rate_limit_info.retry_after => merged from headers or body hints LemonAi.Error.retryable?(:timeout) # => true LemonAi.Error.auth_error?({:http_error, 401, "Unauthorized"}) # => true LemonAi.Error.suggested_retry_delay({:http_error, 429, _}) # => 60_000 ``` It handles the OpenAI/Anthropic map shapes, Google `errors` arrays, FastAPI/Pydantic `detail` arrays, OAuth `error_description`, and JSON:API `errors` — so you get a useful message instead of a raw blob, whatever provider you hit. ## Model lookup ```elixir LemonAi.Models.get_model(:anthropic, "claude-haiku-4-5") # provider + id LemonAi.Models.find_by_id("gpt-4o") # search all providers by id LemonAi.Models.get_models(:openai) # all models for a provider LemonAi.Models.get_providers() # all known provider atoms ``` You can also skip the registry entirely and hand-build a `%LemonAi.Types.Model{}` — the registry is a convenience, not a requirement. ## Configuration reference ```elixir config :lemon_ai, LemonAi.CallDispatcher, stream_result_timeout_ms: 300_000 # how long the dispatcher tracks a stream's result ``` `LemonAi.ModelCache` caches provider `GET /models` availability with a configurable TTL (default 5 minutes). ## Key types All defined in `LemonAi.Types`: ```elixir %LemonAi.Types.Model{ id: String.t(), name: String.t(), api: atom(), # must match a registered api_id provider: atom(), # keyed for circuit breaker / rate limiter base_url: String.t(), reasoning: boolean(), input: [:text | :image], cost: %LemonAi.Types.ModelCost{input: float(), output: float()}, context_window: non_neg_integer(), max_tokens: non_neg_integer(), headers: map(), compat: map() | nil } %LemonAi.Types.StreamOptions{ temperature: float() | nil, max_tokens: non_neg_integer() | nil, api_key: String.t() | nil, headers: map(), reasoning: :minimal | :low | :medium | :high | :xhigh | nil, stream_timeout: timeout(), # default 300_000ms tool_choice: atom() | String.t() | nil # ...plus Vertex/OAuth fields: project, location, access_token, service_account_json } %LemonAi.Types.Context{ system_prompt: String.t() | nil, messages: [message()], # stored newest-first for O(1) append tools: [Tool.t()] } ``` > `Context.messages` is stored reversed (newest first). Use > `LemonAi.Types.Context.get_messages_chronological/1` when passing to an API directly. ## Streaming event types Events emitted by `LemonAi.EventStream`: ```elixir {:start, message} {:text_start, idx, message} {:text_delta, idx, delta, message} {:text_end, idx, text, message} {:thinking_start | :thinking_delta | :thinking_end, idx, ..., message} {:tool_call_start, idx, message} {:tool_call_delta, idx, json_fragment, message} {:tool_call_end, idx, tool_call, message} {:done, stop_reason, message} {:error, stop_reason, message} {:canceled, reason} ``` `stop_reason` is one of `:stop | :length | :tool_use | :error | :aborted`. ## Architecture Every call flows through a dispatcher that checks the circuit breaker, acquires a rate-limit permit, and enforces the concurrency cap before invoking the provider. Streaming responses come back through an `LemonAi.EventStream` GenServer with a bounded queue, backpressure, owner monitoring, and timeouts. ``` LemonAi.stream/3 or LemonAi.complete/3 → LemonAi.ProviderRegistry -- O(1) :persistent_term lookup by api_id → LemonAi.CallDispatcher.dispatch/2 -- circuit breaker + rate limiter + concurrency cap → LemonAi.Provider.stream/3 -- provider behaviour callback → LemonAi.EventStream -- async delivery, backpressure, cancellation ``` Supervision tree: ``` LemonAi.Supervisor (one_for_one) ├── Task.Supervisor (LemonAi.StreamTaskSupervisor) ├── Registry (LemonAi.RateLimiterRegistry) ├── Registry (LemonAi.CircuitBreakerRegistry) ├── LemonAi.ProviderSupervisor -- DynamicSupervisor for per-provider breakers/limiters ├── LemonAi.CallDispatcher └── LemonAi.ModelCache ``` `LemonAi.ProviderRegistry` lives outside the tree in `:persistent_term`, so provider mappings survive process restarts. ## Adding a provider Implement the `LemonAi.Provider` behaviour (`stream/3`, `provider_id/0`, `api_id/0`, and optionally `get_env_api_key/0`), add a model catalog under `LemonAi.Models.*`, and register the module in `LemonAi.Application`: ```elixir LemonAi.ProviderRegistry.register(:my_provider_api, LemonAi.Providers.MyProvider) ``` Inside `stream/3` you start an `LemonAi.EventStream`, run the HTTP request in a supervised task, and push events (`LemonAi.EventStream.push_async/2`) until you complete or error the stream. See any module under `lib/ai/providers/` for the pattern. ## Authentication and OAuth Most providers authenticate with a bearer key from options or the environment. Providers that require OAuth (GitHub Copilot, Google Gemini CLI, OpenAI Codex, Google Antigravity) have helpers under `LemonAi.Auth.*` for the device-code / PKCE flows and token refresh. These are protocol helpers only — they do not read or write any external app's secret store. Common environment variables (used as a standalone fallback when a key isn't passed in options): | Variable | Provider | |----------|----------| | `ANTHROPIC_API_KEY` | Anthropic (and Kimi/OpenCode/MiniMax compat) | | `OPENAI_API_KEY` | OpenAI family | | `GOOGLE_GENERATIVE_AI_API_KEY` | Google AI Studio (also `GOOGLE_API_KEY`, `GEMINI_API_KEY`) | | `AZURE_OPENAI_API_KEY` | Azure OpenAI | | `AWS_ACCESS_KEY_ID` / `AWS_SECRET_ACCESS_KEY` / `AWS_REGION` | Bedrock | Debug/diagnostic toggles: `LEMON_AI_HTTP_TRACE=1` (HTTP trace logging), `LEMON_AI_PROMPT_DIAGNOSTICS=1` (prompt size/token diagnostics), `LEMON_AI_DEBUG=1` (raw Anthropic SSE logging). ## Testing ```bash mix test apps/lemon_ai # from umbrella root mix test apps/lemon_ai/test/lemon_ai/circuit_breaker_test.exs # one file mix test apps/lemon_ai/test/integration --include integration # needs API keys ``` HTTP is mocked with `Req.Test` stubs; see `test/providers/` for patterns. ## Dependencies | Dependency | Purpose | |------------|---------| | `req ~> 0.5` | HTTP client with streaming support | | `jason ~> 1.4` | JSON encoding/decoding | | `nimble_options ~> 1.1` | Options validation | | `plug ~> 1.16` (test only) | `Req.Test` stubs | ## Used by `lemon_ai` is the LLM layer of the [Lemon](https://github.com/z80dev/lemon) agent platform, where it drives long-running agents across every provider above. It has zero dependency on the rest of that platform and is designed to be used entirely on its own. ## License MIT.