Omni.Sources.ModelsDev (Omni v1.6.0)

Copy Markdown View Source

The default model source, reading a bundled models.dev snapshot.

The snapshot (priv/models/models_dev.json) is a verbatim capture of the full models.dev catalog, refreshed with mix omni.snapshot. Every Omni release ships a fresh snapshot, with data-only patch releases for notable catalog changes and at least one release per month; use live mode (below) when you need fresher data than the release cadence provides. The snapshot is transformed into %Omni.Model{} structs at load time: deprecated models and those without tool use or text modalities are filtered out, modalities are narrowed to those Omni supports, and each model's dialect is inferred from models.dev's npm package metadata, falling back to the provider's declared dialect. Models that still fail to build are skipped with a warning.

Providers are matched to catalog entries by their canonical id (the module's id/0), translated mechanically to the models.dev key — :fireworks_ai becomes "fireworks-ai". Any catalog id works, including providers Omni ships no module for:

defmodule MyApp.Providers.Mistral do
  use Omni.Provider, id: :mistral, dialect: Omni.Dialects.OpenAICompletions

  @impl true
  def config do
    %{base_url: "https://api.mistral.ai", api_key: {:system, "MISTRAL_API_KEY"}}
  end
end

Options

  • :provider_id — the catalog id to load models from, overriding the module's id/0. An atom is translated like a canonical id; a string is used as a verbatim models.dev catalog key.
  • :live — when true, fetch fresh catalog data from models.dev at load time instead of reading the bundled snapshot. Defaults to false.
  • :cache_ttl — how long a cached live catalog stays fresh, in milliseconds. Defaults to to_timeout(hour: 24); 0 means always fetch. Ignored unless live: true.
  • :cache_dir — directory for the live catalog cache. Defaults to a per-user directory under System.tmp_dir!() (e.g. /tmp/omni_alice). Ignored unless live: true.

Live mode

config :omni, :models,
  source: {Omni.Sources.ModelsDev, live: true}

Live mode fetches the full models.dev catalog over HTTPS and caches the response on disk. Each load walks a resilience ladder, logging a warning at every degradation step:

  1. Cache fresh (younger than :cache_ttl) — use it, no network.
  2. Cache stale or missing — fetch from models.dev; on success, write the cache and use the fetched data.
  3. Fetch failed — use the stale cache (stale beats empty).
  4. No usable cache — fall back to the bundled snapshot.

The fetch runs synchronously inside :omni's application start, so live mode adds network egress at boot and — when the cache is cold and models.dev is unreachable — up to ~5 seconds of boot latency. The request budget is fixed (no retries) and deliberately not configurable: failure always degrades down the ladder instead of crashing the boot.

The cache file is the raw api.json response, replaced atomically and validated before use. Deleting it at any time is safe, and an unreadable cache is treated as missing. Cache write failures never fail a load — the fetched data is simply used in-memory for that boot.