ExAthena.Chat.Exo (ExAthena v0.17.0)

Talks to a local exo cluster's HTTP API (https://github.com/exo-explore/exo) for chat-time helpers.

list_models/1 hits GET /v1/models?status=downloaded and returns the model ids downloaded somewhere in the cluster, sorted alphabetically. exo model ids are full HuggingFace ids (e.g. mlx-community/Llama-3.2-1B-Instruct-4bit).

ensure_instance/2 guarantees the model has an active instance before a chat request: exo returns 404 ("No instance found for model …") otherwise. Because POST /place_instance is NOT idempotent (it creates duplicate instances), we check GET /state/instances first and only place when absent, then poll state until the instance registers (sub-second in practice; weight loading happens after registration and only affects first-token latency).

Summary

Functions

ensure_instance(model, opts \\ [])

Guarantees model has an active instance in the exo cluster, placing one if absent and polling until it registers.

list_models(opts \\ [])

Functions

ensure_instance(model, opts \\ [])

@spec ensure_instance(
  String.t(),
  keyword()
) ::
  :ok
  | {:error,
     :exo_unreachable
     | :exo_instance_unavailable
     | :unexpected_response
     | {:http, integer()}}

Guarantees model has an active instance in the exo cluster, placing one if absent and polling until it registers.

Options: :base_url, :poll_interval_ms (default 250), :timeout_ms (default 10_000). The timeout is approximate (a lower bound): each poll adds up to one HTTP round-trip beyond the deadline.

list_models(opts \\ [])

@spec list_models(keyword()) ::
  {:ok, [String.t()]}
  | {:error, :exo_unreachable | :unexpected_response | {:http, integer()}}