Holds multiple models resident and routes requests to them by id.

The manager is a node-wide singleton GenServer that owns an ETS table of loaded models. Following the otp-thinking ETS pattern, lifecycle writes serialize through the GenServer, while inference-time lookups read the ETS table directly from the caller — so the manager never becomes a throughput bottleneck for generate/3, stream/3, chat/3, or embed/3.

It is a singleton by design: the client API targets the manager by its module name, and the backing Registry/DynamicSupervisor use fixed names. Start at most one per node — init/1 refuses a second instance with a clear error.

Because the slow parts of load/3 (Hub download + native model load) run in a supervised Task rather than the GenServer process, a long load does not block other lifecycle calls (unload/1, set_default/1, or concurrent load/3s). The memory-budget reservation and the ETS commit are still serialized on the GenServer, so resident models are always accounted for. (The budget remains advisory: a model's footprint is only reserved once its size is known — after resolve — so two models downloading at once may momentarily under-count each other.)

Start it as part of LlamaCppEx.ModelSupervisor, which also starts the Registry and DynamicSupervisor that server-backed models need:

children = [
  {LlamaCppEx.ModelSupervisor,
   memory_budget: :auto,
   models: [
     {"chat", {:hub, "Qwen/Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q8_0.gguf"}, n_gpu_layers: -1},
     {"embed", {:path, "/models/nomic-embed.gguf"}, capabilities: [:embed]}
   ]}
]

Backing modes

• :server (default for generation/chat) — backs the model with a supervised LlamaCppEx.Server, giving continuous batching, streaming, prefix cache, and telemetry.
• :direct (auto-selected when :embed is in :capabilities) — holds the %LlamaCppEx.Model{} and runs stateless LlamaCppEx.generate/3 / LlamaCppEx.Embedding.embed/2. Mandatory for embeddings, since the server has no embedding path.

Routing

• Explicit id: generate("chat", prompt).
• Default model: generate(:default, prompt) routes to the model marked default: true at load, or set via set_default/1.

Unloading and memory

Model cleanup is GC-based: unload/1 stops the backing server (dropping its context and model refs) and removes the ETS entry, then forces a GC. Because reclamation is by garbage collection, any caller still holding a %Model{} obtained via fetch_model/1 keeps the underlying model alive past unload/1. Prefer id-based dispatch and avoid holding raw refs.

Loads are checked against an advisory memory budget (see LlamaCppEx.ModelManager.Budget); over-budget loads are refused with {:error, {:insufficient_memory, ...}}.