LlamaCppEx.ModelSupervisor (LlamaCppEx v0.8.23)

Copy Markdown View Source

Opt-in supervisor for the multi-model manager.

Starts, in order, the Registry and DynamicSupervisor that server-backed models need, then LlamaCppEx.ModelManager. Add it to your application's supervision tree:

children = [
  {LlamaCppEx.ModelSupervisor,
   memory_budget: :auto,
   models: [
     {"chat", {:hub, "Qwen/Qwen3-0.6B-GGUF", "Qwen3-0.6B-Q8_0.gguf"}, n_gpu_layers: -1},
     {"embed", {:path, "/models/nomic-embed.gguf"}, capabilities: [:embed]}
   ]}
]

For quick scripts or IEx, start it directly:

{:ok, _sup} = LlamaCppEx.ModelSupervisor.start_link([])
{:ok, "chat"} = LlamaCppEx.ModelManager.load("chat", {:path, "model.gguf"})

Options

  • :memory_budget - Forwarded to LlamaCppEx.ModelManager (:infinity, :auto, or a byte limit).
  • :models - Models to auto-load after start (loaded by the manager so the supervisor itself does not block on downloads).
  • :name - Names this supervisor. Defaults to LlamaCppEx.ModelSupervisor. It does not rename the manager: LlamaCppEx.ModelManager is a node-wide singleton registered under its module name (the client API targets it there), so only one ModelSupervisor should run per node.

Summary

Functions

Returns a specification to start this module under a supervisor.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

start_link(opts \\ [])

@spec start_link(keyword()) :: Supervisor.on_start()