Botica.Batteries.LlamaServer (Botica v2.2.0)

Copy Markdown View Source

Llama-server lifecycle management — used by any project that needs to download, configure, launch, and health-check a llama-server process running a specific GGUF model.

Capabilities

Roles

Two roles are handled side by side (chat completion vs embeddings):

  • :chat — higher --parallel, larger --ctx-size,
               reasoning-friendly args (spec-decode, cache q8).
  • :embedding — --embedding, smaller model, separate batch
               ergonomics.

Both call into the same generic launcher, differing only in the role-specific args and config block they accept.

Use cases

  • delfos (this project's primary consumer) uses :chat and :embedding simultaneously via its Delfos.LLM.Internal and Delfos.LLM.Embed adapters, each a thin wrapper that delegates to this module.
  • Any other Elixir app that needs a managed local LLM can reuse the same module without duplicating the lifecycle plumbing.

Summary

Types

config()

Resolved config passed to start/2.

role()

Role selectors.

Functions

build_args(role, config)

Builds the full arg list for llama-server. Merges

check(role, config)

Builds a ready-to-use Botica.Check definition for :chat or :embedding so the role can show up in Botica.Doctor.run/1 output. Pass the same config you would pass to start/3.

find_binary(opts \\ [])

Finds the absolute path to llama-server, downloading a precompiled binary if needed. Priority

running?(port)

Probes the /health endpoint of a running llama-server.

start(role, config, opts \\ [])

Spawns a llama-server subprocess and waits for it to become ready. Returns {:ok, :started} (GPU) or {:ok, :started_cpu} depending on the device resolved during build_args/2. Returns :already_running if the server was already reachable before the attempt.

stop(opts \\ [])

Stops the server registered under the given server_id. Idempotent.

Types

config()

@type config() :: %{
  :gguf_path => String.t(),
  :port => pos_integer(),
  optional(:host) => String.t(),
  optional(:api_key) => String.t(),
  optional(:alias) => String.t(),
  optional(:binary) => String.t(),
  optional(:llama_server_path) => String.t(),
  optional(:hardware_plan) => %{device: :gpu | :cpu},
  optional(:extra_args) => [String.t()],
  optional(any()) => any()
}

Resolved config passed to start/2.

role()

@type role() :: :chat | :embedding

Role selectors.

Functions

build_args(role, config)

@spec build_args(role(), config()) :: [String.t()]

Builds the full arg list for llama-server. Merges:

  • defaults for the role (:chat / :embedding)
  • user overrides (in config)
  • auto-resolved device (hardware_plan.device) when :auto
  • extra_args (always appended last)

Returns a list suitable to pass to Arrea.LongRunning.start_link/1 or Port.open/2.

check(role, config)

@spec check(role(), config()) :: Botica.Check.Behaviour.spec()

Builds a ready-to-use Botica.Check definition for :chat or :embedding so the role can show up in Botica.Doctor.run/1 output. Pass the same config you would pass to start/3.

Examples

Botica.Doctor.run(%{
  app_name: "delfos",
  checks: [
    Botica.Batteries.LlamaServer.check(:chat, %{port: 9999, ...}),
    Botica.Batteries.LlamaServer.check(:embedding, %{port: 9998, ...})
  ]
})

find_binary(opts \\ [])

@spec find_binary(keyword()) :: {:ok, String.t()} | {:error, term()}

Finds the absolute path to llama-server, downloading a precompiled binary if needed. Priority:

  1. Explicit override (llama_server_path).
  2. which llama-server in PATH (returns absolute path).
  3. Installable via Botica.Batteries.LlamaServer.Installer.

Options

  • :force_download — re-download even if a binary already exists.

running?(port)

@spec running?(pos_integer()) :: boolean()

Probes the /health endpoint of a running llama-server.

Examples

iex> Botica.Batteries.LlamaServer.running?(9999)
true

start(role, config, opts \\ [])

@spec start(role(), config(), keyword()) ::
  :already_running | {:ok, :started} | {:ok, :started_cpu} | {:error, term()}

Spawns a llama-server subprocess and waits for it to become ready. Returns {:ok, :started} (GPU) or {:ok, :started_cpu} depending on the device resolved during build_args/2. Returns :already_running if the server was already reachable before the attempt.

Options

  • :force — kill the existing server and re-spawn.
  • :health_check — override the health probe (defaults to GET /health). Receives the port.
  • :wait_timeout — max ms to wait for readiness (default 30 s).

stop(opts \\ [])

@spec stop(keyword()) :: :ok

Stops the server registered under the given server_id. Idempotent.