Candil.Backend behaviour (Candil v3.0.0)

Copy Markdown View Source

Behaviour that abstracts the local/remote differences between LLM backends. A backend is responsible for:

  • chat/3 — synchronous single-shot chat completion.
  • chat_stream/3 — server-sent-event streaming chat completion.
  • embed/3 — text → vector embeddings, optionally batched.
  • models/0 — list of Candil.Model.t/0 the backend supports.

Backends are stateless wrappers around whatever HTTP / port machinery Candil already exposes (Candil.Engine, Candil.HTTP, Candil.Embeddings). They are looked up at runtime via Candil.Backend.for/2 and the result is cached per (provider, model) pair.

Built-in backends

Custom backends

Define a module that implements this behaviour and register it via Candil.Backend.register/2:

defmodule MyApp.Backends.Azure do
  @behaviour Candil.Backend

  @impl true
  def chat(model, messages, opts), do: ...
  @impl true
  def chat_stream(model, messages, opts), do: ...
  @impl true
  def embed(model, texts, opts), do: ...
  @impl true
  def models, do: [...]
end

Candil.Backend.register(:azure, MyApp.Backends.Azure)

Summary

Types

chat_response()
embed_vector()
stream_chunk()

Callbacks

chat(arg1, list, keyword)
chat_stream(arg1, list, keyword)
embed(arg1, list, keyword)
models()

Functions

backend_unavailable(provider, model)

Build a typed :backend_unavailable error.

for(provider, model)

Look up the backend module registered for (provider, model). Returns {:ok, module} or {:error, :backend_unavailable}.

infer_provider(provider, model)

Infer the provider atom from a model name and the configured default provider. Public so callers can pre-compute the provider for telemetry/UI.

register(provider, module)

Register a backend module for the given provider atom (:local, :openai, :anthropic, :ollama, etc.).

Types

chat_response()

@type chat_response() :: %{
  content: String.t(),
  finish_reason: String.t() | nil,
  usage: %{input_tokens: non_neg_integer(), output_tokens: non_neg_integer()}
}

embed_vector()

@type embed_vector() :: [float()]

stream_chunk()

@type stream_chunk() :: %{
  content: String.t(),
  finish_reason: String.t() | nil,
  done: boolean()
}

Callbacks

chat(arg1, list, keyword)

@callback chat(String.t() | Candil.Model.t(), [map()], keyword()) ::
  {:ok, chat_response()} | {:error, term()}

chat_stream(arg1, list, keyword)

@callback chat_stream(String.t() | Candil.Model.t(), [map()], keyword()) ::
  {:ok, Enumerable.t()} | {:error, term()}

embed(arg1, list, keyword)

@callback embed(String.t() | Candil.Model.t(), [String.t()], keyword()) ::
  {:ok, [embed_vector()]} | {:error, term()}

models()

@callback models() :: [Candil.Model.t()]

Functions

backend_unavailable(provider, model)

@spec backend_unavailable(atom(), term()) :: Candil.Error.t()

Build a typed :backend_unavailable error.

for(provider, model)

@spec for(atom(), String.t() | Candil.Model.t()) :: {:ok, module()} | {:error, term()}

Look up the backend module registered for (provider, model). Returns {:ok, module} or {:error, :backend_unavailable}.

Falls back to inferring the provider from the model name when no exact match is registered:

  • starts with gpt- / o1- / o3- → :openai
  • starts with claude- → :anthropic
  • contains : → :ollama (model tag convention)
  • otherwise → :local (llama.cpp / llama-server)

See Candil.Backend.infer_provider/2.

infer_provider(provider, model)

@spec infer_provider(atom(), String.t() | Candil.Model.t()) :: atom()

Infer the provider atom from a model name and the configured default provider. Public so callers can pre-compute the provider for telemetry/UI.

Examples

iex> Candil.Backend.infer_provider(:openai, "gpt-4o")
:openai
iex> Candil.Backend.infer_provider(:local, "llama3")
:local
iex> Candil.Backend.infer_provider(:local, "llama3:8b")
:ollama

register(provider, module)

@spec register(atom(), module()) :: :ok

Register a backend module for the given provider atom (:local, :openai, :anthropic, :ollama, etc.).