The worker side of the llama.cpp RPC backend: serves this node's devices to a remote client.

children = [
  {LlamaCppEx.RPC.Server,
   endpoint: "10.100.64.2:50052",
   cache_dir: Path.expand("~/.cache/llama.cpp/rpc"),
   n_threads: 10}
]

Bind to the fabric address, not localhost

endpoint is required and there is deliberately no default. Upstream's default is 127.0.0.1, which is useless here twice over: a remote client cannot reach it, and RDMA can never engage on it, because the transport selects an HCA by matching a GID against the socket's local address.

Nothing about this endpoint is authenticated or encrypted. It is a plain TCP port that accepts commands to allocate memory and execute compute graphs. Bind it to a point-to-point fabric address, never to a routable interface.

The tensor cache is worth having

:cache_dir enables upstream's content-addressed cache for tensors over 10 MiB. Without it every model load re-pushes the whole remote share across the network; with it a warm load is close to free. There is no default — pass a path or accept the cost knowingly.

Which transport did I get?

Transport selection is silent auto-negotiation. There is no env var, no endpoint scheme and no return value that tells you. The only signal is the worker's own log with GGML_RPC_DEBUG=1: the absence of RDMA activate failed, staying on TCP means RDMA is live. To force TCP for an A/B, set GGML_RDMA_DEV to a device name that does not exist.

This process cannot stop the server

ggml_backend_rpc_start_server never returns — its accept loop is while (true) and the cleanup after it is unreachable — so the native server runs on a detached thread that outlives this GenServer. terminate/2 therefore does nothing but say so. The OS process is the unit of restart, and that matters in practice: upstream's RPC worker is reported by multiple independent users to grow RSS during inference and never release it, so a long-lived worker wants restarting between runs and watching during them.

Starting it twice on one endpoint fails cleanly at the bind, which is the one guard rail this design does provide — and it is also why this child is restart: :temporary. A restart could not succeed: the listening socket lives on that detached thread inside the same OS process, so it survives this GenServer dying, and the next init/1 would fail EADDRINUSE at the pre-flight bind (SO_REUSEADDR does not let you bind over a socket that is actively listening). Under the default :permanent a single crash would retry, fail identically, exhaust max_restarts and take the whole supervision subtree down — reporting a bind error rather than the original cause. :temporary is what "the OS process is the unit of restart" means as a child spec.

This module exists to own a native resource's lifecycle and to be supervised, which is the only reason to add a process. It carries no state a function could not compute.