Deployments, draining, and readiness

This guide covers what has to happen when a node goes away: a rolling deploy, a scale-down, a machine migration, a restart. It applies to :local and :horde alike, and it matters more the more nodes you run.

The short version:

A node stops being able to serve agent requests the moment Sagents.Supervisor shuts down, but it keeps accepting TCP connections for the rest of the platform's grace period. If your load balancer is still routing there, every one of those requests fails. Tell the platform the node is not ready before you let the supervision tree stop.

The problem, concretely

Sagents.Registry is the map from agent_id to a process. Every agent lookup reads it. It lives in ETS tables owned by the registry process on the local node, so when that process stops, the tables go with it.

On SIGTERM, OTP shuts your application's supervision tree down in reverse child order. Sagents.Supervisor and its registry terminate. The BEAM keeps running after that for however long the platform allows, commonly 30 to 60 seconds. During that window:

• the node still holds open connections and still accepts new ones,
• but no agent lookup on it can succeed,
• and no retry helps, because the registry is not coming back on this node.

This is not a race with a small window. It is a stable condition for the whole drain period. It is also local: other nodes in the cluster are healthy and can serve the request perfectly well. That is what makes it a routing problem rather than an outage.

The sequence you need

1. platform signals shutdown
2. node reports NOT ready          <-- your drain flag flips here
3. load balancer stops routing to it
4. in-flight requests finish
5. supervision tree stops           <-- Sagents.ready?() goes false here
6. BEAM exits

Out of the box, only steps 5 and 6 happen. Steps 2 through 4 are yours to wire up, and they are the whole fix. The library gives you the signal for step 5 and a clean error if a request slips through anyway.

Note where the two signals sit. Sagents.ready?/0 first answers false at step 5, which is after the load balancer needed it at step 2. So a readiness endpoint wired to Sagents.ready?/0 alone answers 200 for the entire drain and then starts failing requests at the same instant it starts reporting unhealthy. That is worse than nothing, because it looks finished.

Readiness needs two sources: a flag you set at step 2 (step 3 below), and Sagents.ready?/0 for every other way the tree can be down (still booting, crashed, restarting).

Step 1: readiness, not liveness

Sagents.ready?/0 answers whether this node can host and route agent sessions. Wire it into your readiness check, together with the drain flag from step 3.

Keep it out of your liveness check. A draining node is not an unhealthy one, and what that costs depends on the platform: Kubernetes restarts a pod that fails liveness, which is exactly the wrong move, while Fly never acts on a check result at all. The reason to have alive/2 on either platform is that it is the signal that stays green while readiness goes red, the only way to read your platform's check list and tell a node draining normally (alive 200, ready 503) from one that is wedged (both failing). Readiness alone cannot express that difference, and during a rolling deploy it is the difference an operator is looking at.

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  # Liveness: is the BEAM up at all? Never consults Sagents or the drain flag.
  def alive(conn, _params) do
    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(200, "ok")
  end

  # Readiness: should this node receive traffic? The drain flag flips first,
  # at step 2; Sagents.ready?/0 covers every other way the tree can be down.
  def ready(conn, _params) do
    {status, body} =
      if MyApp.Drain.draining?() or not Sagents.ready?() do
        {503, "draining"}
      else
        {200, "ok"}
      end

    conn
    |> put_resp_content_type("text/plain")
    |> send_resp(status, body)
  end
end

# lib/my_app_web/router.ex
scope "/health", MyAppWeb do
  get "/alive", HealthController, :alive
  get "/ready", HealthController, :ready
end

The scope has no pipe_through on purpose. Not the browser pipeline, since a probe has no session and no CSRF token, and not an :api pipeline either: plug :accepts, ["json"] makes the route content-negotiate, and probes frequently send no Accept header at all. The probe then takes a 406 and reads the node as unhealthy for a reason that has nothing to do with Sagents.

Check endpoint.ex for anything above the router that can halt or redirect, too. force_ssl is the common one: probes reach the machine over plain HTTP on the internal network with no x-forwarded-proto to rewrite on, so Plug.SSL answers 301 and the readiness endpoint never runs. Exclude both full paths, and repeat the host defaults, because passing any :exclude value replaces Plug.SSL's defaults entirely:

force_ssl: [
  rewrite_on: [:x_forwarded_proto],
  exclude: [
    hosts: ["localhost", "127.0.0.1"],
    # Exact full-path matches, not prefixes. `paths: ["/health"]` does NOT
    # cover "/health/ready": Plug.SSL compares conn.path_info for equality.
    paths: ["/health/alive", "/health/ready"]
  ]
]

Step 2: supervision tree order

Sagents.Supervisor must come before your Endpoint:

children = [
  MyApp.Repo,
  {Phoenix.PubSub, name: MyApp.PubSub},
  MyAppWeb.Presence,
  Sagents.Supervisor,     # after Repo/PubSub, so agents can persist on the way down
  MyAppWeb.Endpoint,      # after Sagents, so OTP stops the listener FIRST
  {MyApp.Drain, delay: drain_delay()}   # LAST, so it stops FIRST
]

OTP shuts children down in reverse order, and that single rule is what makes all three positions correct at once. Trace it: Drain → Endpoint → Sagents.Supervisor → PubSub → Repo. The drain sleeps while the listener is still up and can still answer the probe, then the listener stops, then the registry, then the things agents needed in order to persist themselves.

The Endpoint's position matters on its own: listed the other way around, it outlives the registry and serves the entire drain period with a dead registry underneath it.

The drain process must be last, which is the placement that is easy to get backwards. Listed anywhere earlier, its wait happens behind an already-stopped listener, where no probe can observe the readiness change it exists to publish.

Correct ordering narrows the window. It does not remove it: the node is still reachable by the load balancer until the platform stops sending traffic, which is what step 1 is for. Do both.

Step 3: give the platform time to notice

A readiness check that flips to false at the same instant the tree comes down buys you nothing. The platform needs to poll it, observe the change, and update its routing table before shutdown proceeds. That is what the drain delay is for.

The general principle, whatever the platform:

1. Intercept the shutdown signal.
2. Make readiness report false.
3. Wait longer than (readiness poll interval × unhealthy threshold), plus the time your longest ordinary request takes.
4. Only then let the supervision tree stop.

A small process at the end of your tree does all four:

defmodule MyApp.Drain do
  @moduledoc """
  Holds shutdown open long enough for the load balancer to observe readiness
  going false and stop routing here.

  Listed LAST in the application tree. OTP stops children in reverse order, so
  last means `terminate/2` runs first, while the Endpoint is still serving and
  can still answer the readiness probe.
  """
  use GenServer

  require Logger

  @flag {__MODULE__, :draining?}

  @doc "Whether this node has begun shutting down. Safe to call from a request."
  def draining?, do: :persistent_term.get(@flag, false)

  def start_link(opts), do: GenServer.start_link(__MODULE__, opts, name: __MODULE__)

  def child_spec(opts) do
    delay = Keyword.get(opts, :delay, 0)

    %{
      id: __MODULE__,
      start: {__MODULE__, :start_link, [[delay: delay]]},
      # Must exceed the delay, or the supervisor brutal-kills this process
      # partway through terminate/2 and the drain silently does not happen.
      shutdown: delay + :timer.seconds(5)
    }
  end

  @impl true
  def init(opts) do
    # Without this, terminate/2 is not called on supervisor shutdown at all.
    Process.flag(:trap_exit, true)
    :persistent_term.put(@flag, false)
    {:ok, Keyword.get(opts, :delay, 0)}
  end

  @impl true
  def terminate(_reason, delay) do
    :persistent_term.put(@flag, true)
    Logger.info("draining: readiness is now false, waiting #{delay}ms before shutdown")
    Process.sleep(delay)
    :ok
  end
end

Two details that are easy to get wrong and hard to notice:

• The flag must be readable without talking to this process. During terminate/2 the GenServer is out of its receive loop, so a GenServer.call(MyApp.Drain, :draining?) from the readiness endpoint blocks for the whole sleep and then exits when the process dies. :persistent_term is read directly by the caller.
• Configure delay: 0 in dev and test. A drain delay that fires on every Ctrl-C is a delay you will disable in week two, and then it is not there in production either. Take the value from config, so drain_delay/0 in your application module reads Application.get_env(:my_app, :drain_delay_ms, 0) and only config/runtime.exs sets it to something like 20_000.

Fly.io

Fly sends the configured kill_signal, waits kill_timeout, then sends SIGKILL. Nothing happens in between, so kill_timeout is not a stage in a graceful sequence: it is the hard deadline before the machine dies mid-drain, and it must exceed your drain delay.

fly launch detects an Elixir project and generates kill_signal = "SIGTERM", so this is normally already right. Confirm the line rather than assuming it, since everything above depends on it.

# fly.toml. Use this form if your file has [http_service], which is what
# `fly launch` has generated for years.
kill_signal = "SIGTERM"   # confirm this is present and says SIGTERM
kill_timeout = "45s"      # hard deadline before SIGKILL; must exceed the drain

# Readiness. This is the check that changes routing, so this is the one that
# makes the drain do anything at all.
[[http_service.checks]]
  path = "/health/ready"
  interval = "5s"
  timeout = "2s"
  grace_period = "10s"

With a 5 second interval the proxy notices within a few seconds of readiness flipping, so a 45 second kill_timeout leaves room for a 20 second drain.

Wire the liveness endpoint too. On Fly this is not the readiness block with a different path: a liveness check belongs in the top-level [checks] section, a different shape that needs a unique name, an explicit port, and an explicit type.

# Top-level checks do NOT affect routing, and Fly never restarts or stops a
# machine over a failed check. This exists so `fly status` / `fly checks list`
# can separate a node draining normally from one that is wedged.
[checks]
  [checks.alive]
    type = 'http'
    port = 8080          # your internal_port; must be bound on 0.0.0.0
    path = '/health/alive'
    method = 'get'
    interval = '15s'
    timeout = '5s'
    grace_period = '30s' # must cover boot

Do not put /health/alive in [[http_service.checks]] instead. That section governs routing, and an endpoint that always answers 200 tells the proxy nothing while adding a check every deploy has to wait on.

If you run agents in more than one region, read the partition section of Clustering as well: fly-replay routing a request to the wrong region has a different and worse failure mode than draining does.

Kubernetes

Kubernetes sends SIGTERM already, so there is no signal to change.

readinessProbe:
  httpGet: { path: /health/ready, port: 4000 }
  periodSeconds: 5
  failureThreshold: 2

# Both endpoints, or the liveness half of step 1 is wired to nothing. Unlike
# Fly, Kubernetes does act on this one: keep the drain flag out of it, or the
# kubelet restarts the pod you are trying to drain.
livenessProbe:
  httpGet: { path: /health/alive, port: 4000 }
  periodSeconds: 15
  failureThreshold: 3

terminationGracePeriodSeconds: 60

With MyApp.Drain in the tree you do not also need a preStop sleep; terminationGracePeriodSeconds must exceed the drain delay plus your longest request.

If you would rather keep the wait outside the BEAM, drop MyApp.Drain and use lifecycle.preStop.exec.command: ["sleep", "20"] instead. preStop runs before SIGTERM is delivered and endpoint removal happens concurrently with it, so the sleep does give the service mesh time to stop routing. The cost is that readiness then has only Sagents.ready?/0 to report, which goes false at step 5 rather than step 2, which is too late to be the signal the mesh acted on. Prefer the in-BEAM flag.

Step 4: handle the error if one slips through

Even with all of the above, a request can arrive mid-transition. Sagents reports that as a value rather than raising, so you can answer it correctly:

case Sagents.Session.ensure_running(config, socket.assigns, request_opts: opts) do
  {:ok, changes} ->
    assign(socket, changes)

  {:error, :registry_unavailable} ->
    # This node is draining. Another node can serve this fine.
    {:error, message: "Service is restarting, please retry", code: 503}

  {:error, reason} ->
    {:error, message: "Could not start session: #{inspect(reason)}"}
end

Answer 503, not 500. A 500 says the request cannot be served. A 503 says it cannot be served here, which is true, and it is the answer clients and load balancers know how to retry.

Which functions report it, and which raise

Classify by how a call reaches the registry, not by which module it lives in. That rule is short, checkable against the source, and it covers functions this page does not name:

The cast and call asymmetry is why a function's signature tells you nothing on its own. Same via tuple, same dead registry, opposite outcome.

The tuple-returning forms, by layer:

• Sagents.Session: start/3, ensure_running/3, resume/4, dismiss/3, stop/2, fetch_running/2
• Sagents.AgentServer: fetch_pid/1, execute/1, cancel/1, resume/2, add_message/3, reset/1, dismiss_interrupt/1, subscribe/3
• Sagents.FileSystem: ensure_filesystem/3, stop_filesystem/2, get_filesystem_pid/1, fetch_filesystem_running/1
• Sagents.AgentSupervisor: get_pid/1, stop/2
• Sagents.ProcessRegistry: fetch/1, and Sagents.FileSystemServer.fetch_pid/1

The raising ones you are most likely to meet by accident are the two ?-predicates, Sagents.Session.running?/2 and Sagents.FileSystem.filesystem_running?/1, because a name ending in ? is the last thing a reader suspects of raising. Both have non-raising siblings in the list above. Sagents.FileSystem.list_filesystems/0 and Sagents.AgentServer.get_status/1 are the next two: get_status/1 wraps its call in try/catch :exit, so every call site reads as though it cannot fail, while the raise happens before reaching the call that catch guards.

Do not skip the filesystem layer because it looks incidental. Sagents.FileSystem gets read as one function when it is a module of them, and its calls sit in the least suspicious place: best-effort cleanup, where the return is usually discarded. Discarding it there is fine and stays fine. Two other shapes are worth checking:

• A catch-all that logs. {:error, :registry_unavailable} reaching an {:error, reason} -> Logger.error(...) clause turns every deploy into an alarm for a condition that is not a fault.
• ensure_filesystem/3 returning the error. It deliberately does not start a filesystem it could not first check for, because this node cannot see whether one already exists elsewhere. A caller that reads any error as "start failed, carry on degraded" is making a different decision than the one that was reported.

Why it is not just nil

:registry_unavailable is deliberately not folded into "no agent is running". They mean different things and require different responses:

If the second were reported as the first, every request during every drain would start a new agent for a conversation that already has one somewhere else. Two processes would then hold and persist state for the same conversation, silently. That is why the distinction is load-bearing rather than cosmetic.

It is also why the raising functions raise rather than answering. A shape like pid() | nil, a boolean, or a list has no room for a third answer, and every plausible default is the dangerous one: nil, false and [] all read as "nothing is running". A loud, named error is the safer answer, even at the cost of a ?-predicate that can raise. On request paths, prefer the tuple-returning forms listed above.

What happens to running agents

Agent state is durable, which is what makes all of this recoverable.

• Graceful shutdown. Agents terminate, persist their state, and broadcast a shutdown event.
• A node vanishing abruptly. Under :horde, surviving members usually restart the departed node's agents on their own. Treat this as an optimization, not a guarantee — see the warning below.
• The next request either way. Sagents.Session.ensure_running/3 starts the agent from persisted state on whichever node handles it. A conversation that was mid-run resumes as a fresh run rather than continuing the interrupted one.

So a deploy costs you in-flight LLM turns, not conversations. Sizing your drain window is a question of how long you are willing to wait for those turns, not of whether state survives.

If the registry itself fails

A node's registry process crashing is rare, but it has a defined outcome on both backends, and it is deliberately a noisy one.

Sagents.Supervisor supervises its children :rest_for_one with the registry first, so a registry failure restarts the agent and filesystem supervisors too. Running agents on that node stop and are re-created from persisted state on the next request.

That looks heavy-handed and is the point. An AgentSupervisor and an AgentServer register their :via names once, at start, and nothing re-registers them. A registry that comes back empty underneath them leaves them running but invisible to every lookup — so the next request reads "nothing is running" and starts a second AgentServer for a conversation that already has one, with both persisting state and nothing reporting the conflict. Restarting them is recoverable; a silent duplicate is not.

Two internal pieces make that restart reach the right processes, and you do not interact with either:

• Sagents.RegistryWatcher. On both backends the process owning the registry's ETS tables runs under a supervisor of the backend's own, which restarts it with empty tables without failing a child of Sagents.Supervisor. Under :horde that process is the one registered as Sagents.Registry; under :local it is a Registry.Partition one level below the name. The watcher monitors it and fails in its place.
• Sagents.LocalRegistry. Under :local the restart races the outgoing registry's partition process, which traps exits and holds its registered name for a few milliseconds after the registry is gone. A start landing inside that window fails with {:already_started, pid}, and a supervisor does not recover from a failed restart. Sagents.LocalRegistry waits for the straggler to exit and retries.

None of this is part of draining. The watcher is listed after the registry, and OTP stops children in reverse order, so it is already gone by the time the registry stops on a normal shutdown. A draining node never looks like a registry failure.

Verifying it

The failure mode this guide exists to prevent has direct test coverage in test/sagents/horde/rolling_deploy_test.exs, which drives real Erlang nodes through a shutdown while probing them:

mix test test/sagents/horde/rolling_deploy_test.exs --include cluster --include slow

To check your own application, hit /health/ready while a deploy is in flight and confirm it reports 503 before the node stops answering agent requests. If readiness is still reporting 200 at the moment lookups start failing, steps 1 through 3 are not wired correctly.

Check the probe is not being redirected on the way in, which force_ssl will do silently:

curl -sS -o /dev/null -w '%{http_code}\n' http://<internal-address>/health/ready

A 301 means the exclusion in step 1 is missing, and no amount of correct readiness logic will help.

Then confirm the platform is actually asking. Both checks above talk to your app, and the app can be perfectly healthy while the platform has been told to poll nothing, which is what a misspelled check sub-key produces:

fly checks list -a <app>     # must list your checks. An empty table means the
                             # key decoded into nothing
kubectl describe pod <pod>   # the Liveness:/Readiness: lines, same question

Do not substitute fly config validate for this. It reports Configuration is valid for a file containing sections and keys flyctl does not recognize: it confirms well-formed TOML, not that Fly understood any particular line. The curl proves your app answers; fly checks list proves the platform is asking. They are independent failures and you need both.

Related

• Clustering & distribution (Horde) for membership, placement, and partitioning.
• Lifecycle for what an individual agent does on shutdown.
• Persistence for what survives and how it is restored.