Returns the count of currently running DurableServer processes.

Returns the current capacity map for this supervisor.

Returns a map with :total (total children across all modules) and per-module capacity information, or nil if no limits are configured.

Examples

iex> current_capacity(MySupervisor)
%{
  :total => %{current: 50, limit: 100},
  MyModule => %{current: 10, limit: 20}
}

iex> current_capacity(UnlimitedSupervisor)
nil

Ensures a DurableServer child process is started under this supervisor.

Unlike start_child/2, this function first checks the registry for an existing process before attempting to start a new one. This is useful when you want to ensure a process exists but don't know if it's already running.

The child spec is {Module, key: key, initial_state: initial_state}. :initial_state is required and must be a map. If a new process is started and no persisted state exists yet, DurableServer passes :initial_state through the module's dump_state/1, the configured backend's encode/decode path, and then load_state/2 before init/1 or init/2. This means the dumped initial state must be encodable by the configured backend, and load_state/2 receives the backend-decoded shape.

Options

• :local_only - When true, the child will only be started on the local node. Skips sticky placement preferences and never attempts remote placement. If the local node is at capacity, returns {:error, {:capacity_limit, reason}}. Default: false.
• :max_placement_retries - Maximum number of remote nodes to try when local placement fails due to capacity limits. Default: 3. Ignored when local_only: true.
• :placement_timeout - Maximum time in milliseconds to keep retrying remote placement. When set, if all placement attempts fail, retries with fresh eligible nodes every 500ms until the deadline. Default: nil (no retry).
• :timeout - Maximum total time in milliseconds to wait for the process to be found or bootstrapped. Returns {:error, :timeout} on expiration. Set to :infinity to disable. Default: 5000ms.

Returns

• {:ok, {pid, meta}} - Process is running (either found or newly started)
• {:error, reason} - Failed to start the process

Examples

# Will start if not running, or return existing process
{:ok, {pid, meta}} = DurableServer.Supervisor.ensure_started_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{initial_value: 42}}
)

# Ensure locally only — never attempt remote placement
{:ok, {pid, meta}} = DurableServer.Supervisor.ensure_started_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{}},
  local_only: true
)

# Calling again returns the same process
{:ok, {^pid, ^meta}} = DurableServer.Supervisor.ensure_started_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{initial_value: 42}}
)

Gets all cluster nodes from the heartbeat cache with their heartbeat metadata.

Returns a map of node names to node info maps containing heartbeat_meta.

Examples

iex> get_cluster_nodes(MyApp.DurableSupervisor)
%{
  "node1@host" => %{heartbeat_meta: %{"region" => "ord"}},
  "node2@host" => %{heartbeat_meta: nil}
}

Gets detailed information about a server from storage.

Returns a rich map with server information regardless of whether the server is currently running. This is useful for admin dashboards, debugging, and re-homing decisions.

Return Value

Returns {:ok, info_map} on success or {:error, :not_found} if the server doesn't exist in storage.

The info map contains:

• :key - The server's unique key
• :module - The DurableServer module
• :vsn - The state version
• :status - Server status (:running, :stopped_graceful, :crashed, etc.)
• :permanent - Whether the server is marked as permanent
• :last_heartbeat_at - Timestamp of last heartbeat (milliseconds)
• :node - The node where the server last ran (from storage)
• :sticky_placement - Current placement values (where it last ran)
• :sticky_placement_history - History of placement changes (most recent first)
• :crash_history - List of crash entries (most recent first), each with :timestamp and :reason
• :user_state - The raw user state (JSON decoded from storage)
• :pid - PID if currently running, nil otherwise
• :running - Boolean indicating if server is currently running

Placement History

The sticky_placement_history tracks placement changes over time. Each entry contains an :at timestamp and :placement values. Only unique placements are recorded (no duplicates when placement doesn't change). The history is capped at a configurable limit (default 5), with oldest entries pruned first.

The first entry is the most recent placement, and the last entry is the oldest known placement (which may be the original if history hasn't been pruned):

info = DurableServer.Supervisor.get_server_info(MySup, "user_123")
case info.sticky_placement_history do
  [current | _rest] ->
    # current.placement is where it's running now
    # current.at is when it moved there
  [] ->
    # No placement history (no sticky config or new server)
end

Examples

iex> DurableServer.Supervisor.get_server_info(MyDurableSup, "user_123")
{:ok, %{
  key: "user_123",
  module: MyServer,
  vsn: 1,
  status: :running,
  permanent: true,
  last_heartbeat_at: 1704067200000,
  node: "node1@host",
  sticky_placement: [%{env_var: "FLY_REGION", value: "sjc"}],
  sticky_placement_history: [
    %{at: 1704067200000, placement: [%{env_var: "FLY_REGION", value: "sjc"}]},
    %{at: 1704000000000, placement: [%{env_var: "FLY_REGION", value: "ord"}]}
  ],
  user_state: %{"count" => 42},
  pid: #PID<0.123.0>,
  running: true
}}

iex> DurableServer.Supervisor.get_server_info(MyDurableSup, "nonexistent")
{:error, :not_found}

Gets all global members matching this supervisor name on the cluster along with their metadata.

Returns a map of all members in the form %{key => {pid, meta}}.

Examples

# Get all members for a supervisor
DurableServer.Supervisor.global_members(MySup)
#=> %{"user_1" => {#PID<0.123.0>, %{...}}, "user_2" => {#PID<0.124.0>, %{...}}}

# Get only members for a specific module
DurableServer.Supervisor.global_members(MySup, MyServer)
#=> %{"user_1" => {#PID<0.123.0>, %{...}}}

Looks up a global durable server by key.

Note: the provided key is not prefixed – the configured supervisor prefix will automatically be applied when looking up the key from underlying storage.

Examples

{DurableServer.Supervisor, name: MyDurableSup, prefix: "myapp/"}
{:ok, {pid, _meta}} = DurableServer.Supervisor.start_child(
  MyDurableSup,
  {Counter, key: "counter123", initial_state: %{value: 0}}
)

iex> {pid, _meta} = DurableServer.Supervisor.lookup(MyDurableUp, "counter123")

Gets the unique node reference for this supervisor instance.

Note: other nodes will rpc us and call this function, which can race our table creation and config insert, so we handle those cases explicitly.

The node_ref is used to detect when a node has been restarted to avoid PID reuse from making stale locks appear valid. Each supervisor maintains its own node_ref in ets storage that gets cleaned up when supervisor dies.

Checks if the DurableServer.Supervisor is ready to handle requests.

Returns true once the supervisor and its lifecycle manager child are registered, false otherwise.

This is safe to call at any time, even if the supervisor hasn't started yet.

Rehomes a DurableServer child to a different node, bypassing sticky placement.

This is useful for manual rebalancing or administrative operations. The operation:

1. Terminates the process gracefully on its current node (if running)
2. Starts the process on the target node (or any eligible node if no target specified)

Parameters

• supervisor - The DurableServer.Supervisor name
• child_spec - The child spec tuple {module, key: "...", initial_state: %{...}}
• opts - Options:
  • :target_node - Specific node atom to place on (optional, defaults to best available)
  • :force - If true, ignore sticky placement entirely (default: true)

Returns

• {:ok, {pid, meta}} - Successfully rehomed the process
• {:error, reason} - Failed to rehome

Examples

# Rehome to a specific node
{:ok, {pid, meta}} = DurableServer.Supervisor.rehome_child(
  MySup,
  {MyServer, key: "server_1", initial_state: %{}},
  target_node: :"node2@host"
)

# Rehome to any available node (ignoring sticky placement)
{:ok, {pid, meta}} = DurableServer.Supervisor.rehome_child(
  MySup,
  {MyServer, key: "server_1", initial_state: %{}}
)

Starts a DurableServer child process under this supervisor.

The child spec is {Module, key: key, initial_state: initial_state}. :initial_state is required and must be a map. Before the first init/1 or init/2 call, DurableServer passes it through the module's dump_state/1, the configured backend's encode/decode path, and then load_state/2. This means the dumped initial state must be encodable by the configured backend, and load_state/2 receives the backend-decoded shape.

Options

• :local_only - When true, the child will only be started on the local node. If the local node is at capacity, returns {:error, {:capacity_limit, reason}} instead of attempting remote placement. Default: false.
• :max_placement_retries - Maximum number of remote nodes to try when local placement fails due to capacity limits. Default: 3. Ignored when local_only: true.
• :placement_timeout - Maximum time in milliseconds to keep retrying remote placement. If all placement attempts fail, the caller retries with fresh eligible nodes every 500ms until the deadline. Useful during rolling deploys when nodes are temporarily unavailable. Set to nil to disable. Default: 15000ms.
• :timeout - Maximum total time in milliseconds to wait for the child bootstrap to complete, including internal retries. Returns {:error, :timeout} on expiration. Set to :infinity to disable. Default: 5000ms.

Examples

# Start with init args
{:ok, {pid, meta}} = DurableServer.Supervisor.start_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{initial_value: 42}}
)

# Start locally only — never attempt remote placement
{:ok, {pid, meta}} = DurableServer.Supervisor.start_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{}},
  local_only: true
)

# Retry placement for up to 15 seconds during rolling deploys
{:ok, {pid, meta}} = DurableServer.Supervisor.start_child(
  MyApp.DurableSup,
  {MyServer, key: "server_1", initial_state: %{}},
  placement_timeout: 15_000
)

# The server module must use DurableServer
defmodule MyServer do
  use DurableServer, vsn: 1

  def init(%{initial_value: value}, info) do
    {:ok, %{value: value, key: info.key}, meta: %{my: "meta"}}
  end
end

Starts a DurableServer.Supervisor with the given options.

Options

• :name - Required. The registered name for this supervisor
• :prefix - Required. Object storage prefix (should end with "/")
• :max_children - Maximum concurrent children (default: :infinity)
• :discovery_interval_ms - Lifecycle discovery interval (default: 60_000)
• :initial_discovery_delay_ms - Initial discovery delay as a fixed integer or {min_ms, max_ms} jitter tuple (default: {1_000, 6_000})
• :discovery_shuffle_batch_size - Discovery shuffle batch size (default: 20_000)
• :parallel_restart_batch_size - Concurrent restart attempts per node (default: 50)
• :restart_start_timeout_ms - Timeout for LM-owned claimed restarts (default: 30_000)
• :restart_claim_preferred_fanout - Initial restart claim contention fanout (default: 2)
• :restart_claim_expanded_fanout - Expanded restart claim contention fanout (default: 4)
• :restart_claim_gate_expand_after_ms - Age before widening claim fanout (default: 30_000)
• :restart_claim_gate_disable_after_ms - Age before disabling the claim gate (default: 120_000)
• :heartbeat_interval_ms - Node heartbeat interval (default: 10_000)
• :heartbeat_staleness_threshold_ms - Node heartbeat stale/orphan threshold (default: 30_000)
• :heartbeat_tracking_mode - Heartbeat cache strategy: :poll or :subscribe
• :heartbeat_reconcile_interval_ms - Full heartbeat cache reconcile interval
• :dead_node_threshold_ms - Dead node cleanup threshold (default: 300_000)
• :crash_threshold_count - Crashes before permanent crash (default: 5)
• :crash_threshold_window_ms - Crash threshold window (default: 3_600_000)
• :module_circuit_breaker_count - Module crash limit (default: 50)
• :module_circuit_breaker_window_ms - Module circuit breaker window (default: 300_000)
• :module_circuit_breaker_cooldown_ms - Module circuit breaker cooldown (default: 600_000)
• :backend - Optional storage backend spec: {BackendModule, opts} or a pre-initialized %DurableServer.StorageBackend{}
• :object_store - Legacy object storage config (used when :backend is not set)
• :init_info - Map of user-defined data passed to each server's init/2 callback (default: %{})
• :placement_region - Optional region label used for placement timeout tuning.
• :placement_erpc_timeout_same_region_ms - Same-region remote placement ERPC timeout in ms. Default: 3000
• :placement_erpc_timeout_cross_region_ms - Cross-region remote placement ERPC timeout in ms. Default: 8000
• :max_singleflight_waiters_per_key_module - Per {key, module} cap for concurrent ensure_started_child/3 waiters. Calls beyond the cap fail fast with {:error, :singleflight_overloaded}. Default: 50_000. Set to nil to disable.

Streams server info for all servers in storage.

Returns a Stream that yields info maps for each server found in storage. Failed fetches are filtered out. Excludes internal node metadata objects.

This is useful for admin dashboards that need to iterate over all servers without loading everything into memory at once.

Examples

# Stream all servers
DurableServer.Supervisor.stream_all_server_info(MySup)
|> Enum.to_list()

# Stream only permanently crashed servers
DurableServer.Supervisor.stream_all_server_info(MySup)
|> Stream.filter(fn info -> info.status == :permanently_crashed end)
|> Enum.to_list()

Terminates a DurableServer child process AND deletes its object storage.

This permanently removes the server and all its persisted state. The operation:

1. Finds the running process (if any) by PID or key
2. Terminates the process gracefully (allowing final state sync)
3. Deletes the object storage data

Parameters

• supervisor - The DurableServer.Supervisor name
• pid_or_key - Either a PID of the running process or the key string

Returns

• :ok - Successfully terminated process and deleted storage
• {:error, reason} - Failed to delete (process may still be terminated)

Examples

# Delete by PID
:ok = DurableServer.Supervisor.terminate_and_delete_child(MySup, pid)

# Delete by key
:ok = DurableServer.Supervisor.terminate_and_delete_child(MySup, "user_123")

Terminates a specific DurableServer child process gracefully.

The child will be given time to sync its state before termination.

Terminates a specific DurableServer child process gracefully, and unmark it for permanent restart.

Useful to stop a previously permanently started durable server so that it won't be considered a candidated for permanent restart in the future.

The child will be given time to sync its state before termination.

Blocks until the supervisor is ready or timeout expires.

Returns :ok if ready, {:error, :timeout} if timeout expires.

This is intended to be called via RPC on a node that may still be booting. The remote node will block until its supervisor is ready, preventing ETS errors from concurrent access during startup.

Options

• :timeout - Maximum time to wait in milliseconds (default: 5000)
• :poll_interval - How often to check readiness in milliseconds (default: 100)

Lists all currently running DurableServer child processes on this node's supervisor.