DurableServer.Terminator (durable_server v0.1.5)

Copy Markdown

Terminator GenServer that coordinates graceful shutdown of DurableServer processes.

The Terminator is placed at the bottom of the DurableServer.Supervisor supervision tree and traps exits. When the supervisor is shutting down, the Terminator's terminate/2 callback is called, which:

  1. Sends sync_and_stop messages to DurableServer children (with limited concurrency)
  2. Monitors each child process for DOWN messages
  3. Enforces both per-child and overall deadlines for children to sync and terminate
  4. Returns to continue the shutdown process

This ensures that DurableServer processes have an opportunity to persist their state before the supervisor tree is torn down, while preventing indefinite hangs during shutdown.

Configuration

The Terminator uses the same configuration as its parent DurableServer.Supervisor:

  • :graceful_shutdown_timeout_ms - Maximum time for each child to persist and stop after its shutdown starts (default: 30_000ms)
  • :graceful_shutdown_total_timeout_ms - Maximum time for discovery shutdown and all child shutdown batches together (default: 55_000ms)
  • :graceful_shutdown_concurrency - Maximum concurrent shutdown operations (default: 50, should match Finch pool size to avoid connection exhaustion)

Graceful Shutdown Protocol

  1. Supervisor begins shutdown process
  2. Terminator's terminate/2 is called with reason and state
  3. Terminator uses Task.async_stream with limited concurrency to: a. Send {:durable, {:sync_and_stop, reason}} to each DurableServer b. Wait for each child up to its own timeout and within the overall shutdown deadline
  4. Each DurableServer calls sync_state/1 then stops normally
  5. After all children stop or timeout is reached, terminate/2 returns
  6. Supervisor continues shutdown process

The concurrency limit prevents overwhelming the Finch connection pool when many DurableServers try to persist their state simultaneously during shutdown.

The graceful shutdown only applies to normal shutdown scenarios (e.g., application stop, supervisor shutdown). For abnormal termination (crashes, kills), the normal supervision tree behavior applies.

Summary

Functions

Returns a specification to start this module under a supervisor.

Callback implementation for GenServer.init/1.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

init(arg)

Callback implementation for GenServer.init/1.

start_link(opts)