Sagents.RegistryWatcher (Sagents v0.12.0)
Copy MarkdownPropagates a registry restart into Sagents.Supervisor's :rest_for_one
restart chain.
What it protects
Every AgentSupervisor and AgentServer registers its :via name in
Sagents.Registry exactly once, when it starts, and nothing re-registers it
afterwards. A registry that loses its contents therefore leaves those
processes running but invisible to every lookup. Lookups answer "nothing is
running", the next request responds by starting an agent, and the conversation
ends up with two AgentServers holding and persisting state for it, with
nothing reporting the conflict.
Sagents.Supervisor uses :rest_for_one so a registry failure restarts
everything listed after it, which re-establishes those registrations. The
watcher is what connects a registry failure to that chain, because on both
backends the process that holds the registrations sits one level below where
the strategy can see it.
What each backend actually looks like
Under :horde, Horde.Registry.start_link/3 starts a supervisor, and the
process registered under the name is its child:
Sagents.Supervisor
└── Sagents.Horde.RegistryImpl <- the child Sagents.Supervisor supervises (a supervisor)
├── Horde.RegistryImpl <- registered as Sagents.Registry, owns the ETS tables
└── Sagents.Registry.CrdtUnder :local, Registry.start_link/1 registers its Registry.Supervisor
under the given name, and the ETS tables belong to a Registry.Partition
child:
Sagents.Supervisor
└── Sagents.Registry <- the child Sagents.Supervisor supervises
└── Sagents.Registry.PIDPartition0 <- owns the ETS tablesThe shapes differ, and so does which process Sagents.Supervisor can see fail
on its own. What does not differ is that in both cases a backend supervisor
can replace the table-owning process with a fresh, empty one without ever
failing a child of Sagents.Supervisor:
:horderestartsHorde.RegistryImplunderSagents.Horde.RegistryImpl.:localrestartsRegistry.PartitionunderSagents.Registry.
A :local registry has a second failure mode, Registry.Supervisor itself
dying, which is a failed child and does reach :rest_for_one unaided. That
one is already handled, and Sagents.LocalRegistry is what makes its restart
survivable. It is the partition case that needs watching.
What is watched
This watcher monitors Sagents.ProcessRegistry.watched_name/0, the process
that owns the tables rather than the process that holds the registry's name.
On :horde those are the same process; on :local they are not.
It stops when that process dies. Listed immediately after the registry in
Sagents.Supervisor, its stop is a child failure the :rest_for_one strategy
does act on, taking the agent and filesystem supervisors down with it. Agents
stop, and the next request re-creates them from persisted state with fresh
registrations.
Watching the partition covers the Registry.Supervisor case too, since the
partition dies whenever its supervisor does.
When it fires, and when it does not
Only the death of the table-owning process fires it. In particular it does
not fire on an orderly shutdown. The watcher is listed after the registry
so that it can monitor it, and OTP terminates children in reverse order, so
that same placement means the watcher is already gone by the time the registry
stops. A draining node therefore never looks like a registry failure. Draining
is handled by Sagents.ready?/0 and the guarded lookups instead, not here.
The trade being made
A registry crash costs running agents rather than leaving duplicates behind. Agent state is durable, so a restart is recoverable and mostly invisible to a user. A silent duplicate is neither: two AgentServers hold and persist state for one conversation, and nothing reports the conflict.
On a healthy multi-node cluster this is more disruptive than doing nothing would have been. Horde replicates registrations through the same CRDT as membership, so a peer repairs a restarted registry within a few hundred milliseconds, pointing at the very processes this watcher just took down.
It fires anyway, because nothing on this node can tell in time whether a repair is coming. A single-node deployment has no peer at all. A partitioned or lagging cluster is indistinguishable from a healthy one from here, right up until the window where guessing wrong leaves agents alive and unregistered. The cost of over-reacting is a restart from durable state; the cost of under-reacting is silent corruption. The asymmetry is the whole argument.
Both halves are pinned in test/sagents/horde/rolling_deploy_test.exs: a peer
repairing a crashed node's registry, and an agent on the crashed node coming
down regardless.
Started automatically by Sagents.Supervisor. You do not start it yourself.
Summary
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
Start the watcher.
Options
:name- registered name, defaults to this module.nilstarts it unnamed, which tests use to run more than one at a time.:registry_name- the process name to watch, defaults toSagents.ProcessRegistry.watched_name/0. Overridden only in tests.