ADR-000003: Retire the partisan_peer_service_events Bus — Membership as a Snapshot with Async Push

View Source
  • Status: ACCEPTED
  • Decision: Retire the partisan_peer_service_events gen_event membership bus. Every peer-service manager publishes membership changes to the lock-free partisan_membership snapshot (ADR-000001) and additionally delivers a non-blocking asynchronous notification to registered subscribers. Consumers observe membership by pull (read the snapshot) or push (subscribe a pid and receive {partisan_membership, Members} messages). The synchronous gen_event fan-out — which blocked the oracle on every subscriber's callback and ran all callbacks serially in one process — is removed. partisan_peer_service:add_sup_callback/1 is retained as a thin, deprecated compatibility shim over the push API.

Context

partisan_peer_service_events is a gen_event manager carrying exactly one event, {update, Members}. Every peer-service manager publishes to it on a membership change via sync_notify, which blocks the publishing manager (the oracle) until every subscriber's callback has run, and runs all callbacks serially in the single gen_event process. Its only consumers today are external applications subscribing through partisan_peer_service:add_sup_callback/1; ADR-000001 already moved Partisan's own consumers (the broadcast groups) onto the lock-free partisan_membership snapshot.

That migration is incomplete in a way that is currently a defect: the snapshot is written by the default (pluggable) manager only. The hyparview, static and client_server managers still publish membership solely to the gen_event bus. Because broadcast groups no longer read that bus, broadcast-tree membership does not propagate under those three managers — their groups observe an empty member set. Making every manager feed the snapshot closes that gap and removes the last reason the bus exists.

The decision

  1. Every manager feeds the snapshot. Each partisan_peer_service_events: update(X) call site (in the pluggable, hyparview, static and client_server managers) is replaced by writing membership to partisan_membership as node specs. This makes the snapshot authoritative under every manager and fixes the propagation gap above.

  2. Asynchronous push. partisan_membership gains subscribe/0,1 and unsubscribe/0,1. On a membership change the oracle, after writing the snapshot, delivers {partisan_membership, Members} to each subscribed pid with a plain asynchronous send. This never blocks the oracle, and each subscriber processes the change in its own process — strictly better than the gen_event model, where one slow or crashing callback stalled the shared dispatch and the publisher.

  3. Pull remains the default. Consumers that do not need immediate reaction read partisan_membership:members/0 / version/0 (lock-free) — the mode Partisan's own broadcast groups already use.

  4. Compatibility shim. partisan_peer_service:add_sup_callback/1 is retained but deprecated: it registers a supervised relay process that subscribes to the push feed and invokes the given fun. Existing callers keep working through a deprecation window; the semantics change is documented below.

  5. Delete the bus. With no publishers or first-class subscribers left, partisan_peer_service_events is removed.

How existing subscribers operate now

A consumer today calls partisan_peer_service:add_sup_callback(fun(Members) -> ... end) and its fun runs, synchronously, inside the shared gen_event process. After this change:

  • No code change required immediately. The same call keeps working via the compatibility shim — but with two behavioural differences a correct subscriber must respect:
    • Delivery is asynchronous. The manager no longer waits for the callback; do not assume membership processing has completed when a join/leave returns on the manager.
    • The callback runs in its own process. A slow or crashing subscriber no longer stalls the oracle or other subscribers; on a crash the shim re-subscribes.
  • Preferred going forward, pick one:
  • What is gone: the synchronous, in-manager callback guarantee. Nothing relied on it (membership subscribers are observers, not part of the membership transaction), which is why it can be dropped.

Rationale

  • It removes a real coupling, not a hypothetical one. sync_notify puts arbitrary subscriber code on the oracle's membership-change path; a single slow callback delays every manager's join/leave handling. Async push severs that.
  • It finishes a migration that is otherwise a bug. The snapshot must be fed by every manager for broadcast to work under non-default managers; once it is, the bus has no publisher worth keeping.
  • Own-process handling beats shared dispatch. gen_event runs all handlers in one process; a crash or a long callback affects the others. Delivering a message to each subscriber's own process removes that shared-fate.
  • Non-breaking by default. Keeping add_sup_callback/1 as a shim means no downstream application must change on day one; the deprecation gives a window.

Alternatives considered

  • Keep gen_event, switch sync_notify → notify (async). A one-line fix that unblocks the oracle without touching the API. Rejected as the end state (it keeps the single serial-dispatch process the project treats as deprecated, and it does not fix the snapshot-propagation gap) but noted as the correct interim step if the full retirement is staged.
  • Pull-only; delete the push API. Rejected: some subscribers legitimately need immediate reaction to membership changes; forcing everyone to poll regresses them and offers no upside over a fire-and-forget push.
  • A dedicated notifier process fanning out. Rejected as unnecessary indirection: the oracle already runs on the change and an async send per subscriber is cheap and non-blocking; a separate process reintroduces a single serial fan-out point without benefit at this scale.

Consequences

  • Public API: partisan_membership gains subscribe/unsubscribe and the {partisan_membership, Members} message contract. add_sup_callback/1 becomes a documented, deprecated shim. partisan_peer_service_events is removed.
  • Every manager now maintains the snapshot — closing the non-default-manager broadcast gap and making partisan_membership the single source of membership truth regardless of manager.
  • Deprecation window: add_sup_callback/1 is now a compatibility shim over the push feed; it stays for consumers that have not yet migrated to the partisan_membership API.
  • What this forecloses: membership is no longer observed through a blocking, shared-dispatch bus; adding a new consumer never touches the oracle's critical path.
  • ADR-000001 — introduced partisan_membership and moved broadcast groups off the gen_event bus; that move now covers every manager, and the bus is retired.

References

  • Erlang/OTP gen_event — sync_notify/2 (synchronous, shared-process dispatch) vs notify/2 (asynchronous).