ADR-000006: RPC as a Correlated Protocol — partisan_erpc as the Primary Surface, rex Kept for What Defines It

• Status: ACCEPTED
• Decision: Stop routing call through the partisan_rpc_backend server. RPC becomes an explicit correlated request/response over the Partisan transport: each request carries its own correlation reference and is executed by a worker process spawned per request, replying directly to the caller. partisan_erpc is re-based on that protocol and becomes the primary surface — the erpc API in full, including the OTP 25+ request-id collections the vendored snapshot had silently drifted behind — while partisan_rpc is re-based as a thin shim over it, mirroring the rpc-on-erpc split OTP itself has used since OTP 23. partisan_rpc_backend is kept permanently as the counterpart of OTP's rex, demoted from "all RPC" to the four operations whose semantics require a server. Inbound concurrency is bounded, and every surface gains per-call transport options so a caller can put RPC on a channel of its own.

Context

partisan_rpc_backend is a single gen_server registered per node. Before this record, every inbound partisan_rpc:call/4,5 arrived as {call, M, F, A, Timeout, {origin, Caller}} and was applied inline in handle_info/2. Three consequences followed from that one line:

• Every RPC arriving at a node was serialised through one process. A single slow, blocking or hung M:F(A) stalled every unrelated RPC behind it in that server's mailbox. This is a throughput property, not a fairness nicety: the node's entire RPC capacity was one process deep.
• Replies were uncorrelated. The framing carries no request id, so a reply that arrives after its caller has timed out can be consumed by an unrelated later call in the same process. A caller with several requests outstanding could not reliably tell them apart at all.
• A crashing RPC was a supervision event. Applying user code in the callback put an arbitrary function's failure modes in the same process as the server.

Applying user code inline is not itself the defect. block_call's documented semantic is "execute on the server itself, serialised with every other block_call, blocking it" — that is what distinguishes it from call. Inline application is therefore correct for block_call and wrong for call, and the defect is that call was routed to the block_call machinery. The fix is a routing change, not a rewrite.

A second, independent problem sat alongside it. partisan_erpc was a vendored copy of an OTP 23/24-era erpc that still reached peers with the auto-imported spawn_request/5 BIF and received results as the exit reason of a distributed monitor. Both are distribution-protocol mechanisms: they ride disterl, not the Partisan transport. So the module that modern OTP code actually calls — rpc has been informally deprecated in favour of erpc for years, and OTP's own rpc source steers readers to erpc inline — did not work over Partisan at all, and no test had ever loaded it. It had also drifted eight functions behind upstream: the entire OTP 25+ request-id collection API was missing, so idiomatic modern fan-out code failed with undef.

The decision

1. A correlated request/response protocol

A request is {?ERPC_REQUEST, Res, Origin, M, F, A} where Res is a correlation reference that is also the reply address. The reply is {?ERPC_REPLY, Res, Payload} sent directly from the worker to Origin; the server is not in the reply path.

Res is an encoded process alias (erlang:alias([explicit_unalias])). That choice does two jobs with one term: it correlates the reply, and it gives the runtime a way to drop a late reply — abandoning a request deactivates the alias, so a reply arriving afterwards is discarded rather than accumulating in an abandoned caller's mailbox. Upstream erpc gets the same guarantee from demonitor(_, [flush]), because there the result travels as a monitor DOWN.

A monitor is still taken, but only to detect that the target became unreachable — deliberately not as the value channel. In Partisan every remote DOWN is relayed through the monitoring node's partisan_monitor server, so carrying return values on exit reasons would funnel every RPC result on a node through that single process, reintroducing the bottleneck this record removes in a less obvious place.

2. Worker per request, never inline

The receiver correlates and spawns; it never applies user code in its callback. The worker traps and translates exceptions at the boundary, so a crashing RPC cannot reach the server. The reply payload deliberately reuses the same tuple shape partisan_erpc:execute_call/4 builds — {Res, return | throw | exit, V} or {Res, error, E, Stack} — so result/4 translates it into the exception semantics erpc specifies with no modification.

3. Vendor the pure logic; write the transport natively

The line is drawn between erpc's pure logic and its transport:

• Vendored verbatim and kept diffable: error translation, trim_stack/4, is_arg_error/4, result/4, response matching. This is the part OTP-compat actually depends on.
• Written natively: everything that reaches a peer. spawn_request/5 plus exit-reason delivery is pure disterl mechanics and none of it transfers.

-compile({no_auto_import, [spawn_request/5, spawn_request_abandon/1]}) makes any missed call site a compile error rather than a silent fallback to disterl. This is the highest-value safety measure in the change and cost one line.

OTP validates this exact split independently: rex itself calls erpc:execute_call/3, erpc:execute_cast/3, erpc:is_arg_error/4 and erpc:trim_stack/4. OTP draws the vendor-the-pure-logic line in the same place.

4. partisan_rpc_backend is rex, permanently

It keeps exactly the operations whose semantics require a server, which are exactly the ones OTP still routes through rex:

Its inline-apply behaviour is retained deliberately for block_call and documented as intentional. partisan_rpc is not deprecated: OTP has not formally deprecated rpc, and it remains the target of the rpc => partisan_rpc rewrite that existing code depends on. It is documented as the legacy surface, with new code steered to partisan_erpc — mirroring the guidance OTP carries in its own source rather than inventing a stricter policy.

5. Inbound concurrency is bounded

Spawning per request without a bound converts a throughput bug into a denial-of-service vector. OTP tolerates the unbounded form because the distribution buffer applies backpressure; Partisan has no equivalent, so the bound is enforced at the receiver: a per-node in-flight cap (rpc_max_concurrency, registered in partisan_config's defaults), surfaced via telemetry, releasing each slot on the worker's DOWN.

Over-cap requests are rejected outright rather than queued. A queue with no credit scheme is an unbounded mailbox with extra steps; rejection is a bound. The rejection is shaped so the caller raises error({partisan_erpc, overloaded}) and partisan_rpc translates it to {badrpc, {'EXIT', overloaded}} through the same path as any other failure.

6. Per-call transport options on every surface

Upstream erpc has no notion of channels, so on its API every request would ride the globally configured forward_options with no way to override it. Since RPC traffic is exactly the kind that benefits from a channel of its own — a slow bulk RPC should not sit behind or ahead of membership gossip — every surface accepts forward_opts():

• call/5 and multicall/5 take forward_opts() in place of a bare timeout, which is read out of the map;
• send_request/5, send_request/7, cast/5 and multicast/5 are additional arities that do not exist upstream, because those shapes have no argument to widen.

Export parity with erpc is unaffected: the module need only be a superset, which a test now asserts on every OTP version in the CI matrix.

Per-call options win over the global configuration; the global value fills in only what the caller omitted. This is stated explicitly because the previous code had the precedence inverted — it read partisan_config:get(forward_options, CallerOpts), which makes the caller's options a mere fallback, so every per-call channel and partition_key was silently discarded the moment anything set the global. One resolver (partisan_rpc:forward_opts/1) now serves both RPC surfaces, so the precedence cannot drift between them.

7. Rolling upgrade

The pre-6.0.0 framing is retained as a receiver only, for the whole of the 6.x series, and removed in 7.0.0:

• a 5.x caller reaching a 6.x node uses the legacy clause and gets the answer it always did — now with the concurrency bound applied, since a cap any un-upgraded caller can walk past is not a cap;
• a 6.x caller reaching a 5.x node cannot use the correlated protocol, because a 5.x server has no clause for it and discards the message; the caller observes a timeout. Every node must be upgraded before partisan_erpc is relied on between them.

That asymmetry is the point: the legacy clause makes a partially-upgraded cluster work for the callers that already exist, rather than making both directions work for callers that do not yet.

Rationale

Why not simply spawn inside the existing framing? It captures most of the throughput win in about fifteen lines, and fixes nothing else. The framing has no request id, so late replies stay mis-attributable and a caller still cannot hold several requests outstanding. Correlation is what the collection API needs, and it cannot be retrofitted onto a protocol with nowhere to put an id.

Why is partisan_erpc the primary surface rather than the compatibility layer? Because that is what OTP code calls. rpc is the legacy surface in OTP too; erpc is where new code goes. Treating partisan_erpc as an afterthought would have left the main path broken while polishing the side path.

Why keep a server at all? Because three operations are defined in terms of one. block_call means "serialised on the server". sbcast must report whether a name was registered on each node. abcast and eval_everywhere are node-wide fan-outs with no per-request reply. Deleting the server would mean reimplementing it under another name.

Alternatives considered

• Delete partisan_rpc_backend entirely. Rejected: it is rex, and OTP keeps rex for the same four operations. The module is not the problem; the routing was.
• Deprecate partisan_rpc. Rejected: OTP has not deprecated rpc, and the rpc => partisan_rpc rewrite is what existing code depends on. Documenting it as legacy achieves the steer without breaking anything.
• Add erpc => partisan_erpc to the compile-time rewrite map. Rejected: the map applies only to OTP-derived modules and generated test suites, never to user code, which the user-facing transform touches only for !. The entry cannot help user code, and it breaks the generated suites, which use erpc:call/4 as their own scaffolding against peer nodes that never start the Partisan application. Code wanting Partisan-transported erpc calls partisan_erpc directly.
• Carry results on monitor exit reasons, as upstream does. Rejected: in Partisan every remote DOWN is relayed through one partisan_monitor process per node, so this would replace a single-process bottleneck with a less-visible one.
• Queue over-cap requests instead of rejecting them. Rejected until real backpressure exists. A queue with no credit scheme is an unbounded mailbox.

Consequences

• A node's RPC capacity is no longer one process deep. A slow RPC delays only itself.
• The head-of-line result is measured. With a 60-second RPC held open on the target for a whole run, the fast callers' p99 sits between the unobstructed one-caller and eight-caller figures — the slow call is invisible in the tail. Throughput also scales with callers rather than flattening. Both are recorded in bench/BASELINE.md. Every other claim here is structural: work that was serialised through one process no longer is.
• A crashing RPC is no longer a supervision event, and late replies are dropped rather than mis-attributed on both the single and collection surfaces.
• partisan_erpc is exercised for the first time. It had never been loaded by a test; it now has local coverage plus a cross-node case, all with connect_disterl disabled so nothing can pass by falling back to distribution.
• An un-upgraded peer's RPCs are bounded too, which is a behaviour change for 5.x callers against a saturated 6.x node: they receive {badrpc, overloaded} where previously the node would have kept spawning.
• The surface is wider than erpc's by four arities and two overloads — a maintenance cost paid deliberately, since without it the primary RPC API cannot select a channel.

Related records

• ADR-000002 — established the pluggable-behaviour pattern and, more relevantly here, the practice of extracting a seam only where the existing implementation already justifies it. The rex/erpc split in §4 is the same judgement applied to a boundary OTP had already drawn.
• ADR-000003 — removed a different single-process fan-out bottleneck (the gen_event membership bus). The reasoning is the same shape: a process that serialises work with no semantic reason to serialise it.

References

• OTP kernel/src/rpc.erl — the rpc-on-erpc arrangement mirrored in §4, including the ?RPCIFY macro and rpcify_exception/2 vendored verbatim, and the inline guidance steering readers to erpc.
• OTP kernel/src/erpc.erl — the pure logic vendored in §3, and the API surface §6 must remain a superset of.
• erlang:alias/1 and erlang:unalias/1 — one-shot reply addresses, the basis for the correlation reference in §1.