ADR-000007: Backpressure — Stop Defeating TCP at the Last Hop

• Status: PROPOSED
• Decision: Do not build an application-layer credit protocol. TCP already provides byte-denominated, receiver-driven flow control end to end, and Partisan preserves it everywhere except one hop: the receiving connection process hands each message to an unbounded Erlang mailbox and immediately re-arms {active, once}. Close that hop by gating the re-arm on the destination's readiness, and TCP propagates backpressure the rest of the way by itself. Per-flow isolation comes from separate sockets (parallelism => N plus a fixed partition_key), not from per-flow credit. "Streams" as a distinct construct is reduced to a documented pattern, with at most a thin convenience wrapper.

Context

Partisan bounds two local resources — inbound RPC concurrency (ADR-000006) and connection mailbox depth (connection_high_watermark) — but is assumed to have no end-to-end backpressure. That assumption is wrong, and tracing the actual path shows why:

Sockets are {active, once} on both sides (partisan_peer_service_client:468, partisan_peer_service_server:215), so the receiving process controls its own read rate and TCP's window tracks it. Every hop is covered except the last: receive_message/3 calls partisan_peer_service_manager:deliver/2, an erlang:send into an unbounded mailbox that always succeeds instantly, and the process then re-arms unconditionally.

One hop, on one node, is the whole gap. It is a local problem, not a distributed one.

The decision

1. Gate the re-arm on the destination's readiness

After delivering an inbound message, the connection process checks the destination's mailbox depth and, when it is over a watermark, defers re-arming the socket until it drains:

%% partisan_peer_service_server:handle_inbound/2, and the client equivalent
case receive_ready(ServerRef) of
    true  -> reset_socket_opts(State);          % today's behaviour
    false -> defer_rearm(State)                 % re-check, then re-arm
end

With {active, once} at most one message is in flight, so not re-arming stops the flow immediately. Data accumulates in the kernel receive buffer, the window closes, the peer's gen_tcp:send/2 blocks, its connection mailbox fills, and connection_high_watermark refuses the sending application with {error, overloaded}.

That is the complete chain. It needs no protocol, no wire message, no credit accounting, and no obligation on the receiving application.

Readiness is observed with process_info(Pid, message_queue_len) — the same BIF partisan_peer_connections:admit/2 already uses on the send side. infinity (the default) means never defer, so this is opt-in exactly as connection_high_watermark is.

2. Isolation comes from sockets, not from credit

Deferring the re-arm stalls the whole socket. Every destination sharing it, and the heartbeat traffic with them, stops too.

That is not a defect of the mechanism — it is what backpressure at a shared transport means — but it decides the shape of everything else: a flow that needs independent backpressure needs its own socket. Partisan already has the means, and it needs no new concept:

• a channel gives a set of connections with their own settings;
• parallelism => N gives N independent sockets on that channel;
• a fixed partition_key pins a flow to one of them, deterministically and for as long as the caller keeps using it.

A "flow" is therefore a {channel, partition_key} pair used consistently. It already has FIFO (one connection, one process, one ordered mailbox), already has sticky routing, and — with §1 — now has backpressure. parallelism is the ceiling on independently-backpressured flows to a peer, which is the honest cost of not multiplexing.

3. No wire-level multiplexing

Framing several flows onto one socket would reintroduce head-of-line blocking between them, which is the problem QUIC exists to solve and which Partisan has no reason to import.

Ruling it out is what makes §1 sufficient: with one flow per socket, TCP's window is already per-flow. A credit protocol only adds something when flows share a socket and need independent backpressure — precisely the case this rules out.

4. partisan_stream is not required, and probably should not exist

A stream API of the usual shape — open/2, send/3, consumed/2, cancel/1 — earns very little once §1 and §2 are in place:

What remains is naming and ergonomics. If a wrapper is added it should be thin — a handle that carries a {channel, partition_key} pair and validates it against the channel's settings at open (refusing monotonic, whose drop-on-backlog policy contradicts FIFO). It must not acquire a credit window, a wire representation, or a lifecycle of its own.

5. RPC needs no stream

rpc_max_concurrency is a receiver-side, per-node cap that is blind to the caller, so one caller can consume the allowance and starve the rest. The remedy under this design is to give an RPC-heavy caller its own channel — which every partisan_erpc surface already accepts per call — and let §1 bound it.

Alternatives considered

Consequences

• Backpressure becomes end-to-end for the first time, without a line of protocol.
• Deferring the re-arm stalls the socket, including other destinations on it and the heartbeat. A long stall can therefore trip the peer's ping timeout and drop the connection. This is the sharpest consequence and it needs a bound: a deferral limit past which the message is dropped or the connection is allowed to fail, rather than an unbounded stall that manifests as a mysterious disconnect.
• A destination fed by several connections is not fully protected by stalling one of them. Covering that needs a destination→connections mapping on the receiving node — a local problem, but a real one, and out of scope here.
• parallelism caps independently-backpressured flows to a peer. The cost of refusing to multiplex.
• Mailbox depth is a proxy, not a measure of readiness. A process that keeps a deliberately deep mailbox would be throttled when it is not actually behind. Opt-in per channel, off by default.
• One process_info/2 per inbound message, on the same order as the send-side check. It should be measured, not assumed — bench/ exists for that, and p2p_roundtrip resolved the equivalent send-side cost.

Related records

• ADR-000006 — bounded inbound RPC concurrency per node; §5 explains why it needs no stream construct to get per-caller isolation.

References

• TCP receive window — the flow-control layer this record defers to rather than duplicating. Reopens when the application reads, not when the kernel buffers, which is the property the whole design rests on.
• QUIC (RFC 9000) — why multiplexing over an ordered byte stream reintroduces head-of-line blocking (§3).
• HTTP/2 (RFC 9113) WINDOW_UPDATE — the two-level credit scheme this record originally copied and now rejects: it exists because HTTP/2 multiplexes, and Partisan does not.
• Reactive Streams (Subscription.request/1) — explicit consumer demand, appropriate in-process where there is no transport underneath to inherit backpressure from.
• RabbitMQ credit_flow (rabbit_common) — credit between Erlang processes, for the same unbounded-mailbox reason. Blocks the producer, which §1 avoids by letting TCP do the blocking at the connection process.
• ZeroMQ high-water mark — the analogue of connection_high_watermark: a local per-socket bound that says nothing about the receiver.