DpExchange.Coinbase.Feed (DpExchangeCoinbase v0.3.8)

Copy Markdown View Source

This venue's subscription lifecycle — internal. The facade's subscribe/2, unsubscribe/2, update_symbols/2 and coverage/1 are served from here.

What a consumer can and cannot learn

A consumer learns what is arriving, through coverage/1. It cannot learn how: this module owns the sockets, the sharding and the pacing, and none of that reaches the facade.

Coverage is observed, never intended

A symbol enters the coverage map when a payload for it arrives, never when it is subscribed. That distinction is the strongest guarantee in the contract and it exists because a venue once reported 325 symbols subscribed and confirmed while 174 were delivering. Reporting the subscription would have said 325.

A symbol that has been subscribed and has delivered nothing is simply absent, which the facade documents as :not_covered.

Sharded — this used to run on one connection, and that stopped being true

This venue's whole scope used to run on a single socket, and that was measured: once its self-killing heartbeat was fixed it subscribed 401 of 401 pairs on one connection, and sharding it anyway opened fourteen connections for no gain.

That stopped being true on 2026-08-26. Coinbase started answering a level2 subscribe over its per-session limit with "too many L2 streams requested in a single session", closing the socket — a total data gap, not degraded coverage: measured 2026-08-27 against a real ~400-symbol universe, 355 of 405 pairs went stale and 1,480 refusals were logged in one window. The measurement about one socket being enough was honest when it was taken; it stopped being true the moment the venue's own limit did.

@pairs_per_socket is 100 — carried over from the reference fix this replaces rather than re-derived, the number came from a real production incident, not this package's own probing. It sizes ticker's own sockets, and only ticker's: the next section is why level2 stopped sharing it.

level2 gets its own, smaller sockets — the two channels stopped sharing a shard

This stopped being true again, on a different axis, measured against DpCryptoManagement's own 406-symbol production universe (issue #22 continuing, not reopening — ticker's starvation above is a separate, already-fixed incident in the same file). At @pairs_per_socket (100), shards/1 splits 406 symbols into [100, 100, 100, 100, 6] — four full shards and a six-symbol remainder. Every level2 subscribe on the four full shards was refused with "too many L2 streams requested in a single session"; the six-symbol shard's was not. 5,099 refusals were logged across sixteen otherwise-healthy boots, and coverage_by_kind/1 answered order_book: 6 throughout — exactly the tail shard's own symbol count, which is the arithmetic that pins the cause on the shard size: ticker has no such ceiling and answered quotes: 406 on the same boots, so nothing else about those boots — the connection, the alias fix, the resubscribe cadence — explains a number that lines up precisely with one shard's population and no other.

No Coinbase document states the real ceiling. Re-read 2026-09-06 specifically looking for a per-session level2 stream count: the Advanced Trade channels reference, the connection overview, and the Advanced Trade rate-limits page ("WebSocket connections and unauthenticated messages are each limited to 8 per second per IP" — a connect-rate ceiling, not a per-session subscription count) all say nothing about how many products one session may carry on level2. The older Exchange product's own separate rate-limits page states a different number entirely — 10 subscriptions per product per channel, meaning duplicate subscriptions to the same product, not the count of distinct products — and this package speaks Advanced Trade (Socket's @endpoint), not Exchange, so that number would not transfer even if it were on point. The prior investigation that first shipped @pairs_per_socket found the docs silent on this; they still are. This does not stop being true just because the number below is now measured — it is measured behaviour, never a documented figure.

This package still cannot narrow it by probing the venue itself. level2 is on @authenticated_channels, and this repo's own testing strategy draws the line at exactly that boundary: tier 2 (live public endpoints, by hand) is fair game, tier 3 (authenticated) "needs credentials this repo must never hold." Bisecting a real session's level2 subscription size against the live venue with a real credential is precisely tier 3, and remains a line this repo does not cross for any endpoint, level2 included. What changed is that a consumer — who holds the credential this repo structurally cannot — did the bisection, in production, and reported the result back as tier-3 evidence. That is a different thing from this package probing itself, and the distinction is why the number below is attributed to them rather than to any measurement this repo ran.

@level2_pairs_per_socket is 30. Measured 2026-09-06 by DpCryptoManagement (issue #22) against the live venue with real credentials, not by this package and not from documentation:

requested (n)6122530313550100
verdictacceptedacceptedacceptedacceptedREFUSEDREFUSEDREFUSEDREFUSED

Method, because a measured number is only worth its method. One fresh Socket.start_link/1 per attempt, never reused — cumulative session state was explicitly the thing under test, which is exactly the question this file's own "cumulative vs. concurrent" section below could not otherwise answer. 3s to establish the connection; subscribe(socket, "level2", symbols, creds); then 20 seconds draining the mailbox continuously while tallying. Every symbol came from the consumer's real, live 406-symbol scope, so silence means refusal rather than an idle book. Verdict refused on a rate_limited notice, accepted on OrderBook/OrderBookDelta payloads. The boundary was confirmed by interleaving two full runs back to back: n=30 accepted, n=31 REFUSED, n=30 accepted, n=31 REFUSED. A contamination check ran too — immediately after a run where n=3134 refused, n=6 and n=30 were re-run and both accepted, which is what rules out the refusals being saturation from the probing itself rather than a genuine per-session ceiling.

Same evidentiary standard as the 6 this replaces, just with the boundary actually located rather than merely bounded far below a known failure: 30 is the largest value with positive evidence of acceptance, 31 the smallest with positive evidence of refusal. Nothing between them needed to be guessed this time, unlike 6 versus the old 100.

The two channels no longer share a shard slice. ticker keeps @pairs_per_socket (100) — it has no known ceiling, and shrinking it to match level2 would multiply connection count for the channel that was never the problem, for no benefit ticker needs. level2 is chunked separately, at @level2_pairs_per_socket, and every chunk of either channel opens its own dedicated, single-channel socket: there is no longer a shard that carries both. For the 406-symbol universe above that is 5 ticker sockets (unchanged) plus 14 level2 sockets (ceil(406 / 30)) — 19 total against 73 under the 6-sized grouping this replaces, and 5 before level2 needed its own grouping at all. Socket's own moduledoc records why more sockets than the original single-socket design is affordable now in a way it would not have been before 2026-09-06: removing in-package level2 book maintenance cut per-frame decode cost roughly tenfold (65–110 ms to 6.6 ms, at the BTC-USD book size DpCryptoManagement measured live) — decode cost, not socket count, was the resource actually in short supply.

Connects are staggered across both groups on one sequence, ticker first. Every new socket this module opens — whichever channel it carries — takes the next tick of shard_spacing_ms (1_000ms by default — see that option's own section below), in an order that places every touched ticker shard ahead of every touched level2 shard. This is what keeps ticker's own boot-time coverage exactly as fast as the section above describes: a 406-symbol subscribe still resolves its synchronous reply and its remaining ticker shards inside the same handful of seconds as before this fix, with level2's 14 shards ramping in behind them. For that universe (19 shards touched together in one reshard/1 call — 5 ticker plus 14 level2) the last shard's tick lands (shard_count - 1) * shard_spacing_ms after boot — the identical span next_resubscribe_delay/1 computes below for its own overlap floor — 18 * shard_spacing_ms: 18 seconds at the current default, materially slower than ticker's own coverage and an accepted, stated cost, now roughly a twentieth of what the 6-sized grouping cost (about six minutes): order_book coverage was permanently 6 / 406 before level2 got its own grouping at all, climbing by 5,099 refusals and counting; ramping to 406 / 406 in under 20 seconds is strictly better than a ceiling that never moves, and nothing about level2 streaming is boot-latency-sensitive the way ticker's starvation was. (An earlier draft of this paragraph estimated the span as (14 - 1) * shard_spacing_ms, counting only level2's own shards — 65 seconds at the 5_000ms default this replaced, not the true 90; it omitted the 5 ticker shards staggered ahead of level2 in the same sequence. Verified against the real reshard/1 position math and a live run of this suite before being corrected here.)

level2 no longer needs to go first on a shared socket, because there is no longer a shared socket. What used to be this section — "level2 before ticker, spaced by @channel_spacing_ms" — existed only because a subscribe burst on one channel could stall the other's send_frame on the same connection. With every socket now carrying exactly one channel, that specific hazard cannot occur, and @channel_spacing_ms is gone. A subscribe can still hit :send_timeout against its own socket's own burst — a level2 socket decoding its own thirty-symbol snapshot, say — so the retry chain below is unchanged in kind; @subscribe_retry_delay_ms keeps its previous value (8_000ms) but is no longer borrowed from a channel-spacing constant, because that constant no longer exists to borrow it from — see its own comment.

An adaptive, self-shrinking shard size was considered and rejected — for now. The venue's own refusal could in principle drive @level2_pairs_per_socket down further at runtime, so a wrong constant could never be silently wrong forever — the option the architect's own brief for this fix raised directly. It was not built: correctly reconciling a shard resize, as opposed to the shard membership change reshard/1 already handles, needs this module to tell a live, still-subscribed shard's bookkeeping apart from a stale one whose symbols the venue already dropped when it refused and closed the connection — and getting that race wrong risks silently under- or double-subscribing a shard, which is a worse failure mode than the loud, honest one this module otherwise insists on everywhere else. Given 30 is not a guess but the boundary itself, now located rather than merely approached from below, a correct runtime resize's complexity did not clear its bar against a fixed, evidence-grounded constant plus the safety net that already exists: Socket's own error_kind/1 already classifies "too many" as :rate_limited and reports it as a Core.Notice on every occurrence (see Socket's moduledoc), and coverage_by_kind/1 already never marks a symbol covered for :order_book on intent alone — both pre-existing, verified unchanged by this fix, not new mechanism built for it. If 30 is ever also refused, a consumer with subscribe_notices/1 wired up hears about it exactly as loudly as every other refusal in this file, and lowering the constant is a one-line, reviewable change rather than a runtime decision this module made silently on its own.

level2_pairs_per_socket — a supervision option, not only a constant

What was rejected above is a runtime, self-adjusting resize — this module deciding on its own, mid-flight, to shrink the shard size because the venue refused one. What ships here instead is a static, consumer-set one: 30 is still the compiled-in default, but a caller of DpExchange.Coinbase (or Feed.start_link/1 directly) can pass level2_pairs_per_socket: n and have every level2 shard chunk to n from boot, through DpExchange.Coinbase.Supervisor's ordinary opts pass-through — no Supervisor code change was needed, because Feed already reads its other diagnostic knobs (resubscribe_interval_ms, subscribe_retry_delay_ms) the identical way.

This exists because the measured 30 is a fact about the venue on 2026-09-06, not a fact about this package, and venue facts change. A hardcoded constant means a venue-side change costs every consumer a package release and a redeploy; an option means a consumer absorbs it the same day — DpCryptoManagement asked for exactly this (DistortionPoint/dp-exchange-core issue #22) once they had already done the measurement that justified it.

Validated, not silently coerced — but only on the axis that is unconditionally nonsense. A non-integer or a value below 1 cannot chunk anything and is refused at init/1 (an ArgumentError, which fails Feed.start_link/1 and therefore DpExchange.Coinbase.Supervisor.start_link/1) rather than coerced into something that happens to run. A value above the measured 30, by contrast, is honoured, not capped — deliberately. Capping at 30 would quietly defeat the option's own stated purpose: a consumer setting it above today's measurement is doing so because they believe the venue's own ceiling has moved, which this package has no way to verify itself (see "why 30" above — this repo cannot bisect an authenticated session live). Refusing to even try would leave the option unable to do the one thing it was built for. What it gets instead is a loud Logger.warning at start naming the measured ceiling, its date and source, and the concrete risk — stated as what has actually been observed, not more than that.

What "the concrete risk" is, precisely, is now settled — DpCryptoManagement re-ran exactly this case. Setting this option above 30 means every ordinary level2 subscribe this package sends for that shard is, by itself, a SINGLE subscribe over the venue's real limit. DpCryptoManagement measured that shape directly, 2026-09-07 (issue #22 continuing): one socket, Socket.subscribe/4 for n = 30 (accepted, books=30, deltas=4258, all 30 symbols delivering), then n = 60 and n = 120 (both REFUSEDrate_limited, "too many L2 streams requested in a single session"books=0, deltas=0, nothing delivering), tried in both ascending and largest-first order. The refusal is wholesale, not a partial grant: n = 60 and n = 120 both delivered nothing at all, not the first 30, in either ordering — the shard's entire coverage is lost. And the socket survives: no :DOWN on a monitor, Process.alive?/1 true after a 40-second drain, both orderings. Cumulative overage (two smaller subscribes releasing nothing in between — see "the ceiling is concurrent, not cumulative" below) and a single oversized subscribe now read as the SAME behaviour: refused wholesale, socket alive. See docs/reference/coinbase/level2-session-limit.md for the full, dated account.

That makes this risk harder to notice than the warning used to say, not easier. A socket that closes, the way the 2026-08-26 incident below describes, at least announces itself — a :link_down notice, a reconnect, a liveness gap a consumer can watch for. A refusal on a socket that stays alive announces nothing of the kind: the venue's refusal still reaches subscribe_notices/1 as a :rate_limited Core.Notice (Socket's own error_kind/1 already classifies "too many" this way), but liveness itself looks perfect and the oversized shard simply never starts delivering. coverage/1 and coverage_by_kind/1 are the only things that reveal it — both report only symbols that actually delivered a payload, so a shard that never delivers never counts as covered, but nothing about the socket itself signals the loss the way a disconnect would.

The 2026-08-26 incident record stays — it is not overturned, it is unexplained. That incident (see "level2 gets its own, smaller sockets" above, measured 2026-08-27) reported the refusal closing the socket: 355 of 405 pairs stale, 1,480 refusals logged in one window. The 2026-09-07 measurement above could not reproduce that outcome, under either ordering. Neither this file nor DpCryptoManagement's own report resolves why: either the venue's behaviour changed between the two dates, or the 2026-08-26 incident had a second, unidentified cause. Both readings stay on record, dated and attributed, rather than one being asserted over the other.

The default stays at 30, without injected headroom, and that was a deliberate choice, not an oversight. 30 is not "the largest value that has not yet failed" (which 6 genuinely was, before this investigation) — it is the actual, located boundary: 30 accepted, 31 refused, confirmed by interleaving and a contamination check (see "why 30" above). Shrinking the default below a boundary that precise, on this package's own initiative, would misrepresent what was measured — the whole point of "declare what you measured, not what you assume." It would also buy nothing against the one risk that is still genuinely open: the unconditional 60-second resubscribe re-issuing unchanged symbols on an already-open socket forever (see "cumulative vs. concurrent" below) is an attempt-shaped risk, not a concurrent-membership-shaped one — a smaller shard does not reduce how many times that timer re-issues the same request, so headroom on shard size does not address it. A consumer who nonetheless wants margin against the concurrent ceiling — for reasons specific to their own scope or risk tolerance — has exactly the lever to take it themselves: pass a smaller level2_pairs_per_socket, e.g. 25, and this package honours it precisely.

ticker gets no equivalent option, on purpose, for now. @pairs_per_socket (100) is also a constant, also inherited rather than re-derived (see above), and also never probed against this venue — the same three facts that motivated level2's option. What is different is that level2's option answers a known, located venue ceiling a consumer already hit in production; ticker has no known ceiling of any kind to tune against, measured or suspected. An option with nothing to point it at is surface for a problem that has not been demonstrated to exist — consistency-for-its-own-sake is not this family's standard, evidence is. If ticker ever develops a measured ceiling the way level2 did, the fix is the same one this section documents, applied to @pairs_per_socket instead: nothing about this design is level2-specific.

The ceiling is concurrent, not cumulative — resolved, not merely bounded

2026-09-07 update. This section used to be titled "Cumulative vs. concurrent — the ceiling this package cannot rule out by itself." The question it posed is answered below. The reasoning that got this package here is kept rather than deleted, because the fix it produced — and the cheaper one that replaced it — both depend on understanding why it was written; see "When a moduledoc records an incident" in this package's own CLAUDE.md.

The 2026-09-06 bisection above (n=6..100) used a fresh Socket.start_link/1 for every attempt on purpose, "because cumulative session state is the thing under test." That control bought the consumer's measurement precision it would not otherwise have — but it also meant that result could only prove the ceiling on concurrently held level2 products. It could not, by its own design, say whether Coinbase's real ceiling counts concurrent subscriptions or cumulative ones — every distinct product a session was ever asked to carry, whether or not all of them were still wanted now. This module's own long-lived sockets were exactly the case that distinction mattered for.

The code was checked, not guessed, while the answer was still unknown. reconcile_shard/7 — reached from reshard/1 whenever subscribe/3, unsubscribe/2 or update_symbols/2 changed a shard Feed already had a socket open for — computed added = wanted -- current and sent exactly those newly-added symbols to Socket.subscribe/4 on the same already-open socket, never a fresh one. wanted_symbols per shard is always ≤ @level2_pairs_per_socket by construction, so no single subscribe call this module ever sent asked one socket for more than the shard ceiling at that instant — but nothing stopped the same socket's own subscription HISTORY from growing past it over time, one added batch at a time. This was never a rare edge case: MapSet.to_list/1's enumeration order is a function of the current key set, not of insertion history — proven directly against a 406-member synthetic set, where adding one member moves 8 symbols to a different chunk index and removing one moves 3 — so ordinary universe churn, not only a caller explicitly asking to add a symbol, routinely hands an already-open shard products it has never carried before.

Commit 9139881 hedged rather than guessed. With the answer unknown and 30 offering none of the headroom the old, deliberately-pessimistic 6 had, growing an already-open level2 shard replaced its socket outright — a fresh connection, subscribed with the shard's whole target set, only then discarding the old one — so no level2 socket this package opened was ever asked, over its whole lifetime, to carry more distinct products than one shard's worth. That made "should never be over the ceiling" true unconditionally, concurrently and cumulatively, regardless of which of Coinbase's two possible countings turned out to be real. It was also, by construction, more expensive than necessary if the answer turned out to be the convenient one: a full reconnect, a fresh snapshot for every symbol on the shard including the ones that never changed, and a coverage gap while the new socket came up — every reshard that grew a shard by even one symbol paid for a whole shard's worth of resubscribe.

DpCryptoManagement answered the question directly, 2026-09-07, issue #22 continuing. Three probes, each on ONE socket, using raw Socket.subscribe/4 — deliberately not Feed or update_symbols/2 — so the result is evidence about the VENUE's own accounting, not about this package's own dedup or bookkeeping. All three drained continuously against real products from the consumer's live scope:

  1. The same 30 products, re-sent roughly 60 seconds apart, 14 times: all accepted. The socket stayed alive throughout, roughly 13 minutes. books=30 on the first attempt and books=0 on every repeat — the venue recognised those products were already subscribed and replayed no snapshot, rather than refusing the repeat. Repeats do not accumulate. This is also, precisely, the "attempt-counting" probe the open question below used to call for: it directly answers whether the unconditional 60-second resubscribe this coordinator runs forever could itself feed an attempt-counted ceiling. It does not, at any shard size, for as long as 14 repeats over 13 minutes is representative of a longer-running socket.
  2. A different 30 products, on the SAME socket, without unsubscribing the first batch first: cumulative=60 — REFUSED, "too many L2 streams requested in a single session". But the socket stayed alive, and the first batch kept delivering (27 of 30 still ticking during the refusal) — only the second, unreleased batch was rejected.
  3. Four batches of 30 products each, always unsubscribing the previous batch before requesting the next: all four accepted, a fresh snapshot every time — 120 distinct products moved through one socket's lifetime, never more than 30 live at any moment.

So the ceiling is 30 CONCURRENT products per session, not 30 over a session's lifetime. unsubscribe releases budget the venue actually honours (probe 3); a long-lived socket does not degrade as its membership churns, provided it releases what it is giving up before it asks for what it is gaining (probe 2, read the other way round). The cumulative-hedge 9139881 shipped is no longer necessary, and — see the next section — it was strictly more expensive than the alternative this measurement unlocks.

Unsubscribe before subscribe — what replaced socket replacement

Socket replacement bought a guarantee this package no longer needs to buy that way. reconcile_shard/7's "level2" clause is gone; every shard, either channel, now reconciles on its EXISTING socket through one function, reconcile_shard_in_place/7 — the mechanism ticker already used, since it never had a ceiling to protect against in the first place. For a shard whose membership changes, removed is unsubscribed and added is subscribed on that one socket — and removed is unsubscribed FIRST, every time, never the other way round, and the subscribe never goes out until the unsubscribe has.

The ordering is load-bearing, and it is the whole point. Probe 3 above works BECAUSE the departing batch's slots were freed before the arriving batch was requested; probe 2 is the identical operation in the other order — request before release — and it was refused. A level2 shard already carrying its full level2_pairs_per_socket that issued a bare additive subscribe would hit exactly probe 2's shape, and per that probe the refusal is QUIET from a consumer's own vantage point: the socket lives, the shard's existing symbols keep flowing, and only the newly-requested ones silently never arrive. That is precisely the failure this fix exists not to introduce, so shrinking always happens before growing: the transient concurrent count during any reconcile is bounded by max(length(current), length(wanted)), and both are already bounded by shard construction — never more than level2_pairs_per_socket, including mid-reconcile, not only at rest.

What guarantee this package actually has, and how it was established — read from Socket, not assumed. Socket.subscribe/4 and Socket.unsubscribe/3 both go through FrameSender.send/3, which calls WebSockex.send_frame/2 — a synchronous :gen.call into the WebSockex connection process that returns only once that ONE process has handed the frame to the underlying TCP socket (or reports why it could not — see FrameSender's own moduledoc). Because attempt_channel_reconcile/6 (and the synchronous clause of reconcile_shard_in_place/7) never calls Socket.subscribe/4 until Socket.unsubscribe/3 for the SAME message has returned :ok, the unsubscribe frame is written to this one TCP connection strictly before the subscribe frame is. TCP delivers bytes on one connection in the order they were written, and Coinbase's own session, so far as this package or DpCryptoManagement has ever observed, processes one connection's frames in the order it receives them — nothing in any probe above, or in the original bisection, has ever shown the venue reordering two frames sent on the same socket. Given that, sending the frames in this order is what makes the venue very likely to process the unsubscribe before the subscribe too, even though this package cannot observe that processing directly.

That is the honest limit of what this package can confirm, and it is worth stating plainly rather than folding into the paragraph above. Socket.unsubscribe/3's :ok means "the frame was handed to this connection's own send," never "the venue has finished releasing this shard's departing slots" — Coinbase's l2_data protocol gives this package no acknowledgement frame for an unsubscribe to wait on (contrast "channel" => "subscriptions", which acknowledges a subscribe, and which dispatch/2 deliberately does not treat as coverage — see this moduledoc's own opening section — let alone waits on here). So this package relies on frame ORDER on one connection, not on a confirmed venue-side STATE transition, to keep a reconcile inside the shard's own cap. Given probe 3's own success — four rounds of exactly this pattern, no waiting for any acknowledgement between the unsubscribe and the next subscribe, all four accepted — ordering alone, without an acknowledgement this protocol does not offer, is what this package actually has, and it is what DpCryptoManagement's own probe already relied on to get the result it got.

A transient send failure on the unsubscribe half withholds the subscribe half entirely, rather than risk the alternative. attempt_channel_reconcile/6 retries a failed unsubscribe with the same bounded backoff a channel subscribe already gets (handle_unsubscribe_failure/8); the subscribe for added is never attempted until the unsubscribe has actually gone out, or the retries are exhausted and the added symbols are withheld for this cycle (reported through subscribe_notices/1, the same as any other subscribe that never took — see "a timed-out subscribe used to be thrown away" below). Silently proceeding to subscribe anyway — on the theory that the earlier unsubscribe frame probably still made it out despite the reported failure, which FrameSender's own moduledoc says is often true — would reintroduce exactly the risk this whole fix exists to close, on the one path where this package cannot check.

This used to leave one gap open: a permanently stranded unsubscribe. It does not any more — see "a stranded unsubscribe is not silently forgotten" below, which was closed after this file first shipped this section, on a coordinator's own trace rather than a self-report: reconcile_shard_in_place/7 recorded state.shards[key].symbols = wanted unconditionally, on both its sync and deferred clauses, regardless of whether the unsubscribe underneath it ever actually succeeded. removed's symbols vanished from this module's own bookkeeping the moment retries gave up, even though they were never released at the venue — a real, permanent budget leak, closed by state.pending_unsubscribes and the mechanism the next section documents.

The reshuffle hazard that made this real in the first place is unchanged, and still handled — just more cheaply now. reshard/1 still recomputes chunks on every subscribe/3, unsubscribe/2 and update_symbols/2, and a MapSet's enumeration order still shifts on almost any membership change (moving 8 of 406 symbols on one add, 3 on one removal — the same proof as before, unchanged by this fix). What changed is what happens when that reshuffle hands an already-open shard symbols it has never carried before: it no longer costs a reconnect, a fresh snapshot for the shard's entire unchanged membership, and a coverage gap while a brand-new socket comes up — it costs exactly two frames, in order, on the connection that was already open.

(This section used to end with two open probes recommended for a future consumer to run — "attempt-counting" and "cumulative distinct-count." Probe 1 above IS the attempt-counting probe; it is no longer open. Nothing on the cumulative-vs-concurrent axis this section is about remains open. What was, for a time, a genuinely open gap on a DIFFERENT axis — a permanently stranded unsubscribe — is the next section's subject, and it is closed too.)

A stranded unsubscribe is not silently forgotten

Traced, not merely reviewed. The withheld-added recovery two sections up was checked by following what reconcile_shard_in_place/7 actually records, not by trusting the summary of it: the WITHHELD side (added) does recover — the 60-second cycle re-issues state.shards[key].symbols, which already equals wanted, so those symbols get subscribed on the next tick regardless of whether this reconcile's own unsubscribe ever succeeded. The STRANDED side (removed) did not — nothing re-issued an unsubscribe for it, and this module's own bookkeeping had already stopped tracking it as owed. Left alone, a shard that hit this even once would degrade into exactly the failure this whole file exists to prevent: the venue's live count for that session sitting at old ∪ wanted rather than wanted, eventually exceeding the shard's own cap, at which point every later subscribe for that shard is refused — quietly, per DpCryptoManagement's own probe 2: socket alive, existing symbols still flowing, new ones silently absent. The recovery path for added makes this WORSE in a sense, not better, because it keeps re-issuing subscribes into a budget the stranded slots guarantee cannot fit.

state.pending_unsubscribes%{{channel, index} => [symbol, ...]} — is that record. strand_unsubscribe/7 writes to it whenever a reconcile gives up on releasing some removed symbols, on EITHER the synchronous clause (no retry chain of its own — one attempt, then straight to stranding, the same "no retry in the primary path" shape open_shard/5's sync clause already has) or the deferred one's bounded retry chain exhausting (handle_unsubscribe_failure/8). handle_info(:resubscribe, _) reads it back on the SAME unconditional cadence retry_missing_shards/1 already uses to recover a shard whose socket never opened at all — the governing principle is that function's own comment, word for word: a shard with "NO automatic recovery path at all, ever" is "a silent, permanent coverage gap indistinguishable from a quiet market, on a venue this coordinator's OWN moduledoc says must never go unretried." A stranded unsubscribe is the identical defect in a different place, and gets the identical fix.

Ordering still applies, and it is reached through the SAME machinery, not a parallel one. Each resubscribe tick now sends {:channel_reconcile, key, socket, channel, pending, symbols, credentials} per shard — pending (a shard's own state.pending_unsubscribes entry, [] in the ordinary case) as removed, and symbols (the shard's WHOLE current membership, not a delta — matching exactly what an unconditional resubscribe already re-issued before this fix) as added. This is the identical attempt_channel_reconcile/6 an ordinary subscribe/3, unsubscribe/2 or update_symbols/2 reconcile already runs, so the retried unsubscribe precedes that shard's resubscribe on the same tick for the same reason it precedes one anywhere else in this file — not a second, separately-reasoned code path that could drift from the first. For the ordinary case (pending == []) this costs nothing beyond what a plain Socket.subscribe/4 already cost: unsubscribe_step/3 short-circuits an empty list without sending anything.

What clears an entry: a successful send of exactly those symbols, per symbol, not per shard. clear_pending_unsubscribe/3 computes pending -- removed rather than deleting the whole key, because a shard can be carrying a stranded release from one reconcile while an unrelated, still-outstanding one from an earlier reconcile also sits there — only the symbols THIS successful send actually covered are removed. strand_unsubscribe/7 is symmetric: it merges (Enum.uniq(&1 ++ removed)), never overwrites, so a later, unrelated stranding does not erase an earlier one still waiting on its own retry.

What clears an entry the other way: its release is actually confirmed sent, not merely that the shard stopped being wanted. This used to be simpler and wrong: drop_unwanted_shards/3 dropped a shard's pending_unsubscribes entry in the same breath it dropped the shard from state.shards, on every reshard/1 call, regardless of whether that entry was still owed. Two ways that broke, both found on the same trace: a vanishing shard whose SYNCHRONOUS unsubscribe attempt failed had strand_unsubscribe/7 record the stranding and drop_unwanted_shards/3 erase it again in the very next statement of the same reshard/1 call — recorded and discarded in one breath. A vanishing shard handled ASYNCHRONOUSLY had its state.shards entry deleted before the deferred {:channel_reconcile, ...} even attempted a send; if that later attempt failed, strand_unsubscribe/7 recreated state.pending_unsubscribes[key] for a key handle_info(:resubscribe, _) could never find again, because it walks Map.keys(state.shards) specifically. Both silently contradicted this section's own claim and the notice text's own promise ("picked back up on the next unconditional resubscribe cycle") for the one shape of stranding most likely in practice: one on a shard nobody wants any more, where nothing else will ever touch that key again to surface the problem.

Fixed by no longer treating "vanishing" and "safe to drop" as the same moment. drop_unwanted_shards/4 now leaves a vanishing key's state.shards entry (and its pending_unsubscribes entry, untouched) alone whenever that key has a stranding recorded OR was just handed to a deferred reconcile this very reshard/1 call — the one case where a stranding might still be recorded a moment later, after this function has already run. finalize_shard/4 and its deferred twin maybe_finalize_vanished_shard/2 are the only places a vanished shard's bookkeeping is now actually retired, and only at the moment either can know it is safe: right after an unsubscribe attempt that leaves nothing pending. Until then the shard stays in state.shards with symbols: [], which is what keeps it visible to handle_info(:resubscribe, _)'s walk and to attempt_channel_reconcile/8's own retry chain — a dropped shard's socket is still left running unmanaged either way, the existing, unchanged tradeoff.

This answers "can state.pending_unsubscribes grow without bound": bounded by however many shards — open OR vanished-but-not-yet-confirmed-clear — carry an unresolved stranding, not by this module's all-time history of failures, because a shard's own entry is retired only once its release is confirmed.

A second, independent gap on the same trace: an ORDINARY reconcile ignored a standing stranding entirely. reconcile_shard_in_place/8 computed added/removed from current -- wanted / wanted -- current alone — current being this module's own optimistic belief, always set to the PREVIOUS wanted regardless of whether that reconcile's own unsubscribe actually succeeded — never consulting state.pending_unsubscribes. Only handle_info(:resubscribe, _)'s 60-second cycle looked at it. A shard sitting at its own cap with one symbol stranded would accept a plain additive subscribe for an unrelated new symbol on the very next ordinary subscribe/2, unsubscribe/2 or update_symbols/2 call, in between ticks — the exact "quiet overflow" DpCryptoManagement's own probe 2 describes, reached through the path this whole unsubscribe-before-subscribe design was built to close, not around it. Both clauses now fold Map.get(state.pending_unsubscribes, key, []) into current before computing added/removed, so a standing stranding is retried — and any newly-wanted symbol withheld behind it — on every reconcile that touches that shard, not only once every resubscribe_interval_ms.

A stranded symbol re-added to the same shard while still pending is not a hazard, because added is always the shard's WHOLE current set, never a delta. Ordinary churn can plausibly move a symbol out (stranding its release) and back in before the next tick retries it. Whatever the retried unsubscribe does to that symbol, the SAME reconcile's subscribe — for the shard's full, current membership — puts it right back if it is still wanted, in the same message, in order. At worst this costs one redundant unsubscribe/resubscribe pair for that one symbol; it never leaves the shard short a symbol it currently wants.

What is not built, and the honest reason: correlating a stranding to a socket RECONNECT specifically. A socket that dies and reconnects gets a fresh session with the venue — every stranded slot on the OLD session is released along with it, whether this module notices or not. But Socket deliberately holds no shard-correlating identity in its :link_up/:link_down notices (see this module's own handle_info({:dp_exchange, :coinbase, %Notice{}}, _) — every socket's notices arrive through the identical, unattributed shape, matching Socket's own moduledoc: "Socket holds no book to key by anything"), and WebSockex's own reconnect keeps the same pid, so pid identity cannot substitute either. Building that correlation would mean widening Socket's own contract — real, cross-module surface — for a case whose cost, left as is, is bounded and low: a stale pending_unsubscribes entry retried against a fresh session sends one unsubscribe for a symbol the venue no longer has subscribed, which this family's stated idempotency assumption (FrameSender's own moduledoc: "subscribes are idempotent on every venue in this family, so a duplicate is harmless") extends to as a working assumption for unsubscribe too, though NOT independently measured against an unsubscribed symbol specifically — an honestly-labelled gap, not a silent one. And because that stale unsubscribe is immediately followed, in the SAME reconcile, by a subscribe of the shard's whole current membership, a stale entry costs at most one wasted frame and self-corrects the same tick — it cannot leave the shard under-subscribed. There is also a practical reason the stale case rarely persists at all: whatever made the earlier sends fail (a busy, backed-up socket) is usually exactly what a reconnect clears, so the very next retry after a reconnect is likely to simply succeed and clear the entry normally, not because the reconnect was detected, but because reconnecting tends to un-wedge the condition that caused the stranding in the first place.

What transient_frame_failure?/1's real reach turns out to be, stated plainly rather than assumed generous. Every reason Socket.unsubscribe/3 can return comes from FrameSender alone — :send_timeout or {:send_exit, reason} — and both are classified transient. Socket.unsubscribe/3 has no analogue of a subscribe's {:credentials_required, channel}: it builds no JWT and checks no credentials, per its own definition (contrast Socket.subscribe/4's subscription_message/3). So the non-transient cond branch in handle_unsubscribe_failure/8 does not fire against the real venue today — "permanently stranded" is reached, in practice, only by EXHAUSTING the bounded retry chain against a socket that keeps answering :send_timeout or {:send_exit, _} without ever dying outright (Process.alive?/1 staying true throughout — a socket that is genuinely dead short-circuits every reconcile path before it reaches this branch at all, matching every other dead-socket guard in this module). The non-transient branch is kept anyway, wired to the identical stranding-and-retry behaviour as an exhausted chain, for the same "do not assume a shape that happens to hold today" reason handle_subscribe_failure/6 keeps its own — but it is not what this mechanism was actually built to close in production; the retry-exhaustion branch is.

A refusal is not always a clean gate — observed, not explained

DpCryptoManagement's bisection above answers "accepted or refused" per n, but two of their runs saw something this package has no account of: at n = 3134 in one run, books = 0 (a clean refusal); at n = 3550 in the same run, 1,300–2,000 books delivered alongside the rate_limited notice — a partially honoured oversized subscription. A later run saw n = 31 refuse with 1,655 books delivered. None of this moves the boundary — every n ≥ 31 refused and every n ≤ 30 did not, in both runs — but it means "refused" can apparently be a spectrum rather than a gate. See docs/reference/coinbase/level2-session-limit.md for the full, dated account; it is recorded there as an observed venue characteristic with no explanation attached, not rationalised into one.

At @level2_pairs_per_socket = 30 this package should never itself trigger a partial refusal, and that reasoning was checked, not assumed: every subscribe this module sends for a level2 shard carries at most 30 symbols — wanted_symbols per shard is ≤ @level2_pairs_per_socket by construction — and the unsubscribe-before- subscribe reconcile above (see "unsubscribe before subscribe") keeps every socket's CONCURRENT count within that same bound at every instant of a reconcile, not only at rest, regardless of which of Coinbase's two possible countings turns out to be real. Nothing in this package's own behaviour asks for 31 or more at once. (These partial refusals were themselves observed on SINGLE, oversized subscribes — the same shape as the 2026-08-26 incident and the level2_pairs_per_socket option's own warning below, neither of which this package's own behaviour triggers either — not on the cumulative overage the 2026-09-07 probes above exercised. The two shapes are now known, 2026-09-07, to behave alike on whether the socket survives a refusal — both do, see the option's own warning above — but that does not extend to the partial-delivery spectrum this section is about: the 2026-09-07 single-oversized re-test saw clean refusals, books=0, at every size it tried, not the partial delivery this section describes, so the spectrum observed here stays its own, separate, unexplained venue characteristic.) Should the venue nonetheless answer a rate_limited notice while some of that same shard's symbols are genuinely delivering — the exact shape DpCryptoManagement observed — coverage/1 and coverage_by_kind/1 need no special case to stay honest: both are built entirely from state.delivering, which is populated only by handle_info({:dp_exchange, :coinbase, payload}, state) when an actual Types.Quote, Types.OrderBook or Types.OrderBookDelta arrives, tagged with whichever symbol it carries. handle_info({:dp_exchange, :coinbase, %Notice{}}, state) — the clause a :rate_limited notice takes — never touches state.delivering at all. A symbol that delivered a book is covered for :order_book whether or not its shard's subscribe was also, separately, answered with a refusal notice; a symbol that delivered nothing stays :not_covered regardless. The two facts were never coupled in the first place, which is the same "coverage is observed, never intended" guarantee this moduledoc opens with, just exercised by a venue behaviour this package did not anticipate rather than one it designed for.

A timed-out subscribe used to be thrown away — now it is retried

FrameSender's own moduledoc says the whole point of turning a send_frame exit into {:error, :send_timeout} is that "a slow socket becomes a failed batch, which a caller can report and retry, rather than a dead connection", and that subscribes are idempotent on every venue in this family, so a duplicate is harmless. This module used to log that error and drop it — the retry half of the design was never wired, so a channel_subscribe that lost the race against a level2 snapshot burst simply stayed unsubscribed until the next unconditional resubscribe tick, which reproduces the identical busy-socket condition and fails identically.

This is not hypothetical. A consumer running against a real ~400-symbol universe measured the exact inversion this predicts, across five boots over roughly 5.5 hours:

statequotes (ticker)order_book (level2)
broken (4 boots)~5 / 406~406 / 406, 11,000+ frames
healthy (1 boot)400 / 4066 / 406

When level2 gets through broadly, its opening snapshot burst is what starves ticker; when the venue refused most level2 subscriptions outright (its own per-session stream limit — see above), ticker had the socket to itself and got everything. A lone :send_timeout on a ticker subscribe was also observed directly in an earlier run. Both are DpCryptoManagement's issue #22.

Classify before retrying — not every failure can be fixed by waiting

{:error, :send_timeout} and {:error, {:send_exit, reason}} are transient: the socket was busy decoding a burst, or briefly gone, and the identical request can reasonably succeed once it catches up. {:error, {:credentials_required, channel}} (see Socket's subscription_message/3) is permanent — no amount of waiting supplies a credential that was never given, and retrying it would only loop, so it fails loudly on the first attempt and is never rescheduled.

The backoff waits out a socket's own burst, not another channel's

At the time this was written, a retry waited out the same busy-socket condition @channel_spacing_ms existed to wait out between level2 and ticker sharing one socket, so @subscribe_retry_delay_ms borrowed that value rather than guessing a second number for the same underlying wait. @channel_spacing_ms is gone now that no socket carries two channels — see "level2 gets its own, smaller sockets" above — @subscribe_retry_delay_ms keeps its value (8_000ms) but stands on its own reasoning: a socket can still be busy decoding its own just-subscribed burst (a level2 socket's own snapshot, however small its shard), and the identical request can reasonably succeed once that clears. @max_subscribe_retries is 2: one initial attempt plus two retries is enough to survive one snapshot burst without turning a stuck socket into an unbounded loop. See the constants' own comments for the arithmetic that keeps the whole retry chain well inside a resubscribe cycle, so it can never stack frames against the unconditional re-issue documented below.

Exhaustion is loud

A channel that never subscribed is exactly the invisible half-dead feed this whole issue is about, and it used to surface as a Logger.warning a consumer had no facade-level way to see. Giving up — whether because the failure was permanent or because retries ran out — now also emits a Core.Notice of kind :coverage_change: those symbols will not deliver this kind of data, which is exactly the fact coverage/1 and coverage_by_kind/1 need a consumer to go re-check rather than discover from a quiet chart.

A reconnect that does not resubscribe is a coverage collapse with no error

WebSockex reconnects a dropped socket on its own, and a bare reconnect leaves it connected and subscribed to nothing — silently, because a socket that is up and receiving nothing is not itself an error. That is a real, measured incident on this venue's own reference implementation: coverage decayed from full to the REST-poll floor over roughly forty minutes with the feed still reporting healthy, because nothing re-asked the venue for anything after the reconnect.

This coordinator re-issues every shard's subscriptions on a timer, unconditionally. Re-subscribing a channel the socket already carries costs one frame the venue ignores; not re-subscribing one it silently dropped costs the shard's whole coverage until someone notices a quiet chart.

Every shard beyond the first must open on its own tick, not the same one

shard_spacing_ms staggers shard opens relative to each other, not relative to a fixed instant. A scope wide enough to need three or more shards — DpCryptoManagement's issue #20, 406 symbols / 5 shards, filed against real production traffic — used to schedule every shard past the first (the synchronous one) with the same fixed delay, so all of them opened in the same instant: exactly the connect burst this module's own design note above warns the venue answers with resets. Only the shard whose burst-mate connections lost that race ever delivered a tick; coverage sat at whatever fraction of one shard survived, indistinguishable from the outside from a quiet market. The 60-second unconditional resubscribe re-issued the same burst every minute. Both paths now schedule each shard's turn position * shard_spacing_ms after the one before it — see "shard_spacing_ms — a supervision option" below for what that value is, where it comes from, and how a consumer can change it.

This staggering has to reach an already-open shard too, not only a brand-new connection. reconcile_shard/7 used to receive the same delay reshard/1 computes for it and drop it on the floor — every already-open shard update_symbols/2 touches in one call had its level2 subscribe scheduled at the identical instant. That is not the connect burst above (no new socket opens), but it is a related hazard: Socket. subscribe/4 blocks THIS GenServer — via FrameSender, up to WebSockex.send_frame/2's 5s window — for as long as its target socket takes to acknowledge, and several such messages landing in this process's own mailbox together serialise into back-to-back blocking sends, stalling coverage/1 and every other call to this Feed for as long as the slowest one takes. Fixed the same way: delay now reaches reconcile_shard/7 and staggers its frames exactly as it already staggered a new shard's.

shard_spacing_ms — a supervision option, so a test does not have to wait out a

production timer to prove staggering happened

30 in @default_level2_pairs_per_socket and 60_000 in @default_resubscribe_interval_ms both became supervision options once they had a reason a consumer might legitimately want to move them. shard_spacing_ms gets the same treatment for a different reason: this package's own test suite needed one first.

Five of this file's own sharding tests exist to prove staggering happened — that a connect burst is spread across ticks, that ticker shards are scheduled ahead of level2 ones, that an already-open shard's reconcile is staggered too (the "already-open shard" fix two sections up) — and every one of them, before this option existed, could only prove that by actually waiting out @shard_spacing_ms in real time: 45 of this suite's roughly 51 seconds, across five tests, one of them 15 seconds on its own. That is not merely slow; a test that synchronises by sleeping out a real production timer is the same shape of hazard wait_until/3 exists elsewhere in this suite to avoid — passing locally, then flaking in a more loaded CI the timer was never sized for. Feed.start_link/1 (via DpExchange.Coinbase.Supervisor's ordinary opts pass-through — no Supervisor code change needed, the same as level2_pairs_per_socket) now accepts shard_spacing_ms: n and every stagger this module schedules — a new shard's connect, an already-open shard's reconcile, the unconditional resubscribe, the derived floor under resubscribe_interval_ms (next_resubscribe_delay/1) — reads it from state instead of the module attribute. A test drives it down to a few tens of milliseconds and keeps every one of those assertions meaningful: the relative ordering (shard 1 before shard 2, ticker before level2) and the fact that a real, positive delay was scheduled at all — see each test's own comment for how it does that without merely asserting the calls happened.

Validated the same way level2_pairs_per_socket is, on the axis that is unconditionally nonsense. A negative value or a non-integer cannot schedule Process.send_after/3 at all and is refused at init/1 with an ArgumentError — see validate_shard_spacing_ms!/1. 0 is NOT refused: Process.send_after/3 accepts it without complaint, and unlike a negative delay there is nothing mathematically broken about "every shard opens in the same instant" — it is simply the connect burst this whole file otherwise exists to avoid, which is the next paragraph's problem, not this one.

This is also a knob pointed at a venue rate limit, and this package just spent one investigation (level2_pairs_per_socket, above) learning what happens when a number that should be the venue's own fact is instead a guess. Unlike level2's per-session product ceiling, though, Coinbase does document a connect-rate number: the Advanced Trade rate-limits page states "WebSocket connections ... are ... limited to 8 per second per IP" (re-read 2026-09-06, the same pass that produced the level2 ceiling above). This module opens one new socket per shard_spacing_ms tick, so that figure converts directly into a floor: ceil(1_000 / 8) = 125ms. A consumer setting something below that is asking this package's own connects alone to exceed a limit Coinbase states outright, independent of whatever else shares that IP.

Below the floor is honoured, not refused — the same shape as level2_pairs_per_socket above its own measured ceiling, and for a symmetric reason: this package has no visibility into a consumer's own network position (a dedicated IP with headroom this repo cannot see, say), so it does not get to assume the conservative number is the only correct one for every consumer. What a sub-floor value gets instead is a loud Logger.warning naming the documented floor, its source, and the concrete risk — connects tighter than the venue's own stated per-IP rate risk the connect-burst resets this moduledoc opens with — legible, not silently accepted, exactly the standard level2_pairs_per_socket set above the measured 30.

The default is now 1_000, chosen rather than inherited — closing docs/design/closed/2026-09-09_shard-spacing-headroom.md. @default_shard_spacing_ms used to be 5_000, carried over unexamined from the reference fix this package replaced, the same way @pairs_per_socket (100) still is — nobody had picked 5_000 FOR this venue; nobody had looked at it at all until the idea doc above asked where it came from and the honest answer was "nowhere." That idea doc's own closing criteria allowed reasoning from the documented rate-limits page alone, without a live probe, for exactly this kind of value: a pure connect-rate number that already converts to an exact floor (125ms, above) with nothing left to bisect. This package has not run a shard_spacing_ms probe against the real venue and does not claim to; the reasoning below is entirely documentation-derived.

  • 1_000ms is one connection per second — an 8x margin under the documented floor, not the floor itself. The idea doc explicitly ruled out adopting 125ms as the new default: no margin for scheduler jitter, GC pauses, or a consumer's own concurrent load sharing the same IP. 500ms (a 4x margin) was also considered and set aside in favour of the larger margin below.
  • Chosen more conservatively than headroom against the documented rate alone would require, because of a real, recent incident on a different venue in this family: Webull's shards crash-looped this same week once abandoned sessions accumulated against an undocumented five-connection ceiling. Coinbase's own rate-limits page documents a RATE (8/second/IP), not a concurrency cap, and documents no concurrency cap at all — but "no documented cap" is not "no cap," and a tighter stagger increases how many of this module's own connects are opening, and therefore how many are mid-handshake, in any short window — the same axis that bit Webull, on a different venue with a different (undocumented) limit. Socket's own live measurement method (docs/reference/coinbase/level2-session-limit.md) took on the order of a few seconds to establish one connection; at 1_000ms spacing, only a handful of this module's own connects are ever simultaneously mid-handshake for the 406-symbol, 19-shard scope this moduledoc already discusses, against roughly two dozen that would be in flight at once at the bare 125ms floor. This is a documented-rate-plus-precaution choice, not a measurement of Coinbase's own concurrency behaviour — this repo has neither measured nor been told Coinbase has any such ceiling; the precaution is carried over from a sibling venue's incident, not from anything Coinbase-specific.
  • For the 406-symbol scope above, boot-to-full-level2-coverage drops from 90 seconds at the old default to 18 seconds at this one — a fivefold improvement, deliberately short of the roughly-twentyfold the bare floor would give, because outrunning the documented rate by the smallest possible margin was never the goal.

This is a behaviour change for every consumer that has not set shard_spacing_ms explicitly: this feed now reaches full coverage noticeably faster after boot and after any event that reopens shards. A consumer who wants the old, more conservative pacing back can still ask for it: shard_spacing_ms: 5_000.

One spacing, several jobs — considered separately, kept as one number

Four places in this module read shard_spacing_ms, and a value right for one is not automatically right for another, so each is worth checking rather than assumed:

  • The initial connect stagger (reshard/1's rest list, above) and retry_missing_shards/1's reopen stagger both open brand-new sockets — a direct fit for the documented 8-connections-per-second-per-IP floor this section is built on.
  • handle_info(:resubscribe, _)'s unconditional re-issue walk staggers frames onto ALREADY-open sockets — no new connection, so the connect-rate floor is not a direct fit. Two independent reasons still keep it on the same number rather than a faster one of its own: Coinbase's own rate-limits page states its 8/second/IP ceiling covers connects AND messages together (docs/reference/coinbase/level2-session-limit.md's own paraphrase, "an 8-per-second-per-IP connect/message rate"), so a faster resubscribe cadence risks the same class of venue reaction a faster connect cadence does; and, independent of anything the venue does, Socket.subscribe/4 blocks THIS GenServer for as long as its target socket takes to acknowledge (see "every shard beyond the first must open on its own tick," above), so spacing resubscribe frames apart protects coverage/1 and every other call to this Feed from stalling behind a burst of blocking sends regardless of what the venue does with them.
  • next_resubscribe_delay/1 is not a fifth job at all — it is a derived floor that has to track whatever the resubscribe walk actually uses, or cycles overlap and stack frames behind each other (see that function's own comment). Splitting it onto a different constant than the walk it measures would just reintroduce the bug it exists to prevent.

Nothing above argues for resubscribing FASTER than the connect pace this section already settled on, so the two concerns never pull the value in different directions. One number does all four jobs honestly; splitting it would add a second constant with no case where the two would ever need to disagree.

The venue rewrites an aliased product id on delivery, and that has to be undone HERE

Measured live, 2026-09-05, against wss://advanced-trade-ws.coinbase.com: subscribing ticker to ["XLM-USDC", "AVAX-USDC"] — sent exactly as asked, both real, listed products — delivers every frame tagged XLM-USD and AVAX-USD. The venue's own subscription acknowledgement even echoes the rewritten names back ("ticker" => ["XLM-USD", "AVAX-USD"]), not the ones actually sent. This is the venue's own declared behaviour, not a guess: Rest.get_alias_map/1 reads the same public /market/products catalogue this module already reaches through Rest.get_symbols/1 and Rest.list_instruments/1, and on this date 112 of the first 114 USDC products carried a non-empty alias naming their -USD counterpart. A caller subscribed under the alias form received nothing under the name it asked for while a name it never asked for arrived instead — measured against a real 406-symbol consumer scope (DpCryptoManagement's issue #22): 174 of 406 requested pairs delivered nothing, while 401 pairs never requested were decoded and stored.

Attribution lives here, not in Socket

Socket stays venue-mechanics-only: it decodes a frame and delivers a struct tagged with whatever product_id the venue actually sent, exactly as it did before this fix. Every consumer-facing rewrite happens in this module's handle_info({:dp_exchange, :coinbase, payload}, state), immediately before a delivered payload is recorded as coverage and fanned out — because this is the one place that already holds wanted (what the caller actually asked for) beside the delivered payload. Duplicating wanted into Socket just to make the same decision twice would be a second place for the two to disagree. Socket holds no book to key by anything — see its own moduledoc — so this holds regardless of which channel delivered the frame (ticker via deliver_ticker/3, or level2 via Socket's deliver_snapshot/4/deliver_delta/4) since all three arrive here as the same {:dp_exchange, :coinbase, payload} shape and every struct Socket sends carries :symbol.

Built once, from the venue's own catalogue, never from string-munging

Rest.get_alias_map/1 is the only source for this map — reusing the same /market/products fetch get_symbols/1 and list_instruments/1 already make, per the standing rule against a second way to ask. Munging -USDC into -USD would be exactly the "nearby substitute" this family forbids, and would be wrong for any pair the venue does not alias — nothing here assumes the suffix relationship holds in general.

It is scheduled once, asynchronously, the first time subscribe/3 or update_symbols/2 is called (maybe_schedule_alias_map_fetch/1, gated on alias_map_status: :unfetched so a second call never re-schedules it) — not from init/1, so a Feed that is merely supervised and never asked to stream anything never makes a network call, and not synchronously inside the triggering handle_call/3, so it never competes with @call_timeout's socket-connect budget. Until it resolves, state.alias_map is simply %{} — indistinguishable, by design, from "the venue aliases nothing here", which resolves to the same safe fallback below.

The fetch has to wait, not fail — DpCryptoManagement's issue #26

Rest.get_alias_map/1 reached Core.HttpClient without rate_limit_blocking: true, so it went through fail-fast check/3 rather than blocking acquire/3 — and this fetch is scheduled off the first subscribe/3, which for any real consumer is boot, the single most contended moment for their own rate limiter (universe discovery, catalogue reads and market overviews all landing at once). It is scheduled into exactly the window most likely to throttle it. One throttled call at that moment, on code that never retried (see below, before this fix), disabled attribution for the life of the process. Measured live: 406 pairs requested as -USDC, delivered as -USD, overlap 5 — coverage_by_kind/1 and the consumer's own tracker each reporting a truthful, and wildly different, count.

This is the third instance of one family-wide pattern — dp_exchange_robinhood's issue #16, this package's own issue #23 sweep (which fixed every other REST call site in this package and missed this one, because the alias-map fetch did not exist to audit when that comment was written), and now this: a background call with nothing waiting on it, failing instead of waiting, while Core.HttpClient's own error message names the fix in its text ("callers that can wait should set rate_limit_blocking: true"). This call site is exactly the caller that can: it runs off Process.send_after, nothing blocks on its result, and its only job is to populate a cache before frames arrive. Waiting a second here is free; failing is total. rate_limit_blocking: true is now set unconditionally by default_alias_map_source/2, which also forwards :limiter, :plug, :timeout, :retry_attempts, :retry_delay and :weight from this module's own opts — the same allowlist shape Rest's own request pipeline uses — so a test can exercise the real fetch pipeline end-to-end (a fake :plug response behind a real, deterministic :limiter) rather than only ever exercising the alias_map_source injection seam.

Classified and retried, the same way a channel subscribe already is

Blocking removes the self-throttle as a failure mode, but does not remove every failure: the limiter's own bounded wait can still time out ({:exchange_error, _venue, :rate_limit_timeout} — the wait itself ran out, not a refusal), and the fetch can still fail for reasons no amount of waiting fixes. transient_alias_map_failure?/1 classifies it exactly the way transient_subscribe_failure?/1 classifies a channel subscribe, and for the same reason: not every failure can be fixed by retrying, so retrying one that can't only delays an honest "this failed" and spends a retry budget a genuinely transient failure needs.

A timed-out wait for the caller's own rate limiter is the one case treated as transient — the identical request can reasonably succeed once the limiter's bucket has drained further, which is the whole reason rate_limit_blocking: true exists here. Everything else — an unrecognised response shape, a refused request, a raw or unclassified reason (including whatever a test's own stand-in returns) — is treated as permanent, matching transient_subscribe_failure?/1's own default-to-permanent stance for anything not explicitly known to clear on its own. Retries are bounded (@max_alias_map_retries, backed off by @alias_map_retry_delay_ms — both overridable via opts for a test's benefit, the same shape as subscribe_retry_delay_ms above) rather than looped forever, so a fetch that genuinely cannot succeed still gives up and reports itself rather than retrying silently without end.

Exhausting the retries, or failing permanently on the first attempt, both leave state.alias_map at %{} and alias_map_status: :unavailable — the same safe fallback as before this fix, just reached only after a transient failure has been given its fair chance to clear.

A failed fetch reports itself — and a late notice subscriber has to be able to hear it

A failed fetch reports itself, as a :data_quality notice to notice_subscribers — "attribution is degraded and here is why" — never a silently guessed mapping. attribution_targets/2 is the single fallback for every case where no wanted name resolves — unfetched, failed, legitimately alias-free, or a frame arriving for a symbol outside wanted altogether (in-flight just after an unsubscribe, or a raw test send/2): deliver under whatever the venue actually sent, exactly the pre-fix behaviour, rather than inventing a name.

The notice used to be unreceivable by construction — also DpCryptoManagement's issue #26. The fetch is scheduled from subscribe/3; a consumer that calls subscribe_notices/1 afterward — the ordinary sequence, since notices are naturally the second thing a caller registers once it already knows it wants data — could register only after the fetch had already failed and fanned out to zero subscribers. A notice announcing a persistent degraded state that can only ever fire in the one window before anyone could be listening for it is worse than no signal: it looks like a working alarm that never rings.

Fixed by replay, not by moving the emission earlier — narrowing the window does not close it, and this file already learned that lesson once with next_resubscribe_delay/1's per-tick storm. state.alias_map_status and the reason that produced it (state.alias_map_failure_reason) already persist for as long as the condition holds, so {:subscribe_notices, subscriber} replays the identical notice to that one newly-registered subscriber whenever it finds the state already :unavailable — once per registration, never on a timer, and never re-sent to a subscriber who already has it, so it cannot become the per-tick storm next_resubscribe_delay/1 was already fixed for once. A consumer that registers before the fetch resolves sees the ordinary fan-out, exactly as before; one that registers after sees the same notice, late but not lost.

Both caller-facing names, when both are wanted

The venue treats XLM-USDC and XLM-USD as one market. If a caller subscribes to both, both are entitled to every update — attribution_targets/2 resolves a delivered id to every name in wanted that names the same market (its own id and, where the catalogue says so, its alias), and the delivery loop sends one copy per resolved name. This holds regardless of whether the venue itself echoes one frame or two per update for a dual subscription — resolution runs per delivered frame against the full candidate set, so two frames naming the same pair of wanted symbols do not double-deliver into a name twice per market update; they each resolve to the same one-or-both names again.

coverage/1 needed no code change to become honest

Coverage was already whatever key delivering holds — see the moduledoc up top. Once delivery is recorded under the caller's own requested name instead of the venue's rewritten one, coverage/1 reports exactly what was asked for, by construction, with nothing endpoint-specific added at the coverage/1 call site itself.

coverage_by_kind/1 — the same fact, split by what actually arrived

coverage/1 answers "is anything arriving for this symbol", and it answers that question truthfully — but it asks nothing about which kind of payload showed up. Types.Quote, Types.OrderBook and Types.OrderBookDelta all carry :symbol, so a level2 book update (whether a full snapshot or an incremental delta) and a ticker quote count identically toward delivering, and a symbol with one of the two dark looks exactly like a symbol with both healthy.

That is not a hypothetical: level2 on this venue delivered upward of 11,000 frames across 406 subscribed symbols while ticker stayed dark on all but a handful of them, and coverage/1 still answered :stream for all 406 — correctly, by its own definition, and useless for telling anyone that quotes had gone silent. Two separate DpCryptoManagement issues (#20 and #22) sat unpinned for days because nothing in this package's own observability could distinguish "everything is fine" from "the book is fine and the ticker is dead". coverage_by_kind/1 exists to make that distinction answerable without adding a second, differently-shaped API: it reports the same observed-arrival fact coverage/1 reports, just partitioned by DpExchange.Core.Capabilities.data_kind/0 instead of collapsed across it.

The kind is read off the payload's own struct — %Types.Quote{} is :quotes, %Types.OrderBook{} and %Types.OrderBookDelta{} are both :order_book — never off a channel name. Snapshot and delta share one kind deliberately: this question is "is book data arriving", not "in what shape", and the struct type itself already tells a caller which shape it is holding. level2 and ticker are this venue's words for its own wire protocol and stop existing the moment a frame becomes a Core.Types.* struct; coverage_by_kind/1 never sees them and could not leak them if it wanted to.

state.delivering therefore keys each symbol to a small map of kind => timestamp rather than a single timestamp, so a symbol that has delivered both a quote and a book update carries both kinds at once, and one going dark does not erase the other. coverage_by_kind/1 folds that structure the other way — kind first, then symbol — to match the shape DpExchange.Core.Venue.coverage_by_kind/1 promises.

Summary

Functions

Returns a specification to start this module under a supervisor.

coverage/1, split by which Core.Types.* kind actually arrived — see the moduledoc's "coverage_by_kind/1" section for why coverage/1 alone could not tell "ticker dark, book healthy" apart from "everything healthy".

The scope split into one list per ticker socket. level2 is chunked separately, at its own, smaller and independently configurable size — see the moduledoc's "level2 gets its own, smaller sockets" and "level2_pairs_per_socket — a supervision option" sections — and has no public function of its own, the same way it has no public visibility anywhere else in this module.

Functions

child_spec(init_arg)

Returns a specification to start this module under a supervisor.

See Supervisor.

coverage(feed \\ __MODULE__)

@spec coverage(GenServer.server()) :: %{
  required(String.t()) => :stream | :internal_poll | :not_covered
}

coverage_by_kind(feed \\ __MODULE__)

@spec coverage_by_kind(GenServer.server()) :: %{
  required(DpExchange.Core.Capabilities.data_kind()) => %{
    required(String.t()) => :stream | :internal_poll | :not_covered
  }
}

coverage/1, split by which Core.Types.* kind actually arrived — see the moduledoc's "coverage_by_kind/1" section for why coverage/1 alone could not tell "ticker dark, book healthy" apart from "everything healthy".

shards(symbols)

@spec shards([String.t()]) :: [[String.t()]]

The scope split into one list per ticker socket. level2 is chunked separately, at its own, smaller and independently configurable size — see the moduledoc's "level2 gets its own, smaller sockets" and "level2_pairs_per_socket — a supervision option" sections — and has no public function of its own, the same way it has no public visibility anywhere else in this module.

start_link(opts)

@spec start_link(keyword()) :: GenServer.on_start()

subscribe(feed \\ __MODULE__, symbols, opts \\ [])

@spec subscribe(GenServer.server(), [String.t()], keyword()) :: :ok | {:error, term()}

subscribe_notices(feed \\ __MODULE__, opts \\ [])

@spec subscribe_notices(
  GenServer.server(),
  keyword()
) :: :ok

unsubscribe(feed \\ __MODULE__, symbols)

@spec unsubscribe(GenServer.server(), [String.t()]) :: :ok | {:error, term()}

update_symbols(feed \\ __MODULE__, symbols)

@spec update_symbols(GenServer.server(), [String.t()]) :: :ok | {:error, term()}