This venue's subscription lifecycle — internal. The facade's subscribe/2,
unsubscribe/2, update_symbols/2 and coverage/1 are served from here.
What a consumer can and cannot learn
A consumer learns what is arriving, through coverage/1. It cannot learn how: this
module owns the sockets, the sharding and the pacing, and none of that reaches the
facade.
Coverage is observed, never intended
A symbol enters the coverage map when a payload for it arrives, never when it is subscribed. That distinction is the strongest guarantee in the contract and it exists because a venue once reported 325 symbols subscribed and confirmed while 174 were delivering. Reporting the subscription would have said 325.
A symbol that has been subscribed and has delivered nothing is simply absent, which the
facade documents as :not_covered.
Sharded — this used to run on one connection, and that stopped being true
This venue's whole scope used to run on a single socket, and that was measured: once its self-killing heartbeat was fixed it subscribed 401 of 401 pairs on one connection, and sharding it anyway opened fourteen connections for no gain.
That stopped being true on 2026-08-26. Coinbase started answering a level2
subscribe over its per-session limit with "too many L2 streams requested in a single session", closing the socket — a total data gap, not degraded coverage: measured
2026-08-27 against a real ~400-symbol universe, 355 of 405 pairs went stale and 1,480
refusals were logged in one window. The measurement about one socket being enough was
honest when it was taken; it stopped being true the moment the venue's own limit did.
@pairs_per_socket is 100 — carried over from the reference fix this replaces
rather than re-derived, the number came from a real production incident, not this
package's own probing. It sizes ticker's own sockets, and only ticker's: the next
section is why level2 stopped sharing it.
level2 gets its own, smaller sockets — the two channels stopped sharing a shard
This stopped being true again, on a different axis, measured against
DpCryptoManagement's own 406-symbol production universe (issue #22 continuing, not
reopening — ticker's starvation above is a separate, already-fixed incident in the
same file). At @pairs_per_socket (100), shards/1 splits 406 symbols into
[100, 100, 100, 100, 6] — four full shards and a six-symbol remainder. Every level2
subscribe on the four full shards was refused with "too many L2 streams requested in a single session"; the six-symbol shard's was not. 5,099 refusals were logged across
sixteen otherwise-healthy boots, and coverage_by_kind/1 answered order_book: 6
throughout — exactly the tail shard's own symbol count, which is the arithmetic that
pins the cause on the shard size: ticker has no such ceiling and answered
quotes: 406 on the same boots, so nothing else about those boots — the connection, the
alias fix, the resubscribe cadence — explains a number that lines up precisely with one
shard's population and no other.
No Coinbase document states the real ceiling. Re-read 2026-09-06 specifically
looking for a per-session level2 stream count: the Advanced Trade channels reference,
the connection overview, and the Advanced Trade rate-limits page ("WebSocket
connections and unauthenticated messages are each limited to 8 per second per IP" — a
connect-rate ceiling, not a per-session subscription count) all say nothing about how
many products one session may carry on level2. The older Exchange product's own
separate rate-limits page states a different number entirely — 10 subscriptions per
product per channel, meaning duplicate subscriptions to the same product, not the
count of distinct products — and this package speaks Advanced Trade (Socket's
@endpoint), not Exchange, so that number would not transfer even if it were on point.
The prior investigation that first shipped @pairs_per_socket found the docs silent on
this; they still are. This does not stop being true just because the number below is now
measured — it is measured behaviour, never a documented figure.
This package still cannot narrow it by probing the venue itself. level2 is on
@authenticated_channels, and this repo's own testing strategy draws the line at
exactly that boundary: tier 2 (live public endpoints, by hand) is fair game, tier 3
(authenticated) "needs credentials this repo must never hold." Bisecting a real
session's level2 subscription size against the live venue with a real credential is
precisely tier 3, and remains a line this repo does not cross for any endpoint,
level2 included. What changed is that a consumer — who holds the credential this
repo structurally cannot — did the bisection, in production, and reported the result
back as tier-3 evidence. That is a different thing from this package probing itself,
and the distinction is why the number below is attributed to them rather than to any
measurement this repo ran.
@level2_pairs_per_socket is 30. Measured 2026-09-06 by DpCryptoManagement
(issue #22) against the live venue with real credentials, not by this package and not
from documentation:
requested (n) | 6 | 12 | 25 | 30 | 31 | 35 | 50 | 100 |
|---|---|---|---|---|---|---|---|---|
| verdict | accepted | accepted | accepted | accepted | REFUSED | REFUSED | REFUSED | REFUSED |
Method, because a measured number is only worth its method. One fresh
Socket.start_link/1 per attempt, never reused — cumulative session state was
explicitly the thing under test, which is exactly the question this file's own
"cumulative vs. concurrent" section below could not otherwise answer. 3s to establish
the connection; subscribe(socket, "level2", symbols, creds); then 20 seconds draining
the mailbox continuously while tallying. Every symbol came from the consumer's real,
live 406-symbol scope, so silence means refusal rather than an idle book. Verdict
refused on a rate_limited notice, accepted on OrderBook/OrderBookDelta
payloads. The boundary was confirmed by interleaving two full runs back to back:
n=30 accepted, n=31 REFUSED, n=30 accepted, n=31 REFUSED. A contamination check ran
too — immediately after a run where n=31–34 refused, n=6 and n=30 were re-run
and both accepted, which is what rules out the refusals being saturation from the
probing itself rather than a genuine per-session ceiling.
Same evidentiary standard as the 6 this replaces, just with the boundary actually
located rather than merely bounded far below a known failure: 30 is the largest value
with positive evidence of acceptance, 31 the smallest with positive evidence of
refusal. Nothing between them needed to be guessed this time, unlike 6 versus the old
100.
The two channels no longer share a shard slice. ticker keeps @pairs_per_socket
(100) — it has no known ceiling, and shrinking it to match level2 would multiply
connection count for the channel that was never the problem, for no benefit ticker
needs. level2 is chunked separately, at @level2_pairs_per_socket, and every chunk of
either channel opens its own dedicated, single-channel socket: there is no longer a
shard that carries both. For the 406-symbol universe above that is 5 ticker sockets
(unchanged) plus 14 level2 sockets (ceil(406 / 30)) — 19 total against 73 under the
6-sized grouping this replaces, and 5 before level2 needed its own grouping at all.
Socket's own moduledoc records why more sockets than the original single-socket design
is affordable now in a way it would not have been before 2026-09-06: removing
in-package level2 book maintenance cut per-frame decode cost roughly tenfold
(65–110 ms to 6.6 ms, at the BTC-USD book size DpCryptoManagement measured live) —
decode cost, not socket count, was the resource actually in short supply.
Connects are staggered across both groups on one sequence, ticker first. Every new
socket this module opens — whichever channel it carries — takes the next tick of
shard_spacing_ms (1_000ms by default — see that option's own section below), in an
order that places every touched ticker shard ahead of every touched level2 shard.
This is what keeps ticker's own boot-time coverage exactly as fast as the section above
describes: a 406-symbol subscribe still resolves its synchronous reply and its remaining
ticker shards inside the same handful of seconds as before this fix, with level2's 14
shards ramping in behind them. For that universe (19 shards touched together in one
reshard/1 call — 5 ticker plus 14 level2) the last shard's tick lands
(shard_count - 1) * shard_spacing_ms after boot — the identical span
next_resubscribe_delay/1 computes below for its own overlap floor — 18 * shard_spacing_ms: 18 seconds at the current default, materially slower than ticker's
own coverage and an accepted, stated cost, now roughly a twentieth of what the 6-sized
grouping cost (about six minutes): order_book coverage was permanently 6 / 406 before
level2 got its own grouping at all, climbing by 5,099 refusals and counting; ramping to
406 / 406 in under 20 seconds is strictly better than a ceiling that never moves, and
nothing about level2 streaming is boot-latency-sensitive the way ticker's starvation
was. (An earlier draft of this paragraph estimated the span as (14 - 1) * shard_spacing_ms, counting only level2's own shards — 65 seconds at the 5_000ms
default this replaced, not the true 90; it omitted the 5 ticker shards staggered ahead
of level2 in the same sequence. Verified against the real reshard/1 position math and
a live run of this suite before being corrected here.)
level2 no longer needs to go first on a shared socket, because there is no longer a
shared socket. What used to be this section — "level2 before ticker, spaced by
@channel_spacing_ms" — existed only because a subscribe burst on one channel could
stall the other's send_frame on the same connection. With every socket now carrying
exactly one channel, that specific hazard cannot occur, and @channel_spacing_ms is
gone. A subscribe can still hit :send_timeout against its own socket's own burst — a
level2 socket decoding its own thirty-symbol snapshot, say — so the retry chain below
is unchanged in kind; @subscribe_retry_delay_ms keeps its previous value (8_000ms)
but is no longer borrowed from a channel-spacing constant, because that constant no
longer exists to borrow it from — see its own comment.
An adaptive, self-shrinking shard size was considered and rejected — for now. The
venue's own refusal could in principle drive @level2_pairs_per_socket down further at
runtime, so a wrong constant could never be silently wrong forever — the option the
architect's own brief for this fix raised directly. It was not built: correctly
reconciling a shard resize, as opposed to the shard membership change reshard/1
already handles, needs this module to tell a live, still-subscribed shard's bookkeeping
apart from a stale one whose symbols the venue already dropped when it refused and
closed the connection — and getting that race wrong risks silently under- or
double-subscribing a shard, which is a worse failure mode than the loud, honest one this
module otherwise insists on everywhere else. Given 30 is not a guess but the boundary
itself, now located rather than merely approached from below, a correct runtime resize's
complexity did not clear its bar against a fixed, evidence-grounded constant plus the
safety net that already exists: Socket's own error_kind/1 already classifies "too
many" as :rate_limited and reports it as a Core.Notice on every occurrence (see
Socket's moduledoc), and coverage_by_kind/1 already never marks a symbol covered for
:order_book on intent alone — both pre-existing, verified unchanged by this fix, not
new mechanism built for it. If 30 is ever also refused, a consumer with
subscribe_notices/1 wired up hears about it exactly as loudly as every other refusal
in this file, and lowering the constant is a one-line, reviewable change rather than a
runtime decision this module made silently on its own.
level2_pairs_per_socket — a supervision option, not only a constant
What was rejected above is a runtime, self-adjusting resize — this module deciding on
its own, mid-flight, to shrink the shard size because the venue refused one. What ships
here instead is a static, consumer-set one: 30 is still the compiled-in default, but
a caller of DpExchange.Coinbase (or Feed.start_link/1 directly) can pass
level2_pairs_per_socket: n and have every level2 shard chunk to n from boot,
through DpExchange.Coinbase.Supervisor's ordinary opts pass-through — no Supervisor
code change was needed, because Feed already reads its other diagnostic knobs
(resubscribe_interval_ms, subscribe_retry_delay_ms) the identical way.
This exists because the measured 30 is a fact about the venue on 2026-09-06, not a
fact about this package, and venue facts change. A hardcoded constant means a venue-side
change costs every consumer a package release and a redeploy; an option means a consumer
absorbs it the same day — DpCryptoManagement asked for exactly this
(DistortionPoint/dp-exchange-core issue #22) once they had already done the
measurement that justified it.
Validated, not silently coerced — but only on the axis that is unconditionally
nonsense. A non-integer or a value below 1 cannot chunk anything and is refused at
init/1 (an ArgumentError, which fails Feed.start_link/1 and therefore
DpExchange.Coinbase.Supervisor.start_link/1) rather than coerced into something that
happens to run. A value above the measured 30, by contrast, is honoured, not
capped — deliberately.
Capping at 30 would quietly defeat the option's own stated purpose: a consumer setting
it above today's measurement is doing so because they believe the venue's own ceiling
has moved, which this package has no way to verify itself (see "why 30" above — this
repo cannot bisect an authenticated session live). Refusing to even try would leave the
option unable to do the one thing it was built for. What it gets instead is a loud
Logger.warning at start naming the measured ceiling, its date and source, and the
concrete risk — stated as what has actually been observed, not more than that.
What "the concrete risk" is, precisely, is now settled — DpCryptoManagement re-ran
exactly this case. Setting this option above 30 means every ordinary level2
subscribe this package sends for that shard is, by itself, a SINGLE subscribe over the
venue's real limit. DpCryptoManagement measured that shape directly, 2026-09-07
(issue #22 continuing): one socket, Socket.subscribe/4 for n = 30 (accepted,
books=30, deltas=4258, all 30 symbols delivering), then n = 60 and n = 120
(both REFUSED — rate_limited, "too many L2 streams requested in a single session" — books=0, deltas=0, nothing delivering), tried in both ascending and
largest-first order. The refusal is wholesale, not a partial grant: n = 60 and
n = 120 both delivered nothing at all, not the first 30, in either ordering — the
shard's entire coverage is lost. And the socket survives: no :DOWN on a monitor,
Process.alive?/1 true after a 40-second drain, both orderings. Cumulative overage (two
smaller subscribes releasing nothing in between — see "the ceiling is concurrent, not
cumulative" below) and a single oversized subscribe now read as the SAME behaviour:
refused wholesale, socket alive. See
docs/reference/coinbase/level2-session-limit.md for the full, dated account.
That makes this risk harder to notice than the warning used to say, not easier. A
socket that closes, the way the 2026-08-26 incident below describes, at least announces
itself — a :link_down notice, a reconnect, a liveness gap a consumer can watch for. A
refusal on a socket that stays alive announces nothing of the kind: the venue's refusal
still reaches subscribe_notices/1 as a :rate_limited Core.Notice (Socket's own
error_kind/1 already classifies "too many" this way), but liveness itself looks
perfect and the oversized shard simply never starts delivering. coverage/1 and
coverage_by_kind/1 are the only things that reveal it — both report only symbols that
actually delivered a payload, so a shard that never delivers never counts as covered,
but nothing about the socket itself signals the loss the way a disconnect would.
The 2026-08-26 incident record stays — it is not overturned, it is unexplained. That incident (see "level2 gets its own, smaller sockets" above, measured 2026-08-27) reported the refusal closing the socket: 355 of 405 pairs stale, 1,480 refusals logged in one window. The 2026-09-07 measurement above could not reproduce that outcome, under either ordering. Neither this file nor DpCryptoManagement's own report resolves why: either the venue's behaviour changed between the two dates, or the 2026-08-26 incident had a second, unidentified cause. Both readings stay on record, dated and attributed, rather than one being asserted over the other.
The default stays at 30, without injected headroom, and that was a deliberate
choice, not an oversight. 30 is not "the largest value that has not yet failed"
(which 6 genuinely was, before this investigation) — it is the actual, located
boundary: 30 accepted, 31 refused, confirmed by interleaving and a contamination
check (see "why 30" above). Shrinking the default below a boundary that precise, on
this package's own initiative, would misrepresent what was measured — the whole point
of "declare what you measured, not what you assume." It would also buy nothing against
the one risk that is still genuinely open: the unconditional 60-second resubscribe
re-issuing unchanged symbols on an already-open socket forever (see "cumulative vs.
concurrent" below) is an attempt-shaped risk, not a concurrent-membership-shaped
one — a smaller shard does not reduce how many times that timer re-issues the same
request, so headroom on shard size does not address it. A consumer who nonetheless wants
margin against the concurrent ceiling — for reasons specific to their own scope or risk
tolerance — has exactly the lever to take it themselves: pass a smaller
level2_pairs_per_socket, e.g. 25, and this package honours it precisely.
ticker gets no equivalent option, on purpose, for now. @pairs_per_socket (100)
is also a constant, also inherited rather than re-derived (see above), and also never
probed against this venue — the same three facts that motivated level2's option. What
is different is that level2's option answers a known, located venue ceiling a
consumer already hit in production; ticker has no known ceiling of any kind to tune
against, measured or suspected. An option with nothing to point it at is surface for a
problem that has not been demonstrated to exist — consistency-for-its-own-sake is not
this family's standard, evidence is. If ticker ever develops a measured ceiling the
way level2 did, the fix is the same one this section documents, applied to
@pairs_per_socket instead: nothing about this design is level2-specific.
The ceiling is concurrent, not cumulative — resolved, not merely bounded
2026-09-07 update. This section used to be titled "Cumulative vs. concurrent — the
ceiling this package cannot rule out by itself." The question it posed is answered below.
The reasoning that got this package here is kept rather than deleted, because the fix it
produced — and the cheaper one that replaced it — both depend on understanding why it was
written; see "When a moduledoc records an incident" in this package's own CLAUDE.md.
The 2026-09-06 bisection above (n=6..100) used a fresh Socket.start_link/1 for every
attempt on purpose, "because cumulative session state is the thing under test." That
control bought the consumer's measurement precision it would not otherwise have — but it
also meant that result could only prove the ceiling on concurrently held level2
products. It could not, by its own design, say whether Coinbase's real ceiling counts
concurrent subscriptions or cumulative ones — every distinct product a session was
ever asked to carry, whether or not all of them were still wanted now. This module's own
long-lived sockets were exactly the case that distinction mattered for.
The code was checked, not guessed, while the answer was still unknown.
reconcile_shard/7 — reached from reshard/1 whenever subscribe/3, unsubscribe/2 or
update_symbols/2 changed a shard Feed already had a socket open for — computed
added = wanted -- current and sent exactly those newly-added symbols to
Socket.subscribe/4 on the same already-open socket, never a fresh one.
wanted_symbols per shard is always ≤ @level2_pairs_per_socket by construction, so no
single subscribe call this module ever sent asked one socket for more than the shard
ceiling at that instant — but nothing stopped the same socket's own subscription
HISTORY from growing past it over time, one added batch at a time. This was never a
rare edge case: MapSet.to_list/1's enumeration order is a function of the current key
set, not of insertion history — proven directly against a 406-member synthetic set, where
adding one member moves 8 symbols to a different chunk index and removing one moves 3 —
so ordinary universe churn, not only a caller explicitly asking to add a symbol,
routinely hands an already-open shard products it has never carried before.
Commit 9139881 hedged rather than guessed. With the answer unknown and 30
offering none of the headroom the old, deliberately-pessimistic 6 had, growing an
already-open level2 shard replaced its socket outright — a fresh connection, subscribed
with the shard's whole target set, only then discarding the old one — so no level2
socket this package opened was ever asked, over its whole lifetime, to carry more
distinct products than one shard's worth. That made "should never be over the ceiling"
true unconditionally, concurrently and cumulatively, regardless of which of Coinbase's
two possible countings turned out to be real. It was also, by construction, more
expensive than necessary if the answer turned out to be the convenient one: a full
reconnect, a fresh snapshot for every symbol on the shard including the ones that never
changed, and a coverage gap while the new socket came up — every reshard that grew a
shard by even one symbol paid for a whole shard's worth of resubscribe.
DpCryptoManagement answered the question directly, 2026-09-07, issue #22 continuing.
Three probes, each on ONE socket, using raw Socket.subscribe/4 — deliberately not
Feed or update_symbols/2 — so the result is evidence about the VENUE's own
accounting, not about this package's own dedup or bookkeeping. All three drained
continuously against real products from the consumer's live scope:
- The same 30 products, re-sent roughly 60 seconds apart, 14 times: all accepted.
The socket stayed alive throughout, roughly 13 minutes.
books=30on the first attempt andbooks=0on every repeat — the venue recognised those products were already subscribed and replayed no snapshot, rather than refusing the repeat. Repeats do not accumulate. This is also, precisely, the "attempt-counting" probe the open question below used to call for: it directly answers whether the unconditional 60-second resubscribe this coordinator runs forever could itself feed an attempt-counted ceiling. It does not, at any shard size, for as long as 14 repeats over 13 minutes is representative of a longer-running socket. - A different 30 products, on the SAME socket, without unsubscribing the first batch
first:
cumulative=60— REFUSED,"too many L2 streams requested in a single session". But the socket stayed alive, and the first batch kept delivering (27 of 30 still ticking during the refusal) — only the second, unreleased batch was rejected. - Four batches of 30 products each, always unsubscribing the previous batch before requesting the next: all four accepted, a fresh snapshot every time — 120 distinct products moved through one socket's lifetime, never more than 30 live at any moment.
So the ceiling is 30 CONCURRENT products per session, not 30 over a session's
lifetime. unsubscribe releases budget the venue actually honours (probe 3); a
long-lived socket does not degrade as its membership churns, provided it releases what
it is giving up before it asks for what it is gaining (probe 2, read the other way
round). The cumulative-hedge 9139881 shipped is no longer necessary, and — see the next
section — it was strictly more expensive than the alternative this measurement unlocks.
Unsubscribe before subscribe — what replaced socket replacement
Socket replacement bought a guarantee this package no longer needs to buy that way.
reconcile_shard/7's "level2" clause is gone; every shard, either channel, now
reconciles on its EXISTING socket through one function, reconcile_shard_in_place/7 —
the mechanism ticker already used, since it never had a ceiling to protect against in
the first place. For a shard whose membership changes, removed is unsubscribed and
added is subscribed on that one socket — and removed is unsubscribed FIRST, every
time, never the other way round, and the subscribe never goes out until the unsubscribe
has.
The ordering is load-bearing, and it is the whole point. Probe 3 above works BECAUSE
the departing batch's slots were freed before the arriving batch was requested; probe 2
is the identical operation in the other order — request before release — and it was
refused. A level2 shard already carrying its full level2_pairs_per_socket that issued
a bare additive subscribe would hit exactly probe 2's shape, and per that probe the
refusal is QUIET from a consumer's own vantage point: the socket lives, the shard's
existing symbols keep flowing, and only the newly-requested ones silently never arrive.
That is precisely the failure this fix exists not to introduce, so shrinking always
happens before growing: the transient concurrent count during any reconcile is bounded by
max(length(current), length(wanted)), and both are already bounded by shard
construction — never more than level2_pairs_per_socket, including mid-reconcile, not
only at rest.
What guarantee this package actually has, and how it was established — read from
Socket, not assumed. Socket.subscribe/4 and Socket.unsubscribe/3 both go through
FrameSender.send/3, which calls WebSockex.send_frame/2 — a synchronous :gen.call
into the WebSockex connection process that returns only once that ONE process has handed
the frame to the underlying TCP socket (or reports why it could not — see FrameSender's
own moduledoc). Because attempt_channel_reconcile/6 (and the synchronous clause of
reconcile_shard_in_place/7) never calls Socket.subscribe/4 until Socket.unsubscribe/3
for the SAME message has returned :ok, the unsubscribe frame is written to this one TCP
connection strictly before the subscribe frame is. TCP delivers bytes on one connection in
the order they were written, and Coinbase's own session, so far as this package or
DpCryptoManagement has ever observed, processes one connection's frames in the order it
receives them — nothing in any probe above, or in the original bisection, has ever shown
the venue reordering two frames sent on the same socket. Given that, sending the frames in
this order is what makes the venue very likely to process the unsubscribe before the
subscribe too, even though this package cannot observe that processing directly.
That is the honest limit of what this package can confirm, and it is worth stating
plainly rather than folding into the paragraph above. Socket.unsubscribe/3's :ok
means "the frame was handed to this connection's own send," never "the venue has finished
releasing this shard's departing slots" — Coinbase's l2_data protocol gives this package
no acknowledgement frame for an unsubscribe to wait on (contrast "channel" => "subscriptions", which acknowledges a subscribe, and which dispatch/2 deliberately
does not treat as coverage — see this moduledoc's own opening section — let alone waits on
here). So this package relies on frame ORDER on one connection, not on a confirmed
venue-side STATE transition, to keep a reconcile inside the shard's own cap. Given probe
3's own success — four rounds of exactly this pattern, no waiting for any acknowledgement
between the unsubscribe and the next subscribe, all four accepted — ordering alone,
without an acknowledgement this protocol does not offer, is what this package actually
has, and it is what DpCryptoManagement's own probe already relied on to get the result it
got.
A transient send failure on the unsubscribe half withholds the subscribe half entirely,
rather than risk the alternative. attempt_channel_reconcile/6 retries a failed
unsubscribe with the same bounded backoff a channel subscribe already gets
(handle_unsubscribe_failure/8); the subscribe for added is never attempted until the
unsubscribe has actually gone out, or the retries are exhausted and the added symbols are
withheld for this cycle (reported through subscribe_notices/1, the same as any other
subscribe that never took — see "a timed-out subscribe used to be thrown away" below).
Silently proceeding to subscribe anyway — on the theory that the earlier unsubscribe
frame probably still made it out despite the reported failure, which FrameSender's own
moduledoc says is often true — would reintroduce exactly the risk this whole fix exists
to close, on the one path where this package cannot check.
This used to leave one gap open: a permanently stranded unsubscribe. It does not any
more — see "a stranded unsubscribe is not silently forgotten" below, which was closed
after this file first shipped this section, on a coordinator's own trace rather than a
self-report: reconcile_shard_in_place/7 recorded state.shards[key].symbols = wanted
unconditionally, on both its sync and deferred clauses, regardless of whether the
unsubscribe underneath it ever actually succeeded. removed's symbols vanished from
this module's own bookkeeping the moment retries gave up, even though they were never
released at the venue — a real, permanent budget leak, closed by
state.pending_unsubscribes and the mechanism the next section documents.
The reshuffle hazard that made this real in the first place is unchanged, and still
handled — just more cheaply now. reshard/1 still recomputes chunks on every
subscribe/3, unsubscribe/2 and update_symbols/2, and a MapSet's enumeration order
still shifts on almost any membership change (moving 8 of 406 symbols on one add, 3 on
one removal — the same proof as before, unchanged by this fix). What changed is what
happens when that reshuffle hands an already-open shard symbols it has never carried
before: it no longer costs a reconnect, a fresh snapshot for the shard's entire unchanged
membership, and a coverage gap while a brand-new socket comes up — it costs exactly two
frames, in order, on the connection that was already open.
(This section used to end with two open probes recommended for a future consumer to run — "attempt-counting" and "cumulative distinct-count." Probe 1 above IS the attempt-counting probe; it is no longer open. Nothing on the cumulative-vs-concurrent axis this section is about remains open. What was, for a time, a genuinely open gap on a DIFFERENT axis — a permanently stranded unsubscribe — is the next section's subject, and it is closed too.)
A stranded unsubscribe is not silently forgotten
Traced, not merely reviewed. The withheld-added recovery two sections up was
checked by following what reconcile_shard_in_place/7 actually records, not by trusting
the summary of it: the WITHHELD side (added) does recover — the 60-second cycle
re-issues state.shards[key].symbols, which already equals wanted, so those symbols
get subscribed on the next tick regardless of whether this reconcile's own unsubscribe
ever succeeded. The STRANDED side (removed) did not — nothing re-issued an unsubscribe
for it, and this module's own bookkeeping had already stopped tracking it as owed. Left
alone, a shard that hit this even once would degrade into exactly the failure this whole
file exists to prevent: the venue's live count for that session sitting at
old ∪ wanted rather than wanted, eventually exceeding the shard's own cap, at which
point every later subscribe for that shard is refused — quietly, per DpCryptoManagement's
own probe 2: socket alive, existing symbols still flowing, new ones silently absent. The
recovery path for added makes this WORSE in a sense, not better, because it keeps
re-issuing subscribes into a budget the stranded slots guarantee cannot fit.
state.pending_unsubscribes — %{{channel, index} => [symbol, ...]} — is that
record. strand_unsubscribe/7 writes to it whenever a reconcile gives up on releasing
some removed symbols, on EITHER the synchronous clause (no retry chain of its own — one
attempt, then straight to stranding, the same "no retry in the primary path" shape
open_shard/5's sync clause already has) or the deferred one's bounded retry chain
exhausting (handle_unsubscribe_failure/8). handle_info(:resubscribe, _) reads it back
on the SAME unconditional cadence retry_missing_shards/1 already uses to recover a
shard whose socket never opened at all — the governing principle is that function's own
comment, word for word: a shard with "NO automatic recovery path at all, ever" is "a
silent, permanent coverage gap indistinguishable from a quiet market, on a venue this
coordinator's OWN moduledoc says must never go unretried." A stranded unsubscribe is the
identical defect in a different place, and gets the identical fix.
Ordering still applies, and it is reached through the SAME machinery, not a parallel
one. Each resubscribe tick now sends {:channel_reconcile, key, socket, channel, pending, symbols, credentials} per shard — pending (a shard's own
state.pending_unsubscribes entry, [] in the ordinary case) as removed, and
symbols (the shard's WHOLE current membership, not a delta — matching exactly what an
unconditional resubscribe already re-issued before this fix) as added. This is the
identical attempt_channel_reconcile/6 an ordinary subscribe/3, unsubscribe/2 or
update_symbols/2 reconcile already runs, so the retried unsubscribe precedes that
shard's resubscribe on the same tick for the same reason it precedes one anywhere else in
this file — not a second, separately-reasoned code path that could drift from the first.
For the ordinary case (pending == []) this costs nothing beyond what a plain
Socket.subscribe/4 already cost: unsubscribe_step/3 short-circuits an empty list
without sending anything.
What clears an entry: a successful send of exactly those symbols, per symbol, not per
shard. clear_pending_unsubscribe/3 computes pending -- removed rather than deleting
the whole key, because a shard can be carrying a stranded release from one reconcile
while an unrelated, still-outstanding one from an earlier reconcile also sits there —
only the symbols THIS successful send actually covered are removed. strand_unsubscribe/7
is symmetric: it merges (Enum.uniq(&1 ++ removed)), never overwrites, so a later,
unrelated stranding does not erase an earlier one still waiting on its own retry.
What clears an entry the other way: its release is actually confirmed sent, not
merely that the shard stopped being wanted. This used to be simpler and wrong:
drop_unwanted_shards/3 dropped a shard's pending_unsubscribes entry in the same
breath it dropped the shard from state.shards, on every reshard/1 call, regardless
of whether that entry was still owed. Two ways that broke, both found on the same
trace: a vanishing shard whose SYNCHRONOUS unsubscribe attempt failed had
strand_unsubscribe/7 record the stranding and drop_unwanted_shards/3 erase it again
in the very next statement of the same reshard/1 call — recorded and discarded in one
breath. A vanishing shard handled ASYNCHRONOUSLY had its state.shards entry deleted
before the deferred {:channel_reconcile, ...} even attempted a send; if that later
attempt failed, strand_unsubscribe/7 recreated state.pending_unsubscribes[key] for a
key handle_info(:resubscribe, _) could never find again, because it walks
Map.keys(state.shards) specifically. Both silently contradicted this section's own
claim and the notice text's own promise ("picked back up on the next unconditional
resubscribe cycle") for the one shape of stranding most likely in practice: one on a
shard nobody wants any more, where nothing else will ever touch that key again to
surface the problem.
Fixed by no longer treating "vanishing" and "safe to drop" as the same moment.
drop_unwanted_shards/4 now leaves a vanishing key's state.shards entry (and its
pending_unsubscribes entry, untouched) alone whenever that key has a stranding
recorded OR was just handed to a deferred reconcile this very reshard/1 call — the
one case where a stranding might still be recorded a moment later, after this function
has already run. finalize_shard/4 and its deferred twin
maybe_finalize_vanished_shard/2 are the only places a vanished shard's bookkeeping is
now actually retired, and only at the moment either can know it is safe: right after an
unsubscribe attempt that leaves nothing pending. Until then the shard stays in
state.shards with symbols: [], which is what keeps it visible to
handle_info(:resubscribe, _)'s walk and to attempt_channel_reconcile/8's own retry
chain — a dropped shard's socket is still left running unmanaged either way, the
existing, unchanged tradeoff.
This answers "can state.pending_unsubscribes grow without bound": bounded by however
many shards — open OR vanished-but-not-yet-confirmed-clear — carry an unresolved
stranding, not by this module's all-time history of failures, because a shard's own
entry is retired only once its release is confirmed.
A second, independent gap on the same trace: an ORDINARY reconcile ignored a
standing stranding entirely. reconcile_shard_in_place/8 computed added/removed
from current -- wanted / wanted -- current alone — current being this module's
own optimistic belief, always set to the PREVIOUS wanted regardless of whether that
reconcile's own unsubscribe actually succeeded — never consulting
state.pending_unsubscribes. Only handle_info(:resubscribe, _)'s 60-second cycle
looked at it. A shard sitting at its own cap with one symbol stranded would accept a
plain additive subscribe for an unrelated new symbol on the very next ordinary
subscribe/2, unsubscribe/2 or update_symbols/2 call, in between ticks — the exact
"quiet overflow" DpCryptoManagement's own probe 2 describes, reached through the path
this whole unsubscribe-before-subscribe design was built to close, not around it. Both
clauses now fold Map.get(state.pending_unsubscribes, key, []) into current before
computing added/removed, so a standing stranding is retried — and any newly-wanted
symbol withheld behind it — on every reconcile that touches that shard, not only once
every resubscribe_interval_ms.
A stranded symbol re-added to the same shard while still pending is not a hazard,
because added is always the shard's WHOLE current set, never a delta. Ordinary churn
can plausibly move a symbol out (stranding its release) and back in before the next tick
retries it. Whatever the retried unsubscribe does to that symbol, the SAME reconcile's
subscribe — for the shard's full, current membership — puts it right back if it is
still wanted, in the same message, in order. At worst this costs one redundant
unsubscribe/resubscribe pair for that one symbol; it never leaves the shard short a
symbol it currently wants.
What is not built, and the honest reason: correlating a stranding to a socket
RECONNECT specifically. A socket that dies and reconnects gets a fresh session with the
venue — every stranded slot on the OLD session is released along with it, whether this
module notices or not. But Socket deliberately holds no shard-correlating identity in
its :link_up/:link_down notices (see this module's own handle_info({:dp_exchange, :coinbase, %Notice{}}, _) — every socket's notices arrive through the identical,
unattributed shape, matching Socket's own moduledoc: "Socket holds no book to key by
anything"), and WebSockex's own reconnect keeps the same pid, so pid identity cannot
substitute either. Building that correlation would mean widening Socket's own contract
— real, cross-module surface — for a case whose cost, left as is, is bounded and low: a
stale pending_unsubscribes entry retried against a fresh session sends one
unsubscribe for a symbol the venue no longer has subscribed, which this family's stated
idempotency assumption (FrameSender's own moduledoc: "subscribes are idempotent on
every venue in this family, so a duplicate is harmless") extends to as a working
assumption for unsubscribe too, though NOT independently measured against an
unsubscribed symbol specifically — an honestly-labelled gap, not a silent one. And
because that stale unsubscribe is immediately followed, in the SAME reconcile, by a
subscribe of the shard's whole current membership, a stale entry costs at most one wasted
frame and self-corrects the same tick — it cannot leave the shard under-subscribed. There
is also a practical reason the stale case rarely persists at all: whatever made the
earlier sends fail (a busy, backed-up socket) is usually exactly what a reconnect clears,
so the very next retry after a reconnect is likely to simply succeed and clear the entry
normally, not because the reconnect was detected, but because reconnecting tends to
un-wedge the condition that caused the stranding in the first place.
What transient_frame_failure?/1's real reach turns out to be, stated plainly rather
than assumed generous. Every reason Socket.unsubscribe/3 can return comes from
FrameSender alone — :send_timeout or {:send_exit, reason} — and both are classified
transient. Socket.unsubscribe/3 has no analogue of a subscribe's
{:credentials_required, channel}: it builds no JWT and checks no credentials, per its
own definition (contrast Socket.subscribe/4's subscription_message/3). So the
non-transient cond branch in handle_unsubscribe_failure/8 does not fire against the
real venue today — "permanently stranded" is reached, in practice, only by EXHAUSTING the
bounded retry chain against a socket that keeps answering :send_timeout or
{:send_exit, _} without ever dying outright (Process.alive?/1 staying true
throughout — a socket that is genuinely dead short-circuits every reconcile path before
it reaches this branch at all, matching every other dead-socket guard in this module).
The non-transient branch is kept anyway, wired to the identical stranding-and-retry
behaviour as an exhausted chain, for the same "do not assume a shape that happens to hold
today" reason handle_subscribe_failure/6 keeps its own — but it is not what this
mechanism was actually built to close in production; the retry-exhaustion branch is.
A refusal is not always a clean gate — observed, not explained
DpCryptoManagement's bisection above answers "accepted or refused" per n, but two of
their runs saw something this package has no account of: at n = 31–34 in one run,
books = 0 (a clean refusal); at n = 35–50 in the same run, 1,300–2,000 books
delivered alongside the rate_limited notice — a partially honoured oversized
subscription. A later run saw n = 31 refuse with 1,655 books delivered. None of this
moves the boundary — every n ≥ 31 refused and every n ≤ 30 did not, in both runs —
but it means "refused" can apparently be a spectrum rather than a gate. See
docs/reference/coinbase/level2-session-limit.md for the full, dated account; it is
recorded there as an observed venue characteristic with no explanation attached, not
rationalised into one.
At @level2_pairs_per_socket = 30 this package should never itself trigger a
partial refusal, and that reasoning was checked, not assumed: every subscribe this
module sends for a level2 shard carries at most 30 symbols — wanted_symbols per
shard is ≤ @level2_pairs_per_socket by construction — and the unsubscribe-before-
subscribe reconcile above (see "unsubscribe before subscribe") keeps every socket's
CONCURRENT count within that same bound at every instant of a reconcile, not only at
rest, regardless of which of Coinbase's two possible countings turns out to be real.
Nothing in this package's own behaviour asks for 31 or more at once. (These partial
refusals were themselves observed on SINGLE, oversized subscribes — the same shape as
the 2026-08-26 incident and the level2_pairs_per_socket option's own warning below,
neither of which this package's own behaviour triggers either — not on the cumulative
overage the 2026-09-07 probes above exercised. The two shapes are now known, 2026-09-07,
to behave alike on whether the socket survives a refusal — both do, see the option's own
warning above — but that does not extend to the partial-delivery spectrum this section
is about: the 2026-09-07 single-oversized re-test saw clean refusals, books=0, at
every size it tried, not the partial delivery this section describes, so the spectrum
observed here stays its own, separate, unexplained venue characteristic.) Should the
venue nonetheless
answer a rate_limited notice while some of that same shard's symbols are genuinely
delivering — the exact shape DpCryptoManagement observed — coverage/1 and
coverage_by_kind/1 need no special case to stay honest: both are built entirely from
state.delivering, which is populated only by handle_info({:dp_exchange, :coinbase, payload}, state) when an actual Types.Quote, Types.OrderBook or
Types.OrderBookDelta arrives, tagged with whichever symbol it carries.
handle_info({:dp_exchange, :coinbase, %Notice{}}, state) — the clause a
:rate_limited notice takes — never touches state.delivering at all. A symbol that
delivered a book is covered for :order_book whether or not its shard's subscribe was
also, separately, answered with a refusal notice; a symbol that delivered nothing stays
:not_covered regardless. The two facts were never coupled in the first place, which is
the same "coverage is observed, never intended" guarantee this moduledoc opens with,
just exercised by a venue behaviour this package did not anticipate rather than one it
designed for.
A timed-out subscribe used to be thrown away — now it is retried
FrameSender's own moduledoc says the whole point of turning a send_frame exit into
{:error, :send_timeout} is that "a slow socket becomes a failed batch, which a caller
can report and retry, rather than a dead connection", and that subscribes are idempotent
on every venue in this family, so a duplicate is harmless. This module used to log that
error and drop it — the retry half of the design was never wired, so a channel_subscribe
that lost the race against a level2 snapshot burst simply stayed unsubscribed until the
next unconditional resubscribe tick, which reproduces the identical busy-socket condition
and fails identically.
This is not hypothetical. A consumer running against a real ~400-symbol universe measured the exact inversion this predicts, across five boots over roughly 5.5 hours:
| state | quotes (ticker) | order_book (level2) |
|---|---|---|
| broken (4 boots) | ~5 / 406 | ~406 / 406, 11,000+ frames |
| healthy (1 boot) | 400 / 406 | 6 / 406 |
When level2 gets through broadly, its opening snapshot burst is what starves ticker;
when the venue refused most level2 subscriptions outright (its own per-session stream
limit — see above), ticker had the socket to itself and got everything. A lone
:send_timeout on a ticker subscribe was also observed directly in an earlier run.
Both are DpCryptoManagement's issue #22.
Classify before retrying — not every failure can be fixed by waiting
{:error, :send_timeout} and {:error, {:send_exit, reason}} are transient: the
socket was busy decoding a burst, or briefly gone, and the identical request can
reasonably succeed once it catches up. {:error, {:credentials_required, channel}} (see
Socket's subscription_message/3) is permanent — no amount of waiting supplies a
credential that was never given, and retrying it would only loop, so it fails loudly on
the first attempt and is never rescheduled.
The backoff waits out a socket's own burst, not another channel's
At the time this was written, a retry waited out the same busy-socket condition
@channel_spacing_ms existed to wait out between level2 and ticker sharing one
socket, so @subscribe_retry_delay_ms borrowed that value rather than guessing a second
number for the same underlying wait. @channel_spacing_ms is gone now that no socket
carries two channels — see "level2 gets its own, smaller sockets" above —
@subscribe_retry_delay_ms keeps its value (8_000ms) but stands on its own reasoning:
a socket can still be busy decoding its own just-subscribed burst (a level2 socket's
own snapshot, however small its shard), and the identical request can reasonably succeed
once that clears. @max_subscribe_retries is 2: one initial attempt plus two retries
is enough to survive one snapshot burst without turning a stuck socket into an unbounded
loop. See the constants' own comments for the arithmetic that keeps the whole retry
chain well inside a resubscribe cycle, so it can never stack frames against the
unconditional re-issue documented below.
Exhaustion is loud
A channel that never subscribed is exactly the invisible half-dead feed this whole issue
is about, and it used to surface as a Logger.warning a consumer had no facade-level way
to see. Giving up — whether because the failure was permanent or because retries ran out
— now also emits a Core.Notice of kind :coverage_change: those symbols will not
deliver this kind of data, which is exactly the fact coverage/1 and coverage_by_kind/1
need a consumer to go re-check rather than discover from a quiet chart.
A reconnect that does not resubscribe is a coverage collapse with no error
WebSockex reconnects a dropped socket on its own, and a bare reconnect leaves it connected and subscribed to nothing — silently, because a socket that is up and receiving nothing is not itself an error. That is a real, measured incident on this venue's own reference implementation: coverage decayed from full to the REST-poll floor over roughly forty minutes with the feed still reporting healthy, because nothing re-asked the venue for anything after the reconnect.
This coordinator re-issues every shard's subscriptions on a timer, unconditionally. Re-subscribing a channel the socket already carries costs one frame the venue ignores; not re-subscribing one it silently dropped costs the shard's whole coverage until someone notices a quiet chart.
Every shard beyond the first must open on its own tick, not the same one
shard_spacing_ms staggers shard opens relative to each other, not relative to a
fixed instant. A scope wide enough to need three or more shards — DpCryptoManagement's
issue #20, 406 symbols / 5 shards, filed against real production traffic — used to
schedule every shard past the first (the synchronous one) with the same fixed delay,
so all of them opened in the same instant: exactly the connect burst this module's own
design note above warns the venue answers with resets. Only the shard whose burst-mate
connections lost that race ever delivered a tick; coverage sat at whatever fraction of
one shard survived, indistinguishable from the outside from a quiet market. The
60-second unconditional resubscribe re-issued the same burst every minute. Both paths
now schedule each shard's turn position * shard_spacing_ms after the one before it —
see "shard_spacing_ms — a supervision option" below for what that value is, where it
comes from, and how a consumer can change it.
This staggering has to reach an already-open shard too, not only a brand-new
connection. reconcile_shard/7 used to receive the same delay reshard/1 computes
for it and drop it on the floor — every already-open shard update_symbols/2 touches in
one call had its level2 subscribe scheduled at the identical instant. That is not the
connect burst above (no new socket opens), but it is a related hazard: Socket. subscribe/4 blocks THIS GenServer — via FrameSender, up to WebSockex.send_frame/2's
5s window — for as long as its target socket takes to acknowledge, and several such
messages landing in this process's own mailbox together serialise into back-to-back
blocking sends, stalling coverage/1 and every other call to this Feed for as long as
the slowest one takes. Fixed the same way: delay now reaches reconcile_shard/7 and
staggers its frames exactly as it already staggered a new shard's.
shard_spacing_ms — a supervision option, so a test does not have to wait out a
production timer to prove staggering happened
30 in @default_level2_pairs_per_socket and 60_000 in
@default_resubscribe_interval_ms both became supervision options once they had a
reason a consumer might legitimately want to move them. shard_spacing_ms gets the
same treatment for a different reason: this package's own test suite needed one first.
Five of this file's own sharding tests exist to prove staggering happened — that a
connect burst is spread across ticks, that ticker shards are scheduled ahead of
level2 ones, that an already-open shard's reconcile is staggered too (the
"already-open shard" fix two sections up) — and every one of them, before this option
existed, could only prove that by actually waiting out @shard_spacing_ms in real
time: 45 of this suite's roughly 51 seconds, across five tests, one of them 15 seconds
on its own. That is not merely slow; a test that synchronises by sleeping out a real
production timer is the same shape of hazard wait_until/3 exists elsewhere in this
suite to avoid — passing locally, then flaking in a more loaded CI the timer was never
sized for. Feed.start_link/1 (via DpExchange.Coinbase.Supervisor's ordinary opts
pass-through — no Supervisor code change needed, the same as level2_pairs_per_socket)
now accepts shard_spacing_ms: n and every stagger this module schedules — a new
shard's connect, an already-open shard's reconcile, the unconditional resubscribe, the
derived floor under resubscribe_interval_ms (next_resubscribe_delay/1) — reads it
from state instead of the module attribute. A test drives it down to a few tens of
milliseconds and keeps every one of those assertions meaningful: the relative ordering
(shard 1 before shard 2, ticker before level2) and the fact that a real,
positive delay was scheduled at all — see each test's own comment for how it does that
without merely asserting the calls happened.
Validated the same way level2_pairs_per_socket is, on the axis that is
unconditionally nonsense. A negative value or a non-integer cannot schedule
Process.send_after/3 at all and is refused at init/1 with an ArgumentError — see
validate_shard_spacing_ms!/1. 0 is NOT refused: Process.send_after/3 accepts it
without complaint, and unlike a negative delay there is nothing mathematically broken
about "every shard opens in the same instant" — it is simply the connect burst this
whole file otherwise exists to avoid, which is the next paragraph's problem, not this
one.
This is also a knob pointed at a venue rate limit, and this package just spent one
investigation (level2_pairs_per_socket, above) learning what happens when a number
that should be the venue's own fact is instead a guess. Unlike level2's per-session
product ceiling, though, Coinbase does document a connect-rate number: the Advanced
Trade rate-limits page states "WebSocket connections ... are ... limited to 8 per
second per IP" (re-read 2026-09-06, the same pass that produced the level2 ceiling
above). This module opens one new socket per shard_spacing_ms tick, so that figure
converts directly into a floor: ceil(1_000 / 8) = 125ms. A consumer setting
something below that is asking this package's own connects alone to exceed a limit
Coinbase states outright, independent of whatever else shares that IP.
Below the floor is honoured, not refused — the same shape as level2_pairs_per_socket
above its own measured ceiling, and for a symmetric reason: this package has no
visibility into a consumer's own network position (a dedicated IP with headroom this
repo cannot see, say), so it does not get to assume the conservative number is the only
correct one for every consumer. What a sub-floor value gets instead is a loud
Logger.warning naming the documented floor, its source, and the concrete risk —
connects tighter than the venue's own stated per-IP rate risk the connect-burst resets
this moduledoc opens with — legible, not silently accepted, exactly the standard
level2_pairs_per_socket set above the measured 30.
The default is now 1_000, chosen rather than inherited — closing
docs/design/closed/2026-09-09_shard-spacing-headroom.md. @default_shard_spacing_ms
used to be 5_000, carried over unexamined from the reference fix this package
replaced, the same way @pairs_per_socket (100) still is — nobody had picked 5_000
FOR this venue; nobody had looked at it at all until the idea doc above asked where it
came from and the honest answer was "nowhere." That idea doc's own closing criteria
allowed reasoning from the documented rate-limits page alone, without a live probe, for
exactly this kind of value: a pure connect-rate number that already converts to an exact
floor (125ms, above) with nothing left to bisect. This package has not run a
shard_spacing_ms probe against the real venue and does not claim to; the reasoning
below is entirely documentation-derived.
1_000ms is one connection per second — an 8x margin under the documented floor, not the floor itself. The idea doc explicitly ruled out adopting125ms as the new default: no margin for scheduler jitter, GC pauses, or a consumer's own concurrent load sharing the same IP.500ms (a 4x margin) was also considered and set aside in favour of the larger margin below.- Chosen more conservatively than headroom against the documented rate alone would
require, because of a real, recent incident on a different venue in this family:
Webull's shards crash-looped this same week once abandoned sessions accumulated
against an undocumented five-connection ceiling. Coinbase's own rate-limits page
documents a RATE (8/second/IP), not a concurrency cap, and documents no concurrency cap
at all — but "no documented cap" is not "no cap," and a tighter stagger increases how
many of this module's own connects are opening, and therefore how many are mid-handshake,
in any short window — the same axis that bit Webull, on a different venue with a
different (undocumented) limit.
Socket's own live measurement method (docs/reference/coinbase/level2-session-limit.md) took on the order of a few seconds to establish one connection; at1_000ms spacing, only a handful of this module's own connects are ever simultaneously mid-handshake for the 406-symbol, 19-shard scope this moduledoc already discusses, against roughly two dozen that would be in flight at once at the bare125ms floor. This is a documented-rate-plus-precaution choice, not a measurement of Coinbase's own concurrency behaviour — this repo has neither measured nor been told Coinbase has any such ceiling; the precaution is carried over from a sibling venue's incident, not from anything Coinbase-specific. - For the 406-symbol scope above, boot-to-full-
level2-coverage drops from 90 seconds at the old default to 18 seconds at this one — a fivefold improvement, deliberately short of the roughly-twentyfold the bare floor would give, because outrunning the documented rate by the smallest possible margin was never the goal.
This is a behaviour change for every consumer that has not set shard_spacing_ms
explicitly: this feed now reaches full coverage noticeably faster after boot and after
any event that reopens shards. A consumer who wants the old, more conservative pacing
back can still ask for it: shard_spacing_ms: 5_000.
One spacing, several jobs — considered separately, kept as one number
Four places in this module read shard_spacing_ms, and a value right for one is not
automatically right for another, so each is worth checking rather than assumed:
- The initial connect stagger (
reshard/1'srestlist, above) andretry_missing_shards/1's reopen stagger both open brand-new sockets — a direct fit for the documented 8-connections-per-second-per-IP floor this section is built on. handle_info(:resubscribe, _)'s unconditional re-issue walk staggers frames onto ALREADY-open sockets — no new connection, so the connect-rate floor is not a direct fit. Two independent reasons still keep it on the same number rather than a faster one of its own: Coinbase's own rate-limits page states its 8/second/IP ceiling covers connects AND messages together (docs/reference/coinbase/level2-session-limit.md's own paraphrase, "an8-per-second-per-IP connect/message rate"), so a faster resubscribe cadence risks the same class of venue reaction a faster connect cadence does; and, independent of anything the venue does,Socket.subscribe/4blocks THISGenServerfor as long as its target socket takes to acknowledge (see "every shard beyond the first must open on its own tick," above), so spacing resubscribe frames apart protectscoverage/1and every other call to thisFeedfrom stalling behind a burst of blocking sends regardless of what the venue does with them.next_resubscribe_delay/1is not a fifth job at all — it is a derived floor that has to track whatever the resubscribe walk actually uses, or cycles overlap and stack frames behind each other (see that function's own comment). Splitting it onto a different constant than the walk it measures would just reintroduce the bug it exists to prevent.
Nothing above argues for resubscribing FASTER than the connect pace this section already settled on, so the two concerns never pull the value in different directions. One number does all four jobs honestly; splitting it would add a second constant with no case where the two would ever need to disagree.
The venue rewrites an aliased product id on delivery, and that has to be undone HERE
Measured live, 2026-09-05, against wss://advanced-trade-ws.coinbase.com:
subscribing ticker to ["XLM-USDC", "AVAX-USDC"] — sent exactly as asked, both real,
listed products — delivers every frame tagged XLM-USD and AVAX-USD. The venue's own
subscription acknowledgement even echoes the rewritten names back
("ticker" => ["XLM-USD", "AVAX-USD"]), not the ones actually sent. This is the venue's
own declared behaviour, not a guess: Rest.get_alias_map/1 reads the same public
/market/products catalogue this module already reaches through Rest.get_symbols/1
and Rest.list_instruments/1, and on this date 112 of the first 114 USDC products
carried a non-empty alias naming their -USD counterpart. A caller subscribed under
the alias form received nothing under the name it asked for while a name it never asked
for arrived instead — measured against a real 406-symbol consumer scope
(DpCryptoManagement's issue #22): 174 of 406 requested pairs delivered nothing, while
401 pairs never requested were decoded and stored.
Attribution lives here, not in Socket
Socket stays venue-mechanics-only: it decodes a frame and delivers a struct tagged
with whatever product_id the venue actually sent, exactly as it did before this fix.
Every consumer-facing rewrite happens in this module's handle_info({:dp_exchange, :coinbase, payload}, state), immediately before a delivered payload is recorded as
coverage and fanned out — because this is the one place that already holds wanted
(what the caller actually asked for) beside the delivered payload. Duplicating wanted
into Socket just to make the same decision twice would be a second place for the two
to disagree. Socket holds no book to key by anything — see its own moduledoc — so
this holds regardless of which channel delivered the frame (ticker via
deliver_ticker/3, or level2 via Socket's deliver_snapshot/4/deliver_delta/4)
since all three arrive here as the same {:dp_exchange, :coinbase, payload} shape and
every struct Socket sends carries :symbol.
Built once, from the venue's own catalogue, never from string-munging
Rest.get_alias_map/1 is the only source for this map — reusing the same
/market/products fetch get_symbols/1 and list_instruments/1 already make, per the
standing rule against a second way to ask. Munging -USDC into -USD would be exactly
the "nearby substitute" this family forbids, and would be wrong for any pair the venue
does not alias — nothing here assumes the suffix relationship holds in general.
It is scheduled once, asynchronously, the first time subscribe/3 or
update_symbols/2 is called (maybe_schedule_alias_map_fetch/1, gated on
alias_map_status: :unfetched so a second call never re-schedules it) — not from
init/1, so a Feed that is merely supervised and never asked to stream anything never
makes a network call, and not synchronously inside the triggering handle_call/3, so it
never competes with @call_timeout's socket-connect budget. Until it resolves,
state.alias_map is simply %{} — indistinguishable, by design, from "the venue
aliases nothing here", which resolves to the same safe fallback below.
The fetch has to wait, not fail — DpCryptoManagement's issue #26
Rest.get_alias_map/1 reached Core.HttpClient without rate_limit_blocking: true,
so it went through fail-fast check/3 rather than blocking acquire/3 — and this fetch
is scheduled off the first subscribe/3, which for any real consumer is boot, the
single most contended moment for their own rate limiter (universe discovery, catalogue
reads and market overviews all landing at once). It is scheduled into exactly the window
most likely to throttle it. One throttled call at that moment, on code that never
retried (see below, before this fix), disabled attribution for the life of the process.
Measured live: 406 pairs requested as -USDC, delivered as -USD, overlap 5 —
coverage_by_kind/1 and the consumer's own tracker each reporting a truthful, and
wildly different, count.
This is the third instance of one family-wide pattern — dp_exchange_robinhood's issue
#16, this package's own issue #23 sweep (which fixed every other REST call site in this
package and missed this one, because the alias-map fetch did not exist to audit when
that comment was written), and now this: a background call with nothing waiting on it,
failing instead of waiting, while Core.HttpClient's own error message names the fix in
its text ("callers that can wait should set rate_limit_blocking: true"). This call
site is exactly the caller that can: it runs off Process.send_after, nothing blocks on
its result, and its only job is to populate a cache before frames arrive. Waiting a
second here is free; failing is total. rate_limit_blocking: true is now set
unconditionally by default_alias_map_source/2, which also forwards :limiter,
:plug, :timeout, :retry_attempts, :retry_delay and :weight from this module's
own opts — the same allowlist shape Rest's own request pipeline uses — so a test can
exercise the real fetch pipeline end-to-end (a fake :plug response behind a real,
deterministic :limiter) rather than only ever exercising the alias_map_source
injection seam.
Classified and retried, the same way a channel subscribe already is
Blocking removes the self-throttle as a failure mode, but does not remove every
failure: the limiter's own bounded wait can still time out
({:exchange_error, _venue, :rate_limit_timeout} — the wait itself ran out, not a
refusal), and the fetch can still fail for reasons no amount of waiting fixes.
transient_alias_map_failure?/1 classifies it exactly the way
transient_subscribe_failure?/1 classifies a channel subscribe, and for the same
reason: not every failure can be fixed by retrying, so retrying one that can't only
delays an honest "this failed" and spends a retry budget a genuinely transient failure
needs.
A timed-out wait for the caller's own rate limiter is the one case treated as
transient — the identical request can reasonably succeed once the limiter's bucket has
drained further, which is the whole reason rate_limit_blocking: true exists here.
Everything else — an unrecognised response shape, a refused request, a raw or
unclassified reason (including whatever a test's own stand-in returns) — is treated as
permanent, matching transient_subscribe_failure?/1's own default-to-permanent stance
for anything not explicitly known to clear on its own. Retries are bounded
(@max_alias_map_retries, backed off by @alias_map_retry_delay_ms — both overridable
via opts for a test's benefit, the same shape as subscribe_retry_delay_ms above)
rather than looped forever, so a fetch that genuinely cannot succeed still gives up and
reports itself rather than retrying silently without end.
Exhausting the retries, or failing permanently on the first attempt, both leave
state.alias_map at %{} and alias_map_status: :unavailable — the same safe
fallback as before this fix, just reached only after a transient failure has been given
its fair chance to clear.
A failed fetch reports itself — and a late notice subscriber has to be able to hear it
A failed fetch reports itself, as a :data_quality notice to notice_subscribers —
"attribution is degraded and here is why" — never a silently guessed mapping.
attribution_targets/2 is the single fallback for every case where no wanted name
resolves — unfetched, failed, legitimately alias-free, or a frame arriving for a symbol
outside wanted altogether (in-flight just after an unsubscribe, or a raw test
send/2): deliver under whatever the venue actually sent, exactly the pre-fix
behaviour, rather than inventing a name.
The notice used to be unreceivable by construction — also DpCryptoManagement's issue
#26. The fetch is scheduled from subscribe/3; a consumer that calls
subscribe_notices/1 afterward — the ordinary sequence, since notices are naturally the
second thing a caller registers once it already knows it wants data — could register
only after the fetch had already failed and fanned out to zero subscribers. A notice
announcing a persistent degraded state that can only ever fire in the one window
before anyone could be listening for it is worse than no signal: it looks like a
working alarm that never rings.
Fixed by replay, not by moving the emission earlier — narrowing the window does not
close it, and this file already learned that lesson once with
next_resubscribe_delay/1's per-tick storm. state.alias_map_status and the reason
that produced it (state.alias_map_failure_reason) already persist for as long as the
condition holds, so {:subscribe_notices, subscriber} replays the identical notice to
that one newly-registered subscriber whenever it finds the state already :unavailable
— once per registration, never on a timer, and never re-sent to a subscriber who
already has it, so it cannot become the per-tick storm next_resubscribe_delay/1 was
already fixed for once. A consumer that registers before the fetch resolves sees the
ordinary fan-out, exactly as before; one that registers after sees the same notice,
late but not lost.
Both caller-facing names, when both are wanted
The venue treats XLM-USDC and XLM-USD as one market. If a caller subscribes to
both, both are entitled to every update — attribution_targets/2 resolves a delivered
id to every name in wanted that names the same market (its own id and, where the
catalogue says so, its alias), and the delivery loop sends one copy per resolved name.
This holds regardless of whether the venue itself echoes one frame or two per update for
a dual subscription — resolution runs per delivered frame against the full candidate
set, so two frames naming the same pair of wanted symbols do not double-deliver into a
name twice per market update; they each resolve to the same one-or-both names again.
coverage/1 needed no code change to become honest
Coverage was already whatever key delivering holds — see the moduledoc up top. Once
delivery is recorded under the caller's own requested name instead of the venue's
rewritten one, coverage/1 reports exactly what was asked for, by construction, with
nothing endpoint-specific added at the coverage/1 call site itself.
coverage_by_kind/1 — the same fact, split by what actually arrived
coverage/1 answers "is anything arriving for this symbol", and it answers that
question truthfully — but it asks nothing about which kind of payload showed up.
Types.Quote, Types.OrderBook and Types.OrderBookDelta all carry :symbol, so a
level2 book update (whether a full snapshot or an incremental delta) and a ticker
quote count identically toward delivering, and a symbol with one of the two dark
looks exactly like a symbol with both healthy.
That is not a hypothetical: level2 on this venue delivered upward of 11,000 frames
across 406 subscribed symbols while ticker stayed dark on all but a handful of them,
and coverage/1 still answered :stream for all 406 — correctly, by its own definition,
and useless for telling anyone that quotes had gone silent. Two separate
DpCryptoManagement issues (#20 and #22) sat unpinned for days because nothing in this
package's own observability could distinguish "everything is fine" from "the book is
fine and the ticker is dead". coverage_by_kind/1 exists to make that distinction
answerable without adding a second, differently-shaped API: it reports the same
observed-arrival fact coverage/1 reports, just partitioned by
DpExchange.Core.Capabilities.data_kind/0 instead of collapsed across it.
The kind is read off the payload's own struct — %Types.Quote{} is :quotes,
%Types.OrderBook{} and %Types.OrderBookDelta{} are both :order_book — never off a
channel name. Snapshot and delta share one kind deliberately: this question is "is
book data arriving", not "in what shape", and the struct type itself already tells a
caller which shape it is holding. level2 and ticker are this venue's words for its
own wire protocol and stop existing the moment a frame becomes a Core.Types.* struct;
coverage_by_kind/1 never sees them and could not leak them if it wanted to.
state.delivering therefore keys each symbol to a small map of kind => timestamp
rather than a single timestamp, so a symbol that has delivered both a quote and a book
update carries both kinds at once, and one going dark does not erase the other.
coverage_by_kind/1 folds that structure the other way — kind first, then symbol — to
match the shape DpExchange.Core.Venue.coverage_by_kind/1 promises.
Summary
Functions
Returns a specification to start this module under a supervisor.
coverage/1, split by which Core.Types.* kind actually arrived — see the moduledoc's
"coverage_by_kind/1" section for why coverage/1 alone could not tell "ticker dark,
book healthy" apart from "everything healthy".
The scope split into one list per ticker socket. level2 is chunked separately, at
its own, smaller and independently configurable size — see the moduledoc's "level2
gets its own, smaller sockets" and "level2_pairs_per_socket — a supervision option"
sections — and has no public function of its own, the same way it has no public
visibility anywhere else in this module.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec coverage(GenServer.server()) :: %{ required(String.t()) => :stream | :internal_poll | :not_covered }
@spec coverage_by_kind(GenServer.server()) :: %{ required(DpExchange.Core.Capabilities.data_kind()) => %{ required(String.t()) => :stream | :internal_poll | :not_covered } }
coverage/1, split by which Core.Types.* kind actually arrived — see the moduledoc's
"coverage_by_kind/1" section for why coverage/1 alone could not tell "ticker dark,
book healthy" apart from "everything healthy".
The scope split into one list per ticker socket. level2 is chunked separately, at
its own, smaller and independently configurable size — see the moduledoc's "level2
gets its own, smaller sockets" and "level2_pairs_per_socket — a supervision option"
sections — and has no public function of its own, the same way it has no public
visibility anywhere else in this module.
@spec start_link(keyword()) :: GenServer.on_start()
@spec subscribe(GenServer.server(), [String.t()], keyword()) :: :ok | {:error, term()}
@spec subscribe_notices( GenServer.server(), keyword() ) :: :ok
@spec unsubscribe(GenServer.server(), [String.t()]) :: :ok | {:error, term()}
@spec update_symbols(GenServer.server(), [String.t()]) :: :ok | {:error, term()}