DpExchange.Webull.Subscription (DpExchangeWebull v0.4.167)

Copy Markdown View Source

The HTTP half of subscribing — internal.

Webull steers an MQTT stream with REST calls. POST /market-data/streaming/subscribe names the session_id that the MQTT connection registered as its client id, and the broker begins publishing to that session.

Split across two protocols like this, the failure modes are unusual and worth naming:

  • A 200 here does not mean data is arriving. It means the venue accepted the request. Whether anything is published depends on the MQTT session being up and carrying the same id. That is why coverage/1 reports only what arrived — on this venue, "asked", "accepted" and "delivering" are three different moments.
  • A session id mismatch fails silently in the most expensive way: the HTTP call succeeds, the broker publishes to a session nobody is listening on, and the socket sits connected and idle. There is no error anywhere. One generated id, used for both, is the only defence.

Every call is signed — this venue has no anonymous endpoints.

sub_types is uppercase (SNAPSHOT, QUOTE, TICK) and is not the MQTT topic namespace. The subscribe request body's subtype field and the MQTT topics a connected session receives on (quote, snapshot, tick — lowercase, see streaming-api.md) look like the same vocabulary and are not: they are two different fields on two different protocols. Sending the lowercase topic names here got every subscribe rejected HTTP 417 UNSUPPORTED_SUB_TYPE — DpCryptoManagement's issue #19, filed right after #18 unblocked the request enough to reach this validation for the first time.

["SNAPSHOT", "QUOTE"] is confirmed live, the same pair the prior in-repo client accepted for months and issue #19 measured directly. The default is that pair plus TICK.

TICK is requested, because it is the one trade-tape route this venue documents for crypto. It joined the default on 2026-09-06, read from streaming-api.md's topic table ("Stocks, Futures and Crypto"). On 2026-09-29, with Socket reporting every way a tick can be dropped, a consumer on ~325 us-crypto symbols measured no Trade and no drop notice (dp-exchange-core issue #40), and it left the default. It came back on 2026-10-02 (dp_exchange_webull issue #7): the venue accepted the request carrying it, so asking costs nothing, while not asking guarantees a trade never arrives. Socket decodes every tick into a Core.Types.Trade, and reports any it cannot. :trades stays out of capabilities/0 until a run shows ticks arriving, because a declaration is a claim about what was measured.

#7 also asked whether that measurement could be trusted. The record answers both of its doubts:

  • TICK was in the measured request, and the venue accepted it. #40's own body records sub_types: ["SNAPSHOT", "QUOTE", "TICK"]. A refused sub-type fails the whole request (HTTP 417 UNSUPPORTED_SUB_TYPE, issue #19), so if TICK had been refused no quote would have arrived either. Quotes from that same request arrived, about 20,000 in 3 minutes.
  • The session was serving. The same 20,000 quotes in 3 minutes came on it. The "Permission grabbed" churn in that consumer's log was real, but it did not stop the session delivering every other topic it was subscribed to.

What the record lacks is the venue's response body to that subscribe, and a run on a session with no competing grab. Both need credentials this repo never holds. Requesting TICK by default makes every consumer's run that re-measurement: every way a tick can be dropped raises a :data_quality notice, so a run with no Trade and no notice is the measurement repeated.

Repeated on 2026-10-02, on 0.4.99, the first run with TICK back in the default: the request was accepted and quotes flowed, but about 6 minutes in a consumer's 1-minute BTC-USD and ETH-USD candles held zero volume, while Gemini's trades gave the same consumer real volume. No tick arrived. TICK stays requested, because asking costs nothing and catches the venue starting to publish one; :trades stays undeclared.

INVALID_SYMBOL names the offending symbols, and this module hands them back

Rejection here is per-request, not per-symbol: one symbol the venue's streaming category does not carry fails the entire batch, and the venue answers with {"error_code" => "INVALID_SYMBOL", "message" => "The symbols does not exist in the category. [SYM1, SYM2, ...]"} — measured from Feed's own resubscribe logs (DpCryptoManagement's issue #24): 17 of one consumer's 342 symbols named this way, every 60-second resubscribe tick, forever, because nothing downstream could act on the answer. Before this, that whole response collapsed into the generic {:exchange_error, :webull, "HTTP 417: ..."} string below — a caller could log it, and could not pattern-match a single symbol out of it.

invalid_symbols/1 parses the bracketed list out of message and this function converts every entry back to CANONICAL form before it returns — no venue-shaped symbol string is allowed to escape this module. Feed (not this module) decides what happens to a rejected symbol; this module's only job, same as :oversubscribed below, is to make the venue's answer something a caller can act on rather than only read.

A message the parser cannot attribute to any symbol is not treated as naming zero symbols. It falls through to the same opaque {:exchange_error, ...} shape a 417 with no recognisable list already produced — inventing an empty exclusion list from a rejection nobody could attribute would look like "nothing was rejected" to Feed, which is the one interpretation this response never supports.

Two more divergences between subscribe.md and this request, neither changed here

subscribe.md:140-196 marks category, grab, session_id and sub_types all required, and its category enum is ["US_STOCK", "US_ETF"] only — US_CRYPTO, which this body sends, is not a member of it. That is the SAME category-vs-page conflict the moduledoc's ["SNAPSHOT", "QUOTE"] note already resolves for sub_types, and it resolves the same way for the same reason: US_CRYPTO is confirmed live (the same DpCryptoManagement issue #19 read), so the wire stays US_CRYPTO and the page's narrower enum is recorded as wrong, or at least incomplete, rather than acted on.

grab ("Whether to grab snapshot data, true/false") is different in kind: it is a real gap, not a resolved one. The page marks it required and gives it no documented default, and its one-line description says nothing about what "grab" means beyond the two literal values — there is no worked example, no note on what a subscribe without it does instead, and nothing elsewhere in this repository's captured pages defines the term further. Sending either "true" or "false" here would be guessing which of two values is safe on a required field this package cannot read the semantics of, which is exactly the substitution CLAUDE.md's fail-closed rule exists to prevent — and the live evidence says the guess is not even needed: ["SNAPSHOT", "QUOTE"] has subscribed successfully for months without grab ever being sent (the same issue #19 measurement). So the wire is left unchanged, grab omitted, and this paragraph is the record of the divergence rather than a silent one: measured-live behaviour (works, without grab) against the documentation (requires it, undefined how). Sending a guessed value on a working request risks breaking it to satisfy a schema this package cannot verify; that trade is refused the same way a request would be if the direction were reversed.

Summary

Functions

Whether opts[:sub_types] is absent or a non-empty list of strings: the check every subscribe and unsubscribe request makes before anything is built, for the facade and DpExchange.Webull.Fake to make at the boundary too.

Starts publication for symbols on the MQTT session registered under session_id.

Stops publication for symbols. Refuses the same malformed symbols as subscribe/3.

Functions

check_sub_types(opts)

@spec check_sub_types(keyword()) :: :ok | {:error, {:invalid_sub_types, term()}}

Whether opts[:sub_types] is absent or a non-empty list of strings: the check every subscribe and unsubscribe request makes before anything is built, for the facade and DpExchange.Webull.Fake to make at the boundary too.

Anything else is {:error, {:invalid_sub_types, value}}. It used to be found only when the request was about to go out: after a shard had been opened for the call, and on the background reconcile path not at all until the CONNACK, so a typo cost a socket and answered late. The value is not judged against the venue's QUOTE / SNAPSHOT / TICK list: the venue owns that vocabulary and answers an unknown member UNSUPPORTED_SUB_TYPE.

subscribe(session_id, symbols, opts)

@spec subscribe(String.t(), term(), keyword()) :: :ok | {:error, term()}

Starts publication for symbols on the MQTT session registered under session_id.

An empty symbol list is :ok without a request: asking the venue to subscribe to nothing spends a call from a budget this venue is already the tightest on.

symbols that is not a list of non-empty strings is {:error, {:invalid_symbols, symbols}} with no request made, rather than a raise in the caller's process.

unsubscribe(session_id, symbols, opts)

@spec unsubscribe(String.t(), term(), keyword()) :: :ok | {:error, term()}

Stops publication for symbols. Refuses the same malformed symbols as subscribe/3.