# Changelog Completed roadmap tasks. For upcoming work, see [ROADMAP.md](ROADMAP.md). --- ## [Unreleased] ### 2026-04-20: Task 106 β€” Add `LICENSE` file **Completed** | [D:1/B:3/U:4 β†’ Eff:3.5] 🎯 **What was done:** - `LICENSE` β€” standard MIT text, copyright "2026 ZenHive" to match `package.links.GitHub`. Previously `mix.exs` declared `licenses: ["MIT"]` but the repo and Hex tarball shipped no `LICENSE` file; the pre-publish bug hunt flagged this as hygiene miss (some mirrors and agent-facing scanners treat the missing file as an unlicensed signal even when metadata says otherwise). **Key decisions:** - Single-file addition, no `mix.exs` changes needed β€” Hex auto-includes `LICENSE` in the tarball. ### 2026-04-20: Task 107 β€” README "Known Caveats" (has?/2, :custom signing, no-testnet list) **Completed** | [D:1/B:5/U:6 β†’ Eff:5.5] 🎯 **What was done:** - `README.md` β€” new **Known Caveats** subsection between "Two Contracts" and "Discovery" surfaces three consumer-facing gotchas that previously lived only in `CLAUDE.md`: 1. **`has?/2` is advertising, not a dispatch guarantee.** 2026-04-20 pre-publish audit counted 116 gaps across 23 priority-tier exchanges where `has: true` has no endpoint mapping. This matches CCXT JS semantics (endpoint map is the dispatch gate, `has` is advertising), but consumers relying on `has?/2` introspection got no warning that dispatch might return `{:error, :not_supported}`. README now points them at `CCXT..__unified_endpoints__/0` as the ground truth. 2. **`:custom` signing caveat for `derive`, `hyperliquid`, `lighter`.** Those three classify as `:custom` and don't ship with a bundled signer β€” private endpoints raise `ArgumentError` unless the caller provides `custom_module:` implementing `CCXT.Signing.Behaviour`. README now includes the example config. Upstream convergence gated on ccxt_extract 🎁 **10-exotic** (tracked in ROADMAP "⏸ Deferred β€” Awaiting Upstream Signing/Request Contract"). 3. **Seven exchanges have no testnet.** `aster`, `bitfinex`, `htx`, `huobi`, `kraken`, `kucoin`, `kucoinfutures` β€” `Exchange.new(id, sandbox: true)` cleanly returns `{:error, :no_testnet_data}` (documented behavior from T61). **Key decisions:** - **Promote, don't duplicate.** Consumers don't read `CLAUDE.md`; agents reading the hex-published docs only see `README.md`. Promotion into README is the shortest path to making these caveats visible at the surface most consumers actually hit. - **No moduledoc change on `CCXT.has?/2` this pass.** Original T107 description also proposed a one-line guard in the `CCXT.has?/2` moduledoc pointing at `__unified_endpoints__/0`. Deferred to a follow-up β€” the README caveat is the load-bearing surface for agents; moduledoc updates aren't cache-invalidated between hex publishes, so they can ride in the next feature release without urgency. - **Zero runtime change.** Docs-only; no code touched. ## [0.6.1] - 2026-04-20 First publishable Hex release. No functional changes over `0.6.0` β€” the `0.6.0` git tag was cut but never publishable because the pre-reduction spec corpus sat over Hex's size cap. Upstream ccxt_extract T116 + T117 and client-side T105 (this release) reduced `priv/specs/json/output/` from ~279 MB to ~16 MB; the built tarball is ~2 MB. ### 2026-04-20: Task 105 β€” Migrate SymbolResolver to `runtime.symbols_index` (schema 3.0.0 adoption) Unblocks the first Hex publish (`0.6.1`). Pre-reduction `priv/specs/json/output/` totaled ~279 MB uncompressed β€” well over Hex's 128 MB cap β€” so the `0.6.0` git tag was never publishable. Upstream ccxt_extract shipped the two-task fix: T116 (compact JSON at write time) and T117 (schema 3.0.0 prune β€” `runtime.markets` β†’ compact `runtime.symbols_index`, `structure.parse_methods` + `structure.ws_methods` dropped from emission). Combined impact brings the corpus to ~16 MB, with substantial headroom under the cap. **What was done:** - `lib/ccxt/spec.ex` β€” `@schema_major` bumped 2 β†’ 3; error messages and guards interpolate the attribute so they update in place. - `test/support/test_generator/symbol_resolver.ex` β€” single live reader of the dropped `runtime.markets.markets` path migrated to `runtime.symbols_index`. The value shape is identical (`%{"BTC/USDT" => %{"spot" => true, ...}}`), so downstream `find_symbol/1` + `first_market/2` semantics (preference-list walk, `data["spot"] == true` check, fallback to first sorted key) are unchanged. Moduledoc updated. - `test/support/test_generator/symbol_public_endpoint_probe.ex` + `test/ccxt/symbol_public_endpoint_probe_test.exs` β€” moduledoc references updated. - `test/ccxt/spec_test.exs` β€” removed the two presence asserts for `structure.parse_methods` and `structure.ws_methods` (dropped fields). The strip-methods check, `sign_method` preservation, `runtime.describe` presence, and deribit small-exchange load tests remain valid against schema 3.0.0. - `lib/ccxt/ws/config.ex` β€” moduledoc no longer points at `structure.ws_methods` as the future source of WS URL templates; now cites the upstream ccxt_extract follow-up that will emit `urls.api.ws.*` directly. - `priv/specs/json/output/` β€” regenerated against upstream schema 3.0.0 via `mix ccxt_extract.update --tier1 --tier2 --dex` (writes to ccxt_extract's own `priv/output/`; rsync to the client tree). `exchange_v2.json` replaced by `exchange_v3.json`. Corpus 279 MB β†’ 16 MB. - `CLAUDE.md` β€” Key Modules entry for `CCXT.Spec` updated from "schema major 2" to "schema major 3". **Key decisions:** - **Migration is zero-logic.** Symbols in the upstream `symbols_index` use the same `"BTC/USDT"` keys and the same `{"spot" => true, "swap" => true}` type flags the old `runtime.markets.markets` snapshots exposed. The only code change needed was the `get_in` path. - **Compact by default upstream, not here.** The Hex size cap is the forcing function; compact JSON was adopted upstream (T116) rather than adding a ccxt_client-side minimization step. Per CLAUDE.md's STOP rule: patch upstream, not consumer-side. - **No 0.6.1 tag in this task.** Tagging + Hex publish is a separate operator step downstream of CI on the migration. - **CLAUDE.md Quick Commands recipe rewritten.** Upstream changed `mix ccxt_extract.update --output` from "final output dir" to "priv root override" at schema 3.0.0; the flag now expects a `ccxt/ts/src/` tree under the target. Recipe now runs in-tree (no `--output`) and rsyncs `priv/output/` into the client. Inline note in CLAUDE.md documents the workaround. ## [0.6.0] - 2026-04-19 ### 2026-04-19: Task 104 β€” Stale scope docstrings Discovered via tidewave `CCXT.describe()` introspection during the 0.6.0 release pass: several moduledocs and comments still advertised "110 exchange client modules" / "41/110 exchanges" / "4 URL patterns across 110 exchanges" / `exchange_count #=> 110`, predating the 2026-04-15 tier rebase that narrowed the default spec set to Tier 1 + Tier 2 + DEX (23 exchanges in `CCXT.Registry.exchanges/0`). The CCXT moduledoc string is user-visible through `CCXT.describe()` output, so agents got a misleading scope narrative. **What was done:** - `lib/ccxt.ex` β€” moduledoc rewrites "110 exchange client modules" to describe the priority-tier scope and the `--exchange ` opt-in for tier3. - `lib/ccxt/dispatch.ex` β€” drop the brittle `41/110` path-interpolation count and the `4 URL patterns across 110 exchanges` claim; replace with scope-agnostic phrasing. - `lib/mix/tasks/ccxt.classify_signing.ex` β€” moduledoc updated from "all 110 exchange specs" to "every compiled exchange spec (priority tiers by default)". - `lib/ccxt/spec.ex` β€” `load_manifest!/0` example output flipped `#=> 110` β†’ `#=> 23` (matches what the current manifest actually emits). - `lib/ccxt/error.ex` β€” trimmed "110+ exchanges" to "every configured exchange". No functional change; purely narrative correctness. Folded into 0.6.0 because one of the affected strings ships as the primary library-overview surface to agents. ### 2026-04-19: Task 101 β€” Fix aster WS public URL (combined-stream β†’ ws endpoint) Aster's `public_url` was `wss://sstream.asterdex.com/stream` β€” the Binance-combined-stream endpoint, which expects stream names in the URL query (`?streams=btcusdt@ticker/...`) and wraps frames as `{"stream": ..., "data": ...}`. Our `:method_subscribe` pattern emits dynamic `{"method": "SUBSCRIBE", "params": [...]}` envelopes, which `/stream` does not accept post-connect. Flipped to `wss://sstream.asterdex.com/ws` (the dynamic-SUBSCRIBE endpoint), matching aster's already-correct `private_url` and Binance's own config choice. Verified via live tidewave smoke: subscribe acks (`id: 1, result: nil`) and live `depthUpdate` + `aggTrade` frames stream on BTCUSDT within seconds. Closes the last open defect from the post-T94 smoke audit. ### 2026-04-19: Task 61 β€” Adopt `runtime.testnet_urls` from upstream schema 2.4.0 Replaces `Exchange.new/2`'s ad-hoc `describe.urls.test` lookup with the structured `runtime.testnet_urls` field shipped by upstream ccxt_extract Task 100. The previous flow silently fell through to production URLs whenever `urls.test` was absent β€” true for aster, kraken, kucoin today and any future "no-testnet" exchange. Sandbox callers got prod URLs without warning, which combined with the aster "production keys against an empty account" caveat made the entire sandbox path a quiet footgun. T61 turns the gap into an explicit error. **What was done:** - `lib/ccxt/exchange.ex` β€” `pick_url_set/2` removed; new `resolve_base_urls/4` clause-routes on `(sandbox?, testnet_urls)`: - `sandbox: false` β€” production path unchanged (`urls.api` β†’ `resolve_urls/3` with `{hostname}` interpolation). - `pattern: "separate_host"` β€” uses `testnet_urls.urls` directly (already `{hostname}`-resolved upstream per `testnet_urls_shape_valid` invariant); skips `resolve_urls/3` to avoid double-interpolation. - `pattern: "sandbox_flag"` β€” keeps prod base URLs; the flag-only branch is unexercised by today's priority set (every `sandbox_flag` exchange also carries `separate_host` URLs) but specified for completeness. - `pattern: "none"` β€” returns `{:error, :no_testnet_data}`. `Exchange.new!/2` raises `ArgumentError` with `"no testnet data in spec ..."` via a new `format_error/1` clause. - `sandbox_flag_field` (typically `"sandboxMode"`) propagates into `exchange.options` independently of `pattern` β€” okx/gate/hyperliquid/binance carry both a separate host AND the runtime flag. Today nothing in `ccxt_client` consumes `options["sandboxMode"]`, but the propagation is the contract; future signing or dispatch code can read it without re-deriving from spec. - `priv/specs/json/output/*.json` β€” priority-tier specs synced from `ccxt_extract/priv/output/` to schema 2.4.0. Manifest and report files left at their previous client-local state (those are not consumed by `runtime.testnet_urls` adoption). - `test/ccxt/exchange_test.exs` β€” four new tests cover `sandbox_flag` propagation (okx), nested-shape preservation (gate), `pattern: "none"` refusal (aster `{:error, :no_testnet_data}` + `Exchange.new!` raise), and the no-sandbox path on a `pattern: "none"` exchange (aster constructs cleanly without `sandboxMode` in options). - `test/support/test_generator/raw_endpoint_probe_config.ex` β€” aster caveat docstring now documents the structural surface (`pattern == "none"` raises) instead of the documentation-only "production keys against an empty account" warning. `@treat_post_as_safe` unchanged: aster stays out, so its POSTs remain `:dangerous`-tagged behind `--include dangerous`. **Why `urls.test` interpolation is skipped:** upstream `testnet_urls.urls` is `{hostname}`-resolved at extract time; the `testnet_urls_shape_valid` contract invariant forbids residual `{hostname}` placeholders. Re-running `interpolate_hostname/2` on already-resolved strings is a no-op today but would mask a future upstream bug. **Why `sandbox_flag_field` is a per-exchange options merge:** matches CCXT's `setSandboxMode(true)` flag-flip semantic. Several exchanges (okx, gate) handle sandbox mode by reading `options["sandboxMode"]` at signing time rather than swapping URLs entirely. The flag-merge keeps the door open without requiring downstream wiring to land in the same PR. **Aster impact:** every aster private/dangerous integration probe now flunks loudly with the `:no_testnet_data` message instead of running against production keys with the previous `:dangerous` tag as the only safety net. The dangerous tag stays β€” belt-and-braces. The structural refusal is the new primary safeguard. **Files changed:** - `lib/ccxt/exchange.ex` - `priv/specs/json/output/*.json` (mechanical resync to schema 2.4.0) - `test/ccxt/exchange_test.exs` - `test/support/test_generator/raw_endpoint_probe_config.ex` - `CLAUDE.md`, `ROADMAP.md`, `CHANGELOG.md` **Verification:** `mix test.json --quiet test/ccxt/exchange_test.exs` green; full `mix test.json --quiet` green (no regressions). Spot-check via `iex -S mix`: bybit sandbox returns `https://api-testnet.bybit.com`; okx sandbox returns `options["sandboxMode"] == true`; aster sandbox returns `{:error, :no_testnet_data}` and raises with "no testnet data" via `new!`. ### 2026-04-19: Task 52 β€” Section visibility (retroactive closeout) Mechanical closeout. The implementation shipped in concert with T49 / T64 (commits around the schema 1.7.1 adoption window): `private_endpoint?/1` removed from `lib/ccxt/dispatch.ex`, `:authenticated` boolean stamped onto each endpoint config at compile time by `CCXT.Exchange.section_authenticated?/2` reading `structure.authenticated_sections`. Detail entry under "Task 52: Section Visibility Config βœ…" further down this file (Phase block, line ~900) was written at the time but the ROADMAP row was never flipped β€” this dated note exists so the rolling Recently Completed window in `ROADMAP.md` has a stable anchor and the cross-instance trail records when the closeout happened. **Files changed:** `ROADMAP.md`, `CHANGELOG.md`. No code, no tests β€” both were already in their final state. **Verification:** `lib/ccxt/dispatch.ex:68` shows the spec-driven flag is the only authentication-routing input. `test/ccxt/exchange_generator_test.exs:61, 378, 397` cover the spec-driven flag for Bybit (real spec), `fapiPrivate` (non-"private"-substring case), and the omitted-list default. ### 2026-04-19: Task 102 β€” Remove orphaned CCXT.WS.Config entries (bingx / bitget / mexc) Closes the WS-config/Registry inconsistency surfaced by the post-T94 live tidewave smoke audit. `CCXT.WS.Config` declared `bingx`, `bitget`, and `mexc` among its entries, but none of the three have spec files under the current `--tier1 --tier2 --dex` extraction scope, so `CCXT.Registry` compiles no module for any of them. The effect: `CCXT.Exchange.new!("bingx")` raised, which made `CCXT.WS.connect/3` for those exchanges impossible despite the config implying otherwise. **Decision β€” remove rather than add:** adding the three back to Registry would require new spec files, which is ccxt_extract territory (per CLAUDE.md "STOP: ccxt_extract Is a Separate Project"). Consumers who want any of them can run `mix ccxt_extract.update --exchange ` and then reinstate the WS.Config entry alongside the generated Registry module. The lost code is a dozen lines in a single map literal β€” reinstating is cheap. **What was done:** - `lib/ccxt/ws/config.ex` β€” dropped the `"bingx"`, `"bitget"`, and `"mexc"` entries from `@configs`. Updated moduledoc header: the opener no longer claims "17 exchanges," and a new "Count: 13 exchanges" line explicitly documents that every entry is Registry-reachable. The "Known URL limitations" bullets for `mexc` and `bingx` were dropped; the `aster` bullet stayed (still configured, still T101-flagged). - `CLAUDE.md` β€” the `CCXT.WS` and `CCXT.WS.Config` rows in the WebSocket module table now say "13 configured, all Registry-reachable" (was "16 configured, 13 Registry-reachable (bingx/bitget/mexc orphaned β†’ T102)"). - `ROADMAP.md` β€” T102 moved from the Active Track table into Recently Completed; suggested-execution-order paragraph updated. **Not done (by design):** - Did not rewrite historical T94 / post-T94-audit CHANGELOG entries β€” those are accurate snapshots of the state at those respective dates (16 configured / 13 reachable / 13 new-in-T94). Rewriting historicals loses the record of how counts moved over time. - Did not extend `CCXT.Registry` or touch `priv/specs/json/output/` β€” both are ccxt_extract concerns. **Files changed:** - `lib/ccxt/ws/config.ex` - `CLAUDE.md`, `ROADMAP.md`, `CHANGELOG.md` **Verification:** `mix test.json --quiet test/ccxt/ws/` green (172 tests, no regressions β€” the three removed keys were never referenced by any test config lookup). Full suite runs in the T103 bundle below. ### 2026-04-19: Task 103 β€” Honor allow_4xx on IntegrationHelper error branch T83's `RawEndpointProbe` passes `allow_4xx: true` into both `assert_public_response/3` and `assert_private_response/3` expecting "4xx means signing + URL + dispatch + error decoding all worked; the exchange rejected the request semantically, which is good evidence the pipeline is correct." Before this change, the opt was only read inside `validate_body!/3` β€” on the `{:ok, %{status, body}}` raw-envelope arm. But normalized 4xx responses exit `CCXT.HTTP` as `{:error, %CCXT.Error{}}`, which bypassed that arm entirely. Both `assert_public_response/3`'s error branch and `handle_private_error/3` flunked regardless of the opt, so T83 public/private/dangerous probes were effectively `allow_4xx: false` on the error path. **What was done:** - `lib/ccxt/error.ex` β€” added `http_status: non_neg_integer() | nil` to the `%CCXT.Error{}` type and `defexception`. Default `nil` is backward-compatible under struct construction and pattern-matching (no existing code pattern-matches on an exhaustive field set). - `lib/ccxt/http.ex` β€” `normalize_error/3` now threads the HTTP status into every returned `%CCXT.Error{}`. A small local `with_http_status/2` helper patches the struct after each `Error.*` constructor call, avoiding signature changes to the error constructors. Covers all four clauses: 429, 401/403, map-body status, and fall-through non-map body. - `test/support/integration_helper.ex` β€” - Extracted the public helper's error branch into `handle_public_error/3`, mirroring the private helper's structure. Added a shared `http_4xx?(err, opts)` predicate: `err.http_status in 400..499 and Keyword.get(opts, :allow_4xx, false)`. When true on either path, the error is logged inconclusive and `:ok` is returned, matching the semantics the probe intended. - `handle_private_error/3` got the same `http_4xx?` clause added before the final flunk. - Behavior for callers that don't pass `:allow_4xx` (T40 `PublicEndpointProbe`, T39 `UnifiedMethodIntegrationProbe`, T79 `SymbolPublicEndpointProbe`) is unchanged β€” they still flunk on 4xx-derived errors. - `test/ccxt/integration_helper_test.exs` β€” new regression test. Covers the matrix: public vs private Γ— `allow_4xx: true` vs `false` Γ— 4xx status, plus the 5xx-with-opt-on negative (still flunks β€” the opt is specifically 4xx-scoped). Constructs `%CCXT.Error{}` values directly; no live HTTP needed. Uses `ExUnit.CaptureLog` to swallow the inconclusive log line. **Design note:** chose an explicit `http_status` field rather than `err.code in 400..499`. Exchange body codes often live in `err.code` as integers (Bybit `retCode`, Binance signed negatives), and they can coincidentally land in the 400–499 range. An explicit field avoids the collision and makes the discriminator unambiguous for future callers (e.g., a future retry classifier that wants "HTTP-layer 5xx" distinct from "exchange body `retCode: 50000`"). **Files changed:** - `lib/ccxt/error.ex` - `lib/ccxt/http.ex` - `test/support/integration_helper.ex` - `test/ccxt/integration_helper_test.exs` β€” new. - `CHANGELOG.md`, `ROADMAP.md` β€” doc updates. **Verification:** full `mix test.json --quiet` green; `mix dialyzer.json --quiet --summary-only` clean. New test file exercises five assertions; see file for the matrix. ### 2026-04-19: Bundle A β€” WS subscription pattern shape fixes (T98 + T99 + T100) Closes the three pattern-module defects that the post-T94 live tidewave smoke audit surfaced. All three modules were ported from `ccxt_client_bak` under a string-only channel contract (`["ticker", "book"]`) but the new caller convention passes **structured maps** (`%{"channel" => "ticker", "symbol" => "tBTCUSD"}`). The port never added map-input handling, so frames leaving the wire didn't match exchange-docs canonical shapes β€” live exchanges rejected them (bitfinex `10300`, gate `code:2 Unknown channel`, kraken `Channel field must be a string`) or the caller hung for 5 s (htx). **What was done:** - `CCXT.WS.Subscription.EventSubscribe` (T98) β€” added a map-passthrough branch in `build/3`. For a single-map channel list, the caller's map is merged alongside `%{op_field => "subscribe"}`, bypassing `args_field`/`args_format`. String-channel callers fall through to the unchanged bak-style framing, so the existing dispatcher tests and the three canary exchanges (bybit/deribit/okx) keep working. Multiple maps per call return `{:error, :multiple_maps_not_supported}` explicitly rather than silently malforming. - `CCXT.WS.Subscription.MethodParams` (T99) β€” same pattern: single-map input becomes the `params` object directly, aligning with Kraken v2's `{"method":"subscribe","params":{"channel":"ticker","symbol":[...]}}` contract. String-channel input preserved for legacy callers. - `CCXT.WS.Subscription.SubBased` (T100) β€” removed the `"id"` field from emitted frames and dropped `id_prefix` config handling. zen_websocket's `RequestCorrelator.extract_id/1` converts `send_message/2` to a correlated `GenServer.call` whenever it sees a non-nil `"id"`; HTX's subscribe ack is fire-and-forget and doesn't round-trip through the correlator, so any `id` field hung the caller until timeout. Frames now return immediate `:ok` β€” verified live: htx subscribe went from 5000 ms hang to 5 ms success with ticker frames streaming. - Tests: map-input test cases added to `event_subscribe_test.exs`, `method_params_test.exs`, and `sub_based_test.exs`, all using real `CCXT.WS.Config.for_exchange/1` configs so they regress against `config.ex` drift rather than hand-rolled test configs. The existing id-presence assertions in `sub_based_test.exs` flipped to id-absence. **Live tidewave smoke verification (post-fix, 2026-04-19):** - **bitfinex** `subscribe(ws, [%{"channel" => "ticker", "symbol" => "tBTCUSD"}])` β†’ `"subscribed"` ack with `pair: "BTCUSD"`, ticker data frames arriving. - **gate** `subscribe(ws, [%{"channel" => "spot.tickers", "payload" => ["BTC_USDT"]}])` β†’ `%{"event" => "subscribe", "result" => %{"status" => "success"}}`. - **kraken** `subscribe(ws, [%{"channel" => "ticker", "symbol" => ["BTC/USD"]}])` β†’ `"status"` welcome frame with `system: "online"`, no `"Channel field must be a string"` rejection. - **htx** `subscribe(ws, ["market.btcusdt.ticker"])` β†’ `:ok` in 5 ms (was 5000 ms GenServer.call timeout); inbound gzip-compressed ticker frames streaming. **Caller contract (documented on `CCXT.WS.subscribe/3` via the pattern modules' moduledocs):** channel lists accept both `String.t()` and `map()` elements. Plain strings route through each pattern's bak-style framing (unchanged). A single map is spread alongside the action field (T98/T99) or passed through as the params object (T99). Multiple maps per call are not supported for the single-envelope patterns covered here. **Out of scope / intentionally left:** - T101 (aster WS URL/shape β€” DEX, different failure mode), T102 (orphan bingx/bitget/mexc Config entries β€” no Registry modules). - T97 per-exchange channel-name formatters β€” still gated on upstream `structure.ws_methods.channel_templates`. - No caller-facing API change. `CCXT.WS.subscribe/3`'s `@spec` already accepts `[String.t() | map()]`; the dual-shape contract is now honored at the pattern layer. ### 2026-04-19: Task 83 β€” Raw-endpoint integration probe Ships `CCXT.Test.Generator.RawEndpointProbe` + `CCXT.Test.Generator.RawEndpointProbe.Config` and resolves the long-standing T40 `:allow_4xx` TODO. Extends integration coverage from "one zero-arg unified method per exchange" (T40's 17-test scope) to "every generated endpoint on every priority-tier module that the probe can legally invoke." **What was done:** - New `CCXT.Test.Generator.RawEndpointProbe` β€” compile-time macro iterating each Registry'd module's `__endpoints__/0`. For each endpoint config, applies gates: (a) path-param resolvability via `RawEndpointProbe.Config.path_param_defaults/1`, (b) HTTP method β€” GET auto-emits; POST/PUT/DELETE tagged `:dangerous` by default unless the exchange opts into `treat_post_as_safe?/1` (read-style POST exchanges like `derive`, `hyperliquid`), (c) authentication β€” private endpoints need creds registered AND non-`:custom` signing (matches T67/T39 precedent). Endpoints failing any gate are skipped silently (no flunks, no noise). - Test emission: module-level `@moduletag :network + :raw`; per-test `:exchange_` + (`:public` | `:private` | `:dangerous`). Delegates to `CCXT.IntegrationHelper.assert_public_response/3` or `assert_private_response/3` with `allow_4xx: true` and the standard opt-gates (`:allow_not_found`, `:allow_invalid_order`, `:allow_no_position`). - New `CCXT.Test.Generator.RawEndpointProbe.Config` β€” seeded empty (`@path_param_defaults = %{}`) + `treat_post_as_safe = [derive, hyperliquid]`. Path-param coverage grows opt-in per exchange. `unresolved_params/2` helper used at compile time to skip endpoints with `{curly}` placeholders not covered by defaults. - `CCXT.IntegrationHelper` β€” `assert_public_response/3` and `assert_private_response/3` now accept `:allow_4xx` option. When enabled, `validate_body!/3` treats 400 ≀ status < 500 as an acceptable outcome (proves signing + URL + dispatch + error-decoding all reached the exchange; the exchange simply rejected the request semantically). Default stays `false` β€” existing T40/T39/T79/T67 callers unaffected. Resolves the pre-existing `TODO(Task 40)` at `integration_helper.ex:267`. - New thin consumer `test/ccxt/raw_endpoint_probe_test.exs` β€” `use CCXT.Test.Generator.RawEndpointProbe`. **Scope at compile time:** covers every Registry'd priority-tier exchange's generated endpoint surface, partitioned into public (auto-run), private (creds-gated), and dangerous (write-path, opt-in). Exchange-specific coverage grows as path-param defaults and testnet creds come online. Inspect the live selection any time via `CCXT.Test.Generator.RawEndpointProbe.__collect_for_inspection__/0`. **Running:** ```bash mix test.json --quiet --include network --only raw # all public raw mix test.json --quiet --include network --only raw --only exchange_bybit # one exchange mix test.json --quiet --include network --only raw --only private # +creds private mix test.json --quiet --include network --include dangerous --only raw # +write paths ``` **Files changed:** - `test/support/test_generator/raw_endpoint_probe.ex` β€” new. - `test/support/test_generator/raw_endpoint_probe_config.ex` β€” new. - `test/ccxt/raw_endpoint_probe_test.exs` β€” new. - `test/support/integration_helper.ex` β€” `:allow_4xx` option threaded through `validate_body!/3`. - `CLAUDE.md`, `ROADMAP.md`, `CHANGELOG.md` β€” doc updates. **Discovered work to capture at execution close-out:** clusters of public-path flunks on the same exchange are T80-style URL/prefix bugs β€” file one follow-up task per cluster. WS integration probe across the reachable exchanges stays a separate task, waiting on T98–T101 fixes first. ### 2026-04-19: Post-T94 WebSocket smoke audit Ran a live tidewave smoke against every reachable exchange in `CCXT.WS.Config` immediately after T94 shipped β€” no mocks, production endpoints (plus deribit testnet). Goal was "does WS actually work end-to-end beyond the three canaries." Result: **6 of 13 reachable exchanges stream live data cleanly, 5 distinct defects filed as new tasks, 3 orphaned config entries surfaced, and the T94 "17 exchanges / 14 new" count was corrected to "16 configured / 13 reachable / 13 new."** **Verified end-to-end (live frames received):** | Exchange | Pattern | Signal | |---|---|---| | bybit | `op_subscribe` | `tickers.BTCUSDT` snapshot + delta | | deribit (testnet) | `jsonrpc_subscribe` | Correlated JSON-RPC subscribe + `ticker.BTC-PERPETUAL.100ms` ticks | | okx | `op_subscribe_objects` | Subscribe ack + BTC-USDT ticker ticks | | binance | `method_subscribe` | Live trade events (canonical `e`/`s`/`E` shape) | | bitmex | `op_subscribe` | Connects + subscribe `:ok` + info frame | | coinbaseexchange | `type_subscribe` | Subscribe reached server (rejected on bad product id β€” wiring correct) | **Defects filed as tasks (see ROADMAP.md Active Track):** - **T98 πŸ›** β€” `:event_subscribe` drops `"symbol"` on bitfinex (β†’ `code:10300 channel: unknown`) AND double-wraps caller-supplied envelopes on gate (β†’ `code:2 Unknown channel`). One pattern module, two shape defects. - **T99 πŸ›** β€” `:method_params_subscribe` emits a non-string `channel` field that kraken v2 rejects (`{"error":"Channel field must be a string","success":false}`). - **T100 πŸ›** β€” `:sub_subscribe` frame includes an `"id"` field; `ZenWebsocket.Client.send_message/2` blocks on JSON-RPC id correlation but HTX's response id shape doesn't match, so the call crashes after 5s `GenServer.call` timeout. Affects htx + huobi. - **T101 πŸ›** β€” aster connects to `wss://sstream.asterdex.com/stream` and `subscribe/3` returns `:ok`, but zero frames arrive in 12s. URL is binance-combined-stream style; pattern output doesn't match its subscribe contract. - **T102 πŸ›** β€” `CCXT.WS.Config` declares `bingx`, `bitget`, `mexc` but `CCXT.Registry` has no compiled module for any of them β†’ `Exchange.new!("bingx")` raises. Either add to Registry or remove from Config. Includes the "17 β†’ 16 / 14 β†’ 13" count correction. **Count correction (propagated):** `CCXT.WS.Config.supported_exchanges/0` returns **16** entries, not 17. T94 added **13** priority-tier entries on top of the 3 canaries, not 14. Of those 16, **13 are reachable today** (3 with no Registry module). Updated in CLAUDE.md lines 194 + 196 and the T94 CHANGELOG entry below. **Verification method:** tidewave `project_eval` drove `CCXT.WS.connect/3` + `subscribe/3` + 5–8 `receive` frames per exchange, then `close/1`. Each probe self-contained in one eval call so frames route to the same process that connected. Full transcripts in session log. **No production code changed in this audit** β€” this is doc work only. The fixes land as T98–T102. ### 2026-04-19: Task 94 β€” WS subscription patterns + per-exchange config Completes the WebSocket track begun in T92 (Layer 1+2 foundation) and continued in T93 (auth patterns). Replaces the three hand-rolled `CCXT.WS.Helpers.{bybit,deribit,okx}_subscribe` builders with a full subscription-pattern dispatcher, wires T93's auth modules into per-exchange config, and extends the config map from 3 canaries to **16 priority-tier entries** (original task description said 17; actual wired count is 16 = 3 canaries + 13 new β€” count corrected post-T94 smoke audit, see entry above). **Roadmap count correction (11 β†’ 14):** the ROADMAP entry and original bak count were both wrong. Bak ships 14 pattern modules β€” the prose missed `sub_based` (HTX/Huobi), `reqtype_sub` (BingX), and `type_subscribe` (KuCoin/Coinbase), all priority-tier-required. All 14 are ported as a unit even though 3 (`action_subscribe`, `method_as_topic`, `method_topics`) are unused by any priority-tier exchange β€” the modules are small and porting as a set lets `:custom` escape-hatch exchanges in future consumers use them without another port. **New modules:** - `CCXT.WS.Subscription` β€” function-head dispatcher parallel to `CCXT.WS.Auth`. `@patterns` lists the 14 atoms; `module_for_pattern/1` maps each to a pattern module; `build_subscribe/3` + `build_unsubscribe/3` dispatch with `{:error, {:unknown_pattern, atom}}` for unknown atoms. - `CCXT.WS.Subscription.Behaviour` β€” `@callback subscribe(channels, config) :: map() | [map()]` and optional `unsubscribe/2`. The `map() | [map()]` return shape is load-bearing: HTX/BingX (and Upbit's `array_format` custom) emit one frame per channel; Deribit/Bybit/OKX emit a single envelope listing all channels. - 14 pattern modules under `CCXT.WS.Subscription.*`: `OpSubscribe`, `OpSubscribeObjects`, `MethodSubscribe`, `MethodParams`, `MethodSubscription`, `MethodTopics`, `MethodAsTopic`, `Jsonrpc`, `EventSubscribe`, `TypeSubscribe`, `SubBased`, `ReqtypeSub`, `ActionSubscribe`, `Custom`. **Four of these have shipped frame-shape defects surfaced by the post-merge tidewave smoke** β€” see T98–T100 in the audit entry above for specifics and ROADMAP.md for fix ownership. **Dispatcher integration:** `CCXT.WS.subscribe/3` now calls `Subscription.build_subscribe(config.subscription_pattern, channels, merged_config)`. A new private `send_payload/2` handles the list-return case via `Enum.reduce_while/3`, halting on the first `{:error, _}`. Runtime opts merge into the exchange's static `subscription_config` before pattern dispatch β€” useful for injecting a fresh JSON-RPC id or an inline auth token per call. **Config shape change:** `CCXT.WS.Config` entries now carry `subscription_pattern` + `subscription_config` + `auth_pattern` + `auth_config` in place of the old `subscribe_builder: {mod, fun}` tuple. Wired **13 new exchanges** (binance, kraken, kucoin, gate, htx, huobi, bitmex, bitfinex, coinbaseexchange, bitget, mexc, bingx, aster β€” plus refreshed shape for the 3 canaries bybit/deribit/okx). **Three of the 13 (bingx, bitget, mexc) have no compiled `CCXT.Registry` entry** so `Exchange.new!` does not resolve them today β€” tracked as T102. Net reachable WS exchanges: **13** = 3 canaries + 10 reachable new entries. URLs, heartbeat intervals, and auth pattern atoms sourced from `ccxt_client_bak/priv/specs/extracted/.exs` `ws:` sections. Three T93 auth-pattern mapping corrections baked in: htx/huobi/bitmex all use `:direct_hmac_expiry` (not bak's `:htx_variant`); bitget uses `:iso_passphrase` (not bak's `:generic_hmac`); kraken uses `:rest_token`, gate uses `:sha512_newline` (T93 CHANGELOG ground truth). **Known MVP URL limitations** (documented in `CCXT.WS.Config` moduledoc): single URL per `{section, sandbox}` slot. bybit is linear-only (spot/inverse/option need product routing); binance is spot-only (USDβ“ˆ-M/COIN-M/portfolio margin need routing); htx/huobi spot-only (contract endpoints live on `api.hbdm.vn`); mexc/bingx/aster spot/linear only. kucoin URLs left `nil` β€” they're fully dynamic via REST `POST /api/v1/bullet-{public,private}` pre-call, deferred to the adapter layer. **Files changed:** - `lib/ccxt/ws/subscription.ex` β€” new dispatcher. - `lib/ccxt/ws/subscription/behaviour.ex` β€” new behaviour. - `lib/ccxt/ws/subscription/*.ex` (14 files) β€” new pattern modules. - `lib/ccxt/ws/config.ex` β€” shape change + 14 new entries + updated moduledoc. - `lib/ccxt/ws.ex` β€” `subscribe/3` rewritten to use dispatcher; new `merge_subscription_config/2` + `send_payload/2` private helpers; `Subscription` alias added. - `lib/ccxt/ws/helpers.ex` β€” 3 subscribe builders deleted; `interpolate_hostname/2` retained for URL routing. - `test/ccxt/ws/subscription_test.exs` β€” new dispatcher tests (patterns/0, module_for_pattern/1, build_subscribe/build_unsubscribe happy + unknown paths, "every pattern module builds without raising" sweep). - `test/ccxt/ws/subscription/*_test.exs` (14 files) β€” frame-shape tests per pattern module covering subscribe + unsubscribe + config overrides + empty-channel edge cases. - `test/ccxt/ws/url_routing_test.exs` β€” swapped stale `"kraken"` sentinel for `"hyperliquid"` in the "unsupported exchange" tests (kraken is now wired). **Verification:** - Full suite: 1275/1275 pass, 258 excluded (network/dangerous/fixture_replay tags). - WS-only: 148/148 pass. - Canary live smoke (`--include network test/ccxt/ws/canary_test.exs`): 3/3 pass against real bybit/deribit/okx production endpoints. - `mix format --check-formatted` clean. `mix dialyzer` reports 3 pre-existing `invalid_contract` warnings on `CCXT.WS.{send_message, close, get_state}` specs β€” predate T94 (verified via `git stash` diff), not introduced by this task. **Discovered work:** Task 97 (per-exchange channel-name formatters) filed in ROADMAP.md Β§ Active Track. Today callers must pass pre-formatted channel strings; T97 adds `format_channel/3` so callers can pass `(:watch_ticker, "BTC/USDT")`. Gated on upstream ccxt_extract emitting `structure.ws_methods.channel_templates` β€” bak's implementation derives templates from JS AST which duplicates ccxt_extract's role. ### 2026-04-19: Task 96 β€” Register aster testnet creds Closes out the third gap from the 2026-04-18 audit. `{:aster, testnet: true}` appended to `CCXT.Testnet.register_all_from_env/1` in `test/test_helper.exs`. Verified via tidewave MCP before registration: signed `GET /fapi/v2/balance` against `https://fapi.asterdex.com` returned HTTP 200 with `accountAlias: "mYuXSgTiAuXquX"` and an empty asset list. **Caveat worth remembering (documented in the test_helper comment + CLAUDE.md):** the aster spec has no `urls.test` entry. The "testnet" env vars are production API keys against a zero-balance account. `CCXT.Credentials.sandbox: true` + `Exchange.sandbox: true` still propagate, but base URLs fall through to production. Any future `:dangerous` write-path probe must be opt-in per exchange β€” aster's write path would hit the real exchange. T39/T67 private probes for aster now register the creds at test boot instead of flunking with setup instructions. ### 2026-04-19: Task 95 β€” Reconcile testnet env var naming Closes out the first two gaps from the 2026-04-18 audit. Chose the consumer-side alias path over operator-side env var renames so the fix lands entirely in the repo β€” no shell edits required, no silent skips masking partial provisioning. **Change:** `CCXT.Testnet.read_env_credentials/3` now probes `_TESTNET_` first, then `_TEST_` as a silent fallback when `:testnet` is true. New `resolve_env/4` helper walks the candidate list and returns `{canonical_var_name, value_or_nil}`; the canonical name is what `build_env_credentials/2` surfaces in error/:skipped contexts, so operator-facing messages still point at the preferred convention. **Covered:** `BINANCE_FUTURES_TEST_API_KEY/SECRET` on the dev host now registers successfully under `{:binance, :futures, testnet: true}` without renaming shell state. **Not covered β€” tracked but operator-blocked:** Kraken's secret env var is both misnamed (`KRAKEN_TESTNET_SECRET_KEY` vs the expected `KRAKEN_TESTNET_API_SECRET`) AND empty on the dev host. A naming fallback alone can't unblock it; the secret needs to be populated first. Deliberately NOT adding `{:kraken, testnet: true}` to `register_all_from_env/1` β€” it would just return `:skipped` silently and hide the real blocker behind a green test boot. Documented in CLAUDE.md testnet-creds section so a future session re-runs that exchange only after populating the secret + renaming to the canonical form. **Files changed:** - `lib/ccxt/testnet.ex` β€” new `resolve_env/4` + `fetch_env/1` helpers; `read_env_credentials/3` now iterates over `["_TESTNET", "_TEST"]` candidates when `testnet: true`. Module docstring updated to document the alias. - `test/ccxt/testnet_test.exs` β€” two new tests: (1) `_TEST_` fallback picked up when `_TESTNET_` unset, (2) `_TESTNET_` wins when both are set (canonical preference). ### 2026-04-18: Testnet credential wiring audit β€” Tasks 95 and 96 added Audit of the shell env vs `test/test_helper.exs:13-18` surfaced three gaps: - **Binance USDβ“ˆ-M futures broken** β€” env var named `BINANCE_FUTURES_TEST_API_KEY/SECRET` (`_TEST_`), but `CCXT.Testnet.env_var_prefix/2` emits `BINANCE_FUTURES_TESTNET_*`. Registration silently skips and every futures private probe flunks with "missing creds" despite creds being present on the host. Added as **Task 95** β€” fix by renaming env vars OR teaching `build_env_credentials/2` to accept a `_TEST_` alias. - **Kraken partially provisioned** β€” key under `KRAKEN_TESTNET_API_KEY` but secret under non-standard `KRAKEN_TESTNET_SECRET_KEY` AND empty. Folded into Task 95 (same resolution path, different secret availability). - **Aster unregistered** β€” `ASTER_TESTNET_API_KEY/SECRET` set in env but aster not in the `register_all_from_env/1` list. Added as **Task 96** β€” one-line addition to the list. Confirmed-working testnet creds today: bybit, deribit, binance spot. All other priority-tier exchanges flunk private probes via `IntegrationHelper.require_credentials!/2` with setup instructions (by design). CLAUDE.md's new "Testnet Credentials Available" subsection now documents the actual state (working / broken / not provisioned) so fresh instances don't assume futures works when it doesn't. ### 2026-04-18: Task 93 β€” WebSocket auth pattern modules Ports the 9 auth pattern modules from `ccxt_client_bak/lib/ccxt/ws/auth/` to the new `%CCXT.Exchange{}` / `%CCXT.Credentials{}` shape and adds a parallel `CCXT.WS.Auth` behaviour + dispatcher mirroring `CCXT.Signing`. No integration into `CCXT.WS.connect/3`, no per-exchange wiring β€” T94 composes these modules into exchange configs. **Files added:** - `lib/ccxt/ws/auth.ex` β€” behaviour dispatcher. Function-head routing on a `pattern()` atom (parallel to `CCXT.Signing.sign/4`). Public API: `pre_auth/4`, `build_auth_message/4`, `build_subscribe_auth/5`, `handle_auth_response/3`, plus introspection helpers `patterns/0`, `module_for_pattern/1`, `requires_pre_auth?/1`, `inline_auth?/1`. Unknown patterns surface as `{:error, {:unknown_pattern, atom}}`. - `lib/ccxt/ws/auth/behaviour.ex` β€” four callbacks: `pre_auth/3`, `build_auth_message/3`, `handle_auth_response/2`, optional `build_subscribe_auth/4`. - `lib/ccxt/ws/auth/direct_hmac_expiry.ex` β€” bybit, bitmex, htx family. `"GET/realtime{expires}"` HMAC-SHA256 (hex default, base64 via `config[:encoding]`). Frame: `%{op, args: [api_key, expires, sig]}`. - `lib/ccxt/ws/auth/iso_passphrase.ex` β€” okx, kucoin, bitget. Requires `credentials.password`. Payload `timestamp+"GET"+"/users/self/verify"`, HMAC-SHA256 base64. - `lib/ccxt/ws/auth/jsonrpc_linebreak.ex` β€” deribit. JSON-RPC 2.0 `public/auth` with payload `"{ts_ms}\n{nonce}\n"`. `handle_auth_response/2` extracts `result.expires_in` β†’ `{:ok, %{ttl_ms: N}}` for re-auth scheduling. - `lib/ccxt/ws/auth/sha384_nonce.ex` β€” bitfinex. `AUTH{nonce}` HMAC-SHA384 hex; flat frame (no `args` nesting). - `lib/ccxt/ws/auth/sha512_newline.ex` β€” gate, gateio. Newline-separated payload `"{event}\n{channel}\n{req_params_json}\n{time_s}"`, HMAC-SHA512 hex, two-level (outer + inner `payload`) frame. - `lib/ccxt/ws/auth/listen_key.ex` β€” binance family, aster. `pre_auth/3` resolves REST endpoint by market type (`:spot`/`:linear`/`:inverse`/`:margin`/`:isolated_margin`/`:portfolio`); normalizes `:futureβ†’:linear`, `:deliveryβ†’:inverse`. `build_auth_message/3` β†’ `:no_message` β€” listen key goes in the WS URL. - `lib/ccxt/ws/auth/rest_token.ex` β€” kraken. `pre_auth/3` returns `{:ok, %{endpoint:, credentials:}}` or `{:error, :no_token_endpoint}`. **Stateful-config adaptation:** bak mutated `config[:token]` after REST round-trip; consumer-side port threads the token through `config[:token]` explicitly (caller responsibility) rather than letting pattern modules mutate shared state. - `lib/ccxt/ws/auth/inline_subscribe.ex` β€” coinbaseexchange. `build_auth_message/3` β†’ `:no_message`; auth attaches per-subscribe via `build_subscribe_auth/4`. Secret base64-decoded before HMAC-SHA256. - `lib/ccxt/ws/auth/expiry.ex` β€” pure TTL utility (not a behaviour implementer). `compute_ttl_ms/2` prefers response-level `auth_meta.ttl_ms` over config `auth_ttl_ms`; `schedule_delay_ms/1` applies the 80% safety margin and 24h cap. Consumed by the `jsonrpc_linebreak` handler's output. - Per-pattern unit test files + a dispatcher suite + an `Expiry` utility suite. All offline, byte-equal assertions with frozen clocks via the `timestamp_ms_override` / `nonce_override` pattern already wired into `CCXT.Signing.*_from_config/1`. - `test/ccxt/ws/auth_live_smoke_test.exs` β€” opt-in live smoke against real sandboxes (`@moduletag :network + :ws_auth_smoke`, excluded by default). Covers bybit `:direct_hmac_expiry` and deribit `:jsonrpc_linebreak`. Run with `mix test.json --include network --include ws_auth_smoke --only ws_auth_smoke`. Both pass end-to-end against `stream-testnet.bybit.com/v5/private` and `test.deribit.com/ws/api/v2`. Binance `:listen_key` is covered by its unit test (`pre_auth/3` is pure endpoint resolution, no WS frame) and will be exercised end-to-end in T94 when the adapter makes the REST call. **Reused existing utilities (no new helpers added):** - `CCXT.Signing.hmac_sha256/2`, `hmac_sha384/2`, `hmac_sha512/2` - `CCXT.Signing.encode_hex/1`, `encode_base64/1`, `decode_base64/1` - `CCXT.Signing.timestamp_ms_from_config/1`, `timestamp_seconds_from_config/1`, `nonce_from_config/2` β€” the deterministic-override helpers originally added for the T65 fixture-replay harness; pattern modules consult them so unit tests freeze timestamps/nonces without monkey-patching. - `CCXT.Credentials` (`.api_key`, `.secret`, `.password`, `.uid`). **Mapping corrections relative to the T93 ROADMAP entry:** The entry's prose mapping swapped kraken and gate. Authoritative bak module assignments (reflected in this implementation's module moduledocs): - `kraken` β†’ `:rest_token` (bak `rest_token.ex` is explicitly "Kraken style") - `gate` / `gateio` β†’ `:sha512_newline` (bak `sha512_newline.ex` is explicitly "Gate style") T94 picks up the corrected mapping when wiring per-exchange `CCXT.WS.Config` entries. Also: **deribit gets a module** (`:jsonrpc_linebreak`) β€” the ROADMAP said "no module needed β€” handled in T92" but T92 only wired public subscribes; private deribit streams need the `public/auth` JSON-RPC roundtrip before any private subscribe, and the pattern module emits the frame. **Scope decisions vs. bak:** - Ported 8 behaviour-implementing modules + 1 utility (`Expiry`). Skipped bak's `:htx_variant` stub (`{:error, {:not_implemented, :htx_variant}}` placeholder) and `:generic_hmac` alias (bak fell back to DirectHmacExpiry). When htx WS auth lands in T94, it either reuses `:direct_hmac_expiry` directly or gets its own pattern module with a descriptive atom β€” not an unhelpful variant stub. - `rest_token.ex` stateful-config mutation β†’ explicit `config[:token]` parameter (see above). - `inline_auth?/1` returns `true` for both `:inline_subscribe` and `:rest_token` (bak had `:inline_subscribe` only) β€” kraken's per-subscribe token injection is semantically inline auth even though the pattern is named for its REST pre-auth step. **Quality:** - Full suite passes; `mix format --check-formatted` clean; `mix credo --strict` reports 0 new issues; `mix dialyzer.json` reports 0 new warnings (pre-existing `invalid_contract` entries in `lib/ccxt/ws.ex` from T92 unchanged); `mix sobelow` clean. **Deviations from the plan:** none. ### 2026-04-18: Task 92 β€” WebSocket Layer 1+2 foundation Opens the WebSocket Track. Ports the pure URL-resolution + connection-lifecycle layers from `ccxt_client_bak/lib/ccxt/ws/`, adapted to the new `%CCXT.Exchange{}` struct and narrowed to three canary exchanges (bybit, deribit, okx). No auth, no subscription-pattern dispatch, no message normalization β€” those land in T93 / T94 / (deferred). **Files added:** - `lib/ccxt/ws.ex` β€” public entry; `connect/3`, `subscribe/3`, `send_message/2`, `close/1`, `get_state/1`, `get_url/1`. Thin wrapper around `ZenWebsocket.Client` carrying a `%CCXT.WS{exchange, zen_client, url, section}` struct so `subscribe/3` can look up the exchange-native frame builder. Collapsed the planned `CCXT.WS.Client` module into this one module per the zen-websocket "don't create wrapper modules" guidance β€” one thin wrapper, not two. - `lib/ccxt/ws/url_routing.ex` β€” `public_url/1`, `private_url/1` from `%CCXT.Exchange{}`. Sandbox-aware (`exchange.sandbox`), reuses bybit's `{hostname}` interpolation pattern from REST. - `lib/ccxt/ws/config.ex` β€” per-exchange WS config map. Three canary entries (bybit/deribit/okx) carrying production + sandbox URLs, heartbeat config, and `{mod, fun}` pointer to the subscribe-frame builder. Centralized (single module attribute) rather than one-file-per-exchange β€” simpler review surface at ~17-entry maximum and easier to replace with spec-driven data when upstream extraction lands. - `lib/ccxt/ws/helpers.ex` β€” three pure subscribe-frame builders (`bybit_subscribe/2`, `deribit_subscribe/2`, `okx_subscribe/2`) + `interpolate_hostname/2`. Minimum needed to validate the architecture; T94 replaces these with the 11 pattern modules ported from bak. - `test/ccxt/ws/url_routing_test.exs` β€” pure unit tests covering production/sandbox URLs + `{hostname}` interpolation + unknown-exchange (returns `nil`). Async, no network. - `test/ccxt/ws/canary_test.exs` β€” real-endpoint tests (`@moduletag :network + :ws_canary`). Each canary: connect β†’ subscribe β†’ receive β‰₯1 decoded-map frame within 10s β†’ close cleanly. **Files modified:** - `mix.exs` β€” added `{:zen_websocket, "~> 0.4"}` to runtime deps. `mix deps.get` pulled `certifi`, `gun`, `cowlib` transitively. - `CLAUDE.md` β€” Key Modules table extended with 4 new rows. - `ROADMAP.md` β€” T92 marked βœ…, Suggested Execution Order callout advanced to T93/T94. **Canary channels used:** bybit `tickers.BTCUSDT` (linear public), deribit `ticker.BTC-PERPETUAL.100ms` (the plan's `book.BTC-PERPETUAL.raw` was rejected upstream with `raw_subscriptions_not_available_for_unauthorized` β€” raw book variants require auth even on testnet; `ticker.*.100ms` is definitively public), okx `%{channel: "tickers", instId: "BTC-USDT"}`. **Deviations from the plan:** 1. Collapsed `CCXT.WS.Client` into `CCXT.WS` (one wrapper, not two) per zen-websocket include's explicit "Don't create wrapper modules" rule. The struct with connection context lives on `CCXT.WS` itself. 2. Deribit canary channel changed mid-flight (see above). Plan had `book.BTC-PERPETUAL.raw`; runtime probe proved this requires auth; swapped to public ticker channel. **Pre-finding that shaped the design:** JSON specs do not expose WS URLs cleanly today β€” they live as raw JS AST literals inside `structure.ws_methods`. Filed as follow-up: push to ccxt_extract to surface `urls.api.ws.{public,private,sandbox_public,sandbox_private}` in a future schema bump. When that lands, `CCXT.WS.Config` becomes a one-line read from `exchange.spec["urls"]["api"]["ws"]` and this file is deletable. **Quality:** `mix compile --warnings-as-errors` clean. Full suite green (network-gated tests excluded by default). Canary tests pass against real production endpoints. **Files touched:** `mix.exs`, `mix.lock`, `lib/ccxt/ws.ex`, `lib/ccxt/ws/config.ex`, `lib/ccxt/ws/helpers.ex`, `lib/ccxt/ws/url_routing.ex`, `test/ccxt/ws/url_routing_test.exs`, `test/ccxt/ws/canary_test.exs`, `CHANGELOG.md`, `ROADMAP.md`, `CLAUDE.md`. ### 2026-04-18: Task 91 β€” Scope outbound currency-alias application (πŸ› fix) **Bug:** `CCXT.Symbol.to_exchange_id/{1,2}` reversed `exchange.common_currencies` to build a forward alias map and applied it to every outbound symbol. `XBT β†’ BTC` is a defensive base default carried by **every** priority-tier exchange's spec β€” so reversal produced `BTC β†’ XBT` everywhere, silently rewriting outbound symbols on 22 of 23 priority-tier exchanges (binance/bybit/okx/kucoin/htx/deribit/hyperliquid all returned `XBT*` shapes the exchanges don't accept). Only Kraken was correct, because Kraken actually accepts `XBT` on input. Tidewave probe surfaced 2026-04-18. **Fix (Option 2 from the roadmap entry β€” per-exchange opt-in):** - `%CCXT.Exchange{}` gains `outbound_aliases: %{}` field (`lib/ccxt/exchange.ex`). - New `@outbound_aliases %{"kraken" => %{"BTC" => "XBT"}}` module attribute is the single source of truth; `Exchange.new/2` populates the field via `Map.get(@outbound_aliases, exchange_id, %{})`. - `lib/ccxt/symbol.ex:677` and `:701` swapped from `forward = reverse_aliases(exchange.common_currencies)` to `apply_pattern(parsed, config, exchange.outbound_aliases)`. - `from_exchange_id/3` direction is **unchanged** β€” still reads `exchange.common_currencies` directly, preserving the universal inbound `XBT β†’ BTC` round-trip. **Why this shape over alternatives:** spec-driven (Option 1) waits on upstream Phase 16 Task 97 / our T60 β€” too slow for a bug breaking 22 exchanges today. Heuristic (Option 3, "scan markets to decide") was rejected as Three-Strikes risk in the original roadmap entry. Option 2 is the recommended bridge: when T60 lands, only the population path migrates from the hardcoded clause to a spec lookup; the struct field shape stays. No call-site churn at migration. **Reused, not deleted:** `apply_forward_alias/2` (lib/ccxt/symbol.ex:1238) already short-circuits on empty maps β€” the `%{}` default for 22 of 23 exchanges is a true no-op. `reverse_aliases/1` is kept as a public utility (no internal callers, but it's documented public API and removal is a separate decision). **Tests:** - `make_exchange/3` helper in `test/ccxt/symbol_test.exs` extended to `make_exchange/4` (new optional `outbound_aliases` parameter, defaults `%{}`). All existing call sites continue working via default. - Existing Kraken forward-direction edge case ("currency aliases applied in forward direction") renamed to "outbound aliases applied (Kraken BTC β†’ XBT)" and updated to pass `outbound_aliases` explicitly. - New `to_exchange_id/2 + from_exchange_id/3 with real specs (Task 91 regression)` describe block exercises real `CCXT.Exchange.new!/1` exchanges: - binance / bybit / okx outbound `BTC/USDT β†’ BTC*USDT*` (no XBT leak) - Kraken outbound `BTC/USDT β†’ XBT*` shape preserved (asserts presence of `XBT` and absence of bare `BTCUSDT`, decoupled from Kraken's market-list `X*` prefix rules) - Kraken inbound `XBTUSDT β†’ BTC/*` round-trip preserved - Plus a new edge-case test "common_currencies alone does NOT trigger outbound rewrite" β€” direct regression for the original bug shape. **Quality:** Full unit suite green (1125 passed, +7 net from prior 1118). Credo strict 0 new issues. Sobelow skip file regenerated via `--mark-skip-all` (line numbers shifted from the new module attribute). Network probes (T79 symbol-public, T39 unified integration) deferred to on-demand re-run β€” likely the dominant root cause behind T79's 32/41 failures and a major contributor to T39 unified integration breakage. **Files touched:** `lib/ccxt/exchange.ex`, `lib/ccxt/symbol.ex`, `test/ccxt/symbol_test.exs`, `.sobelow-skips`, `ROADMAP.md`, `CHANGELOG.md`, `CLAUDE.md`. ### 2026-04-18: WebSocket Track opened β€” T92 / T93 / T94 added to ROADMAP **Planning entry, no code yet.** Three sequential tasks added to ROADMAP.md (Active Track table rows 15–17 + new "WebSocket Track (Unnormalized)" phase detail section): - **Task 92 (D:4/B:9/U:9 β†’ 2.25 🎯)** β€” WS Layer 1+2 foundation: URL routing + connection lifecycle ported from `ccxt_client_bak/lib/ccxt/ws/{url_routing,client,helpers}.ex` adapted to the new `%CCXT.Exchange{}` struct, layered on `zen_websocket` (5-function client). Three canary exchanges (bybit / deribit / okx) validate the architecture before T93/T94 land. - **Task 93 (D:5/B:8/U:8 β†’ 1.6 πŸš€)** β€” port the 9 auth pattern modules from bak (`direct_hmac_expiry`, `expiry`, `inline_subscribe`, `iso_passphrase`, `jsonrpc_linebreak`, `listen_key`, `rest_token`, `sha384_nonce`, `sha512_newline`). Shared behaviour interface mirrors REST `CCXT.Signing` shape. - **Task 94 (D:6/B:8/U:7 β†’ 1.25 πŸ“‹)** β€” port the 11 subscription pattern modules + per-exchange config for the 17 non-`:custom` priority-tier exchanges. Inbound frame routing via channel-keyed registry. Same incremental ship cadence: 3 canaries first, one exchange per commit thereafter. **Why this scope, why now:** - `structure.ws_methods` exists in spec (33 entries on bybit) but is raw JS AST β€” not yet derived into the equivalent of REST's `sign_method` / `request_defaults` / `unified_endpoints`. So a fully spec-driven WS dispatcher is upstream-blocked, but format-independent porting from bak is unblocked. - `zen_websocket` already provides Layer 3 (~80% of stateful adapter behavior β€” reconnection, subscription restoration, heartbeat health). A thin `CCXT.WS.Adapter` GenServer is deliberately deferred until consumer pressure surfaces a gap. - Same philosophy as the raw REST surface: pass exchange-native frames through, no normalization. T44 (response shape transformers) handles normalization for both REST and WS once upstream Phase 12 lands. **Out of scope for the track as filed:** - Message normalization (paired with T44, gated on T39 evidence + upstream Phase 12) - The 3 `:custom` DEX exchanges (hyperliquid / derive / lighter) β€” defer alongside REST custom signing - Tier3 exchanges β€” on-demand only - Layer 3 stateful adapter beyond what zen_websocket provides β€” add only on demonstrated need ### 2026-04-18: Task 91 surfaced β€” reverse currency-alias leak (bug, not fixed) **Discovery during a tidewave probe of unified dispatch.** Filed as Task 91 in ROADMAP.md (πŸ›, D:3/B:9/U:9 β†’ Eff:3.0 🎯, ⬜ pending). Not fixed in this entry β€” recorded so the next session can pick it up with full context. **Symptom:** `CCXT.Symbol.to_exchange_id("BTC/USDT", exchange)` produces `XBTUSDT` (binance, bybit, hyperliquid), `XBT-USDT` (okx, kucoin), `xbtusdt` (htx), `XBT_USDT` (deribit) β€” none of which those exchanges accept. Only Kraken (`XXBTUSDT`) is correct. **Root cause:** `to_exchange_id/2` (lib/ccxt/symbol.ex:677, mirrored at :701) reverses `exchange.common_currencies` (`XBT β†’ BTC`) into a forward alias map (`BTC β†’ XBT`) and applies it on every outbound currency. `XBT β†’ BTC` lives in `common_currencies` on every exchange because the base CCXT JS `Exchange` class declares it as a defensive inbound default β€” but reversing it overscopes the alias to outbound on exchanges that don't accept XBT. **Impact:** silently breaks every symbol-required unified call (`fetch_ticker`, `fetch_ohlcv`, `fetch_order_book`, `fetch_trades`, `fetch_funding_rate`, private methods with `symbol`) on all 22 non-Kraken priority-tier exchanges. Likely the dominant root cause behind T79's 32/41 symbol-public-probe failures and a major contributor to T39 unified integration breakage. T86 (symbol denormalization wiring, 2026-04-17) ships the call site; T91 is the correctness fix. **Recommended approach:** consumer-side `outbound_aliases` field on `%CCXT.Exchange{}`, populated only for exchanges known to accept the alias on input (Kraken initially); spec-driven inbound/bidirectional marking arrives via T60 (Phase 16 Task 97) β€” bridge to that. Alternative options + tradeoffs in the ROADMAP entry. ### 2026-04-18: Task 63 β€” Override contribution workflow **What was done:** - New `CONTRIBUTING.md` at repo root. Includes: "Where Does My Fix Belong?" routing table, an "Exchange Overrides" section that walks the full upstream workflow (reproduce β†’ RFC 6901 JSON Pointer β†’ `priv/overrides/.json` β†’ regenerate specs β†’ verify β†’ paired PRs), explicit anti-pattern list (no consumer-side override maps, no third heuristic patch, no "fix upstream later" workarounds), and a code-contribution PR checklist. - `README.md` gains a new "Fixing Exchange Bugs Upstream" section pointing at `CONTRIBUTING.md` Β§ Exchange Overrides and `ccxt_extract/SCHEMA.md` Β§ Override Contract. Documentation list now links `CONTRIBUTING.md`. **Why:** - All three upstream prerequisites shipped earlier in April: Task 60 (override file format), Task 61b (generic RFC 6901 merge via `OverrideRegistry.apply_all/2`, 2026-04-16), and Task 61c (Schema 2.0.0 promoting `_provenance` to required, 2026-04-17 β€” adopted on our side by Task 53 the same day). The consumer-side docs have been a missing piece since then β€” contributors had no single place to learn the override workflow, so fixes risked landing as `%CCXT.Exchange{}` workarounds instead of upstream corrections. Task 64 explicitly rejected a consumer-side override registry; this doc formalizes that direction. **Key decisions:** - **CONTRIBUTING.md is new, not a CLAUDE.md extension.** CLAUDE.md is AI-instance-facing (architectural decisions, internal conventions); CONTRIBUTING.md is contributor-facing (workflow, checklist, commit style). They cross-reference where topics overlap (Three-Strikes Rule, Config Over Inference) but each stays in its lane. - **Override workflow is prescriptive, not a tutorial.** The section lists six numbered steps with concrete tool names (`mix ccxt_extract.update --exchange `, `OverrideRegistry.apply_all/2`) and defers format details to `ccxt_extract/SCHEMA.md` rather than duplicating them. Duplication would drift. - **Anti-patterns are named explicitly.** Listing "consumer-side override maps" / "third heuristic patch" / "fix upstream later workarounds" as forbidden (with the Task 64 rejection as the worked example) gives future contributors a clear "don't" rather than an implicit hope they'll read `CLAUDE.md`. - **Scope: docs only.** No runtime changes. No test suite impact. No task is unblocked by this landing β€” it's pure workflow hygiene closing the loop on the upstream override infrastructure. **Verification:** - `CONTRIBUTING.md` created at repo root; `README.md` diff adds the upstream section + `CONTRIBUTING.md` link. - Cross-references resolve: `ccxt_extract/SCHEMA.md` Β§ Override Contract exists (per upstream Task 60 / 61b shipping notes in ROADMAP.md lines 44 + 142); `CLAUDE.md` Β§ Config Over Inference and Β§ Three-Strikes Heuristic Rule both present. - No code changes β†’ no test runs needed. **Out of scope:** - Override authoring tutorial with an end-to-end example. Deferred until a real override lands on `ccxt_client`'s side of the split so the example can be authentic rather than hypothetical. - Automated override-application verification in `ccxt_client`'s CI. Would duplicate `ccxt_extract`'s test suite; not useful here. --- ### 2026-04-18: Task 34 β€” Log warning for missing path params **What was done:** - `CCXT.Dispatch.do_interpolate/4` (the recursive helper behind `CCXT.Dispatch.interpolate_path/2`) now emits `Logger.warning` when a `{param}` placeholder in a path template has no matching key in the params map. The warning names the missing param, the current path template, and the preserved placeholder, so operators see the silent skip in logs and can diagnose without having to run the exchange round-trip. - `require Logger` added to `lib/ccxt/dispatch.ex` (directly below the `@moduledoc`, above the existing `alias` block). - Removed the stale `TODO(Task 34): Consider logging a warning when path params are missing.` comment at the function head. - New unit test `test "logs warning when path param is missing"` in `test/ccxt/dispatch_test.exs` under the `describe "interpolate_path/2"` block. Uses `ExUnit.CaptureLog.capture_log/1` to capture the log output and asserts both the param name and the path template appear in the warning. **Why:** - Missing path params are currently preserved as-is (placeholder left in the path) so the exchange returns a descriptive error. That's the right runtime behavior, but it's invisible to operators β€” the path goes out with `{id}` embedded in it and only the exchange's own error response surfaces the mistake. The existing TODO marker at `lib/ccxt/dispatch.ex:148` had been sitting for multiple sessions as a known quality-of-life gap; Task 34 closes it with minimum mechanism. **Key decisions:** - **`Logger.warning`, not `Logger.error`.** The silent preservation is intentional by design (exchanges return descriptive errors); the log line is an operator hint, not a failure signal. `warning` matches the semantic β€” "something looks wrong, probably worth checking" β€” without promoting it to an error the supervisor might react to. - **Warning on *every* occurrence, not once-per-call.** The existing `skipped` accumulator already guarantees we only hit the `:not_found` branch once per unique param name within a single `do_interpolate` traversal (repeat placeholders of the same missing name get routed to the `is_map_key(skipped, ...)` guard clause). So the warning naturally fires once per missing param per call, not once per placeholder occurrence. - **No `Logger.metadata` / structured fields.** Stayed with a plain message string. The call shape is simple (param name + path + placeholder) and adding metadata would force a heavier logger configuration on consumers without clear payoff. Can be revisited if telemetry consumers need structured access. - **Kept the existing `ArgumentError` raise for explicit `nil`.** Only the `:not_found` branch logs; the `nil` branch still raises because an explicit `nil` is a caller bug, not a missing key. That separation is documented in the function comment and now reflected in the warning's conspicuous absence from the nil path. - **Test uses `capture_log/1`, module already has `@moduletag capture_log: true`.** The moduletag suppresses log output during normal test runs; `ExUnit.CaptureLog.capture_log/1` is the right API to capture it regardless of that tag. **Verification:** - `mix test.json --quiet test/ccxt/dispatch_test.exs` β†’ green (new case added alongside existing ones). - Static: `require Logger` imported in `lib/ccxt/dispatch.ex`; unused-require warning from the initial edit cleared once the `Logger.warning` call was added. - The `ArgumentError` raise path is unaffected β€” existing nil-value tests pass unchanged. **Out of scope:** - Telemetry event emission. Task 33 covers telemetry infrastructure and intentionally doesn't overlap. - Rate-limiting the warning to avoid log floods on high-volume loops. Each warning corresponds to a real caller-side bug; if someone triggers thousands of them they want every one in the log. --- ### 2026-04-18: Task 35 β€” "options" section ambiguity (retroactive closeout) **What was done:** - Marked Task 35 βœ… in ROADMAP.md. No code or test changes in this session. - The underlying fix shipped earlier, in-line with Task 15 (commit `b1457a5`, 2026-04-08): `@http_methods` in `lib/ccxt/exchange.ex` was scoped to `~w(get post put delete patch head)` with an explanatory comment naming Gate/GateIO as the reason OPTIONS is excluded from the HTTP-method set. Because `"options"` is no longer matched by the HTTP-method guard in `traverse_api_tree/4`, the recursion falls through to the section branch and `options` is correctly treated as a section name. - Regression test already present: `test/ccxt/exchange_generator_test.exs:273` ("traverses 'options' as section name, not HTTP verb (Gate pattern)") asserts `build_endpoint_configs/3` produces `:public_options_get_contracts`, `:private_options_post_orders`, etc., and that `sections == ["public", "options"]`. - Runtime verification against the live Gate spec: `CCXT.Gate.__endpoints__() |> Enum.filter(&("options" in &1.sections))` returns the full `public.options.*` and `private.options.*` endpoint set as sections (not HTTP verbs). `gateio.json` has the same structure and is covered by the same code path. **Why:** - Task 35 remained marked `⬜ Pending` in ROADMAP despite the fix shipping under Task 15's commit. The task-driver scan flagged it as a quick-win correctness task; on inspection the fix was already in place and guarded by a regression test. A retroactive closeout keeps the roadmap honest and prevents re-selection in future planning passes. **Key decisions:** - **The simpler fix stayed.** Original Task 35 description proposed "leaf-detection logic: if the value contains HTTP method sub-keys, it's a tree, not endpoints." That heuristic isn't needed β€” no CCXT exchange actually uses HTTP OPTIONS as a request method, so dropping `"options"` from `@http_methods` is both sufficient and less machinery. The `@http_methods` comment spells out the reasoning so future readers don't reintroduce OPTIONS. - **No new tests added.** Coverage at `test/ccxt/exchange_generator_test.exs:273` already pins down the contract (section-traversal names, sections list includes `"options"` as a section). Adding a second test would be redundant. **Verification:** - Static: `@http_methods ~w(get post put delete patch head)` in `lib/ccxt/exchange.ex` β€” no `options`. - Runtime: `CCXT.Gate.__endpoints__() |> Enum.filter(&("options" in &1.sections))` yields the full options endpoint set as sections, not skipped as a verb. - Existing regression test passes under the main suite (last full green: Task 64 run). **Out of scope:** - Any behavioral change. This entry is hygiene β€” roadmap status catches up to code reality. --- ### 2026-04-18: Task 64 β€” Sentinel-only role-gating fix in `CCXT.HTTP` **What was done:** - Added one new function head to `CCXT.HTTP.classify_by_eq_success/1`: when an `error_body_checks` entry is sentinel-only (`has_code_role: false`) and has `===` sentinel values, a value match classifies as `{:error, value}` directly. Code-bearing entries (`has_code_role: true`) keep their existing inverted semantics β€” there a `===` match is the *success* code that overrides the default `error_code?` heuristic. - 5 new unit tests in `test/ccxt/http_test.exs` under a new `request/4 sentinel-only error_body_checks` describe block: synthetic Hyperliquid-shape `error_body_check` (`field: "status"`, `roles: [:status_sentinel]`, sentinels `===` `"err"` / `"unknownOid"`) β†’ `{"status": "err"}` and `{"status": "unknownOid"}` classify as error; `{"status": "ok"}` and missing-field bodies stay success; a kucoin code-role guard test confirms `200000` still classifies as success (no regression on the inverted code-bearing path). - Verified the two error-classification tests fail without the fix via targeted `git stash push lib/ccxt/http.ex` + rerun: 2/5 new tests fail without the gate change, confirming the tests exercise the regression path. The other 3 already passed because their cases (`:unknown` β†’ success fallthrough, missing field, kucoin code-role) didn't depend on the loosened gate. **Why:** - T49 (Error code field config, schema 1.7.0) shipped per-exchange `error_body_checks` populated from `structure.handle_errors.error_code_fields`, but left two regression classes flagged in ROADMAP. One was Hyperliquid: spec correctly emits `field: "status"`, `roles: ["status_sentinel"]`, `sentinel_values: [{"===", "err"}, {"===", "unknownOid"}]`, but the consumer-side classifier in `CCXT.HTTP.classify_by_eq_success/1` (line 395) and `classify_by_error/1` (line 403) required `has_code_role: true` for `===` matches. Pure sentinel-only entries fell through to `:unknown`, then to the hardcoded `error_code_fields` fallback (`code` / `ret_code` / `retCode` / `error_code`) β€” none of which exist on Hyperliquid responses β€” and the error response classified as success. - The `has_code_role: true` gate was originally written to prevent mixed-role entries (`error_code` + `status_sentinel`, e.g. KuCoin's `200000`) from double-classifying β€” there the `===` match is the *success* indicator, not an error. Sentinel-only entries are a valid input the gate wasn't designed for. **Key decisions:** - **Single new function head, not a refactor of existing logic.** The two existing clauses (line 395 for code-role + sentinel mixed entries, line 399 fallback) stay byte-for-byte identical. The new clause sits between them and only fires when `has_code_role: false` and `eq_vals` exist. Minimum-blast-radius β€” no possibility of regressing the KuCoin/Bithumb/Binance success paths that the existing gate guards. - **Scope rejected: `error_overrides` registry on `%CCXT.Exchange{}`.** T64's original framing called for a per-exchange override map keyed by exchange id, with `:code_and_message` and `:status_sentinel_only` semantic flavors. Rejected because (a) only one of the three named exchanges (Hyperliquid) is in priority-tier scope today β€” Coinmate and Indodax are tier3; (b) Hyperliquid's bug isn't a missing-data case the spec can't express, it's a downstream interpretation error in `CCXT.HTTP`; (c) the trajectory of recent landings (Tasks 53 / 90 / upstream 73c / 61a–c) is "richer extraction β†’ thinner consumer," and an override registry cuts against that grain. Long-term `treat_as` semantics belong in upstream Phase 13 / Task 58 (error contract), not consumer-side overrides. - **Three-strikes counter does not advance.** This is patch #5 on the error-classifier family by raw count, but the rule applies to *extensions* of the heuristic surface (new role combinations, new field-name lists, new special cases). Removing a too-restrictive gate that was never intended to apply to sentinel-only entries is a gate-correction, not a heuristic extension. The CHANGELOG and the rescoped ROADMAP entry both name this distinction explicitly. - **Coinmate/Indodax (tier3) cases stay parked.** Their regression shape is different (`usable_error_body_check?` drops fields with both `error_code` and `error_message` roles entirely β€” not a downstream gate issue but an upstream filter issue). When/if a consumer surfaces those exchanges via `--exchange `, the fix lives in `CCXT.Exchange.usable_error_body_check?`, not in `CCXT.HTTP`. Not pre-built. - **Re-extraction was offered then skipped.** `mix ccxt_extract.update --tier1 --tier2 --dex` was attempted to confirm the regression repros against fresh specs, but `--output` redirection to the client's spec dir is currently broken in ccxt_extract (it pulls source-lookup paths along with write paths). Confirmed instead by inspecting the existing `priv/specs/json/output/hyperliquid.json`, which is already at schema 2.0.1 (post-Task 90 sweep) β€” the field shape that would drive the regression is current. ccxt_extract `--output` bug is a separate issue, not in scope here. **Verification:** - `mix test.json --quiet test/ccxt/http_test.exs` β€” 35/35 passed (+5 new). - `mix test.json --quiet` β€” 1118 passed, 253 excluded, 0 failed. - `mix credo --strict` β€” 0 new issues (14 pre-existing TODO suggestions, none introduced). - Targeted reverse check: `git stash push lib/ccxt/http.ex` β†’ `mix test.json` β†’ 2/5 new tests fail with `{:ok, %{...}}` left-vs `{:error, %Error{}}` right, confirming the tests exercise the gate. Stash restored. **Out of scope (deliberate):** - ccxt_extract `--output` source-path bug (extraction stage looks for `priv/ccxt/ts/src` under the redirected output dir instead of the project's own priv). Real bug but outside this task and outside the project boundary per the "STOP: ccxt_extract Is a Separate Project" rule. - Coinmate/Indodax (tier3) `code_and_message` regression shape β€” different fix surface, parked until consumer surfaces. - Long-term richer error semantics (`treat_as`, retry classification, class hierarchy) β€” upstream Phase 13 / Task 58. --- ### 2026-04-18: Task 53 β€” Schema 2.0.0 / provenance adapter (fail-fast guards in `CCXT.Spec`) **What was done:** - `CCXT.Spec.load!/1` now validates every per-exchange spec after JSON decode and before structure stripping: `schema_version` must be a string whose numeric major equals `2`, and `_provenance` must be a non-empty top-level map. Mismatches raise with a message naming the exchange and the specific offending field (missing `schema_version`, missing `_provenance`, invalid `_provenance`, unsupported major). - `CCXT.Spec.load_manifest!/0` gains the same `schema_version` major guard β€” without the `_provenance` check, since `_manifest.json` is a different document type and has never carried `_provenance` by design. - Net-new code in `lib/ccxt/spec.ex`: `@schema_major 2`, `validate_schema!/2` (four function heads covering present/invalid/missing combinations), `validate_manifest_schema!/1`, and a shared `assert_major!/2` helper. Moduledoc updated with a "Schema Version" section documenting the contract. - 6 new unit tests in `test/ccxt/spec_test.exs` covering a real-spec happy path and five sad paths (non-2 major, missing / null / empty `_provenance`, missing `schema_version`). Sad paths use a small `write_tmp_spec!/2` test helper that writes a JSON fixture into the live spec dir via `CCXT.Spec.spec_path/1` and registers `on_exit` cleanup. **Why:** - Upstream ccxt_extract shipped Schema 2.0.0 (2026-04-17) making the top-level `_provenance` map a required, non-null field (RFC 6901 pointer keys β†’ `"raw"` / `"derived"` / `"override"`); upstream Task 61d added a `provenance_covers_schema` contract-test invariant so consumers can trust `_provenance` is exhaustive over emitted sections. The client's loader, however, never enforced `schema_version` at all β€” a stale `priv/specs/json/output/` or a future regression would silently load malformed data and surface downstream as an opaque `MatchError`. These guards convert that silent failure into a specific compile-time raise. - Task 90 (2026-04-17) already swept the spec dir up to 2.0.1 and confirmed the loader *transparently* accepted the new schema; Task 53 completes the adapter by adding the fail-fast layer the upstream contract now justifies. **Key decisions:** - **Guard lives in the loader, not in downstream code.** Validation runs once per spec load, before strip, so any consumer (`CCXT.Exchange` generator at compile time, `CCXT.Registry`, `CCXT.Signing.Classifier`, test helpers) gets the same contract for free. No opt-in flag, no wrapper β€” this is a hard invariant. - **Message names the spec.** Error messages embed `inspect(exchange_id)` (or the literal `"manifest"`) and the specific failing field, so a bad file under `priv/specs/json/output/` points at its source directly instead of surfacing as a generic decode error. - **No `CCXT.Spec.provenance/1` accessor.** Minimalist first. `_provenance` is already reachable via `spec["_provenance"]`; adding a one-line wrapper before any consumer surfaces would be speculative. - **Manifest gets only the major guard.** `_manifest.json` has `schema_version` but no `_provenance` by design (different document type). Enforcing major 2 there keeps the manifest and spec loaders in lockstep without inventing a contract the manifest doesn't carry. - **Validation before strip.** `strip/1` only touches `structure.*` today, but the ordering is defensive β€” if `@stripped_keys` ever expands to include a top-level field, the guard has already run on the raw map. - **Dead `@parent_map` entry in `lib/ccxt/signing/classifier.ex:37` (`"exchange_v1" => "hitbtc"`) left as-is.** It's an exchange-ID coincidence with the renamed schema descriptor, not a reference to the schema file; unrelated to this task and flagged for a future hygiene pass. - **`.sobelow-skips` regenerated via `mix sobelow --mark-skip-all`.** The file tracked pre-existing `File.read!` false positives (`spec_path`/`manifest_path` are validated by `validate_id!/1`) by line number; the new guards shifted those line numbers. **Verification:** - `mix test.json --quiet test/ccxt/spec_test.exs` β€” 30/30 passed (+6 new). - `mix test.json --quiet` β€” 1107 passed, 253 excluded, 0 failed (baseline: 1101 passed, +6 new). - `mix credo --strict lib/ccxt/spec.ex test/ccxt/spec_test.exs` β€” 0 issues. - `mix sobelow --exit` β€” exit 0. **Out of scope (flagged for follow-up, not done here):** - Stale doc-comment version references in `lib/ccxt/exchange.ex` / `lib/ccxt/dispatch.ex` (`"schema 1.7.1+"`, `"schema 1.8.0"`) β€” prose only, no logic. - Dead `"exchange_v1"` entry in `lib/ccxt/signing/classifier.ex` `@parent_map`. - `CCXT.Spec.provenance/1` accessor β€” add when a consumer surfaces. --- ### 2026-04-17: Task 90 β€” Adopt `structure.request_defaults` from upstream ccxt_extract Task 73c **What was done:** - Refreshed `priv/specs/json/output/` to schema 2.0.1 β€” 23 priority-tier exchange specs + aggregates pulled from upstream ccxt_extract's regenerated output. Renamed embedded schema file `exchange_v1.json` β†’ `exchange_v2.json`. Manifest filtered to preserve the `tier1 + tier2 + dex` scope the client has always compiled. - Added `:request_defaults` field to `%CCXT.Exchange{}` struct (default `%{}`), populated at `Exchange.new/2` construction time from `spec["structure"]["request_defaults"]`. Filters to `kind: "literal"` entries only β€” `kind: "unresolved"` entries are dropped per the Honesty Rule (`lib/ccxt/exchange.ex` `build_request_defaults/1`). - Added `CCXT.Unified.maybe_merge_request_defaults/3` β€” merges per-method literal defaults into the caller's params with `Map.put_new` semantics (caller-supplied params win, matching CCXT JS's `this.extend(request, params)` precedence). Wired into `CCXT.Unified.call/5` immediately after the T86 symbol-denormalization step and before `Dispatch.call/4`. - 6 new unit tests in `test/ccxt/unified_test.exs` covering: literal merge into empty caller params, caller overrides default, defaults merge alongside unrelated caller keys, unknown-method no-op, empty-defaults graceful degradation, and a real-spec check that hyperliquid materializes `fetchTime β†’ %{"type" => "exchangeStatus"}`. **Why:** - Task 81 diagnosed hyperliquid's entire public surface as blocked by CCXT JS's per-method `const request = { … }` literal-body injection (not extracted to spec pre-73c). Upstream Task 73c shipped 2026-04-17 at schema 2.0.1, emitting `structure.request_defaults` as `method β†’ {key β†’ {value, kind, reason}}` and explicitly naming `hyperliquid.fetchTime` as its concrete failure driver. This task adopts that data on the consumer side. **Key decisions:** - **Insertion point: `Unified.call/5`, not `Dispatch.call/4`.** Matches the T86 symbol-denormalization boundary β€” `Unified` owns unified-method semantics (including per-method default-body injection); `Dispatch` stays generic so raw generated endpoint callers (`CCXT.Hyperliquid.public_post_info/3`) bypass the merge and pass exchange-native params through untouched. - **Materialize on struct, not accessed via module.** Same pattern as `symbol_patterns`, `signing_pattern`, `error_body_checks` β€” populate once at `Exchange.new/2` time, read at call time. No per-call spec walk. Key shape is `%{js_method_name => %{field => value}}` which matches `Unified.call/5`'s existing `capability_name` arg. - **Caller wins over defaults.** `Map.put_new` preserves caller-supplied values; defaults only fill absent keys. Mirrors CCXT JS's `extend(request, params)` precedence and avoids silently overriding caller-controlled routing (e.g., a call to `fetch_time` with `type: "spotMeta"` still routes to `spotMeta`, not `exchangeStatus`). - **`kind: "literal"` only.** Upstream 73c's `kind: "unresolved"` entries represent per-call values (variable references, conditional expressions) that would need caller-side resolution logic beyond a flat merge. Per the Honesty Rule those are left for upstream to resolve; the consumer-side merge is trusted to fill in only unambiguous literals. - **Spec scope preserved.** Upstream regenerated all 110 exchanges at schema 2.0.1. We kept the existing 23 priority-tier set (`tier1 + tier2 + dex`) by copying only those exchange files and post-filtering the manifest to match. No scope drift from the refresh. - **`CCXT.Spec` loader already accepted the new schema.** The loader never enforced `schema_version`, so schema 2.0.0 β†’ 2.0.1 + exchange_v2.json rename required no loader changes. Task 53's "Schema 2.0.0 / provenance adapter" line-item in ROADMAP remains open only for the optional `_provenance` accessor work β€” not blocking. **Verification:** - Runtime check: `CCXT.Exchange.new!("hyperliquid").request_defaults["fetchTime"]` returns `%{"type" => "exchangeStatus"}`. 20 hyperliquid unified methods carry literal defaults. - Merge semantics: empty caller β†’ default merged; caller with `type: "spotMeta"` β†’ caller value preserved; unknown method β†’ no-op; empty request_defaults map β†’ no-op. - Full suite green via `mix test.json --quiet --summary-only` (1088+ tests). - Spec refresh: `priv/specs/json/output/` files all report `"schema_version": "2.0.1"`; manifest scoped to 23 exchanges. **Follow-ups:** - Live network re-run of T40 hyperliquid.fetch_time probe to confirm the flunk flips to pass. Deferred to on-demand run (`mix test.json --include network --only public_probe`). - Hyperliquid's full public surface (`fetchMarkets`, `fetchCurrencies`, `fetchOrderBook`, `fetchTrades`, `fetchFundingRates`, `fetchStatus`, `fetchOpenInterest`, `fetchOHLCV`, `fetchFundingRateHistory`, etc.) should now traverse the correct request shape end-to-end; the T39 unified-integration probe will exercise those when run against testnet. - Task 81 tier3 subset (aftermath/ascendex/oxfun/deepcoin/paradex/ndax) remains on-demand β€” some likely share the 73c bug class now resolved, others (deepcoin query param, paradex auth-misclassification) are distinct and won't benefit. ### 2026-04-17: Task 81 β€” Public-endpoint request-shape diagnosis (priority subset) **What was done:** - Investigated the single remaining priority-tier failure from Task 40's public-endpoint probe: `hyperliquid.fetch_time` returning `"Failed to parse the request body as JSON"`. - Diagnosis: spec correctly routes `fetchTime β†’ publicPostInfo` (`priv/specs/json/output/hyperliquid.json` unified_endpoints); `CCXT.HTTP` correctly sends JSON POST bodies when params are non-empty (`lib/ccxt/http.ex:204-209`); the missing piece is the per-method default body `{ "type": "exchangeStatus" }` that CCXT JS's `fetchTime()` injects into params before dispatch (`ccxt_extract/priv/ccxt/ts/src/hyperliquid.ts:425-434`). This injection lives in the method body and is not extracted to spec today. **Why:** - T81's job is diagnosis, not patching β€” the original description explicitly requires splitting findings into "ccxt_extract upstream asks vs. consumer-side fixes." Patching around a spec-level gap in ccxt_client would violate the 🚨 STOP rule. **Key decisions:** - **No ccxt_client code changes.** Upstream ccxt_extract **Task 73c** (🎁 **11-request**, Phase 11) already tracks this exact fix β€” explicitly names `hyperliquid.fetch_time` as the concrete failure driving the task. Task 73c emits `structure.request_defaults` as `method β†’ {key β†’ {value, kind, reason}}`, schema bump 1.8.0 β†’ 1.8.1. - **Follow-up consumer-side task created: Task 90** β€” when 73c ships, merge `structure.request_defaults[method]` entries with `kind == "literal"` into params at `Unified.call/5`, sibling to the existing Task 86 symbol-denormalization hook. Unblocks all hyperliquid public endpoints (`fetchMarkets`, `fetchCurrencies`, `fetchOrderBook`, `fetchTrades`, `fetchFundingRates`, `fetchStatus`, `fetchOpenInterest`, `fetchOHLCV`, `fetchFundingRateHistory`, etc. β€” all 16+ operations route through `POST /info` with a `type` discriminator). - **Tier3 subset (aftermath, ascendex, oxfun, deepcoin, paradex, ndax) left as on-demand notes.** Some likely share the 73c bug class; others look different (deepcoin's `instType field is required` is a missing *query* param; paradex's `invalid bearer jwt` on public `fetch_time` suggests auth misclassification). Not compiled by default β€” reproduce via `--exchange ` if a consumer surfaces need. - **T81 row left in Active Track table** (status flipped to πŸ”Ά) rather than moved out, so the tier3 breakage notes stay visible for on-demand reproduction. Priority subset also tracked in the ⏸ Deferred β€” Awaiting Upstream table pointing at Task 73c + Task 90. **Verification:** - Runtime check: `CCXT.Hyperliquid.__unified_endpoint__(:fetch_time)` returns `%{method: :post, path: "info", sections: ["public"], authenticated: false, weight: 20, name: :public_post_info, url_prefix: "/"}` β€” spec routing is correct. - Grep `priv/specs/json/output/hyperliquid.json` for `exchangeStatus`: present as a `byType` cost entry (`describe.api.public.post.info.byType.exchangeStatus: 2`) but **not** as a default body value for `fetchTime`. Confirms the spec-level gap. - Cross-reference CCXT JS `priv/ccxt/ts/src/hyperliquid.ts:425-434` β€” `fetchTime()` injects `{ 'type': 'exchangeStatus' }` in method body, then calls `publicPostInfo`. - Upstream `ccxt_extract/ROADMAP.md` line 223 Task 73c text names `hyperliquid.fetch_time` as the concrete failure. **Follow-ups:** - Task 90 (new, upstream-blocked on ccxt_extract Task 73c) β€” consume `structure.request_defaults` in `Unified.call/5`. ### 2026-04-17: Task 39 β€” Unified-method integration probe **What was done:** - `CCXT.Test.Generator.UnifiedMethodIntegrationProbe` β€” compile-time generator emitting per-`(exchange, method)` tests for the unified-method surface T40/T79/T67 don't already cover: - Public zero-arg: `fetch_status`. - Public symbol-required: `fetch_order_book`, `fetch_trades`, `fetch_funding_rate`. - Private zero-arg with real testnet credentials: `fetch_balance`, `fetch_open_orders`, `fetch_my_trades`, `fetch_positions`, `fetch_account`, `fetch_trading_fees`. - `CCXT.Test.Generator.SymbolResolver` β€” shared helper extracted from `SymbolPublicEndpointProbe` (T79). Both T79 and T39 now call `SymbolResolver.pick_symbol/1` for compile-time unified-symbol resolution against the spec's `runtime.markets.markets` map. - Consumer test module `CCXT.UnifiedMethodIntegrationTest` β€” three lines; tags (`:network` + `:unified_integration` at module level, plus `:public` / `:private` + `:exchange_{id}` per test) are emitted from inside the generator's `__using__/1` quote. - Compile-time emission covers every priority-tier exchange across the public_zero / public_symbol / private categories, gated per `(method, category)` by endpoint-authenticated flags, spec-resolved symbol availability, and signing-pattern filters. **Why:** - T40 (zero-arg public) + T79 (symbol public) + T67 (private invalid creds) each cover a narrow slice of the unified pipeline. None of them produce an end-to-end pass/fail for a private call with real credentials, and none cover `fetch_order_book` / `fetch_trades` / `fetch_status` / `fetch_funding_rate` β€” all of which are high-value unified methods a consumer actually uses. T39's job per ROADMAP is to gather per-exchange breakage evidence feeding the unified-normalization decision (T44 transformers vs. T64 overrides vs. wait-for-upstream-Phase-12). Gap-fill produces that evidence with the lowest blast radius. **Key decisions:** - **Gap-fill, not replacement.** The roadmap's original T39 description called for one configurable generator replacing T40/T79/T67. Rejected: the three existing probes are shipped, producing stable breakage maps, and restructuring them to gain configurability adds no evidence value while risking regressions. The new probe sits alongside them. - **Method coverage added `fetch_status` and `fetch_funding_rate` as cheap extras** beyond the T39 roadmap description's minimum set β€” both have trivial signatures (zero-arg / symbol-only) and their endpoint mappings exercise request paths not otherwise touched. - **Private path uses `IntegrationHelper.require_credentials!/2`** β€” flunks with the exact env-var export commands when testnet creds aren't registered, per the project rule in `~/.claude/includes/critical-rules.md` ("NEVER skip tests silently on missing credentials"). The `@moduletag :network` gate keeps these out of default `mix test` runs; explicit `--include network` surfaces the missing-creds diagnostics as actionable setup instructions. - **Compile-time credential gating rejected.** Option: emit private tests only for exchanges where `Testnet.registered?/2` is true at compile time. Problem: credential state at compile time β‰  credential state at test time, and forcing recompiles when env vars change trades a clear flunk diagnostic for silent coverage drift. - **`:custom` signing patterns filtered out of the private path only.** Public endpoints bypass signing entirely, so T39 keeps the T40/T79 convention of not filtering there. Mirrors the `InvalidCredsProbe` filter for the private side. - **Sandbox key is `:default`** β€” for binance/binanceusdm/binancecoinm variance the `_FUTURES` sandbox slot isn't wired in. Tracked implicitly via per-method credential-slot selection if evidence runs surface need; no point building the override now. - **Write-path methods (`create_order`, `cancel_order`, `withdraw`, …) deliberately excluded** β€” captured as **Task 89** with prerequisites (test-asset config module, testnet-only flunk guard, chained cancel-after-create in the same test body). Half-built write infrastructure is worse than none: tests exist without the config they need to run safely, and `--include dangerous` six months from now hits unconfigured defaults. **Verification:** - Full suite green via `mix test.json --quiet --summary-only`. New generator doesn't affect the default test run β€” all emitted tests are `:network`-gated. - Compile-time inspection via `UnifiedMethodIntegrationProbe.__collect_for_inspection__/0` confirms every priority-tier exchange emits at least one case and coverage spans the three categories (public_zero / public_symbol / private). - Network evidence run deferred to on-demand execution (`mix test.json --include network --only unified_integration`). **Follow-ups:** - Task 89 (new) β€” write-method probe, blocked on per-exchange test-asset config module. ### 2026-04-17: Task 86 β€” Wire symbol denormalization into unified dispatch **What was done:** - `CCXT.Unified.call/5` now invokes `CCXT.Symbol.to_exchange_id/2` on `params["symbol"]` when the key is present and the value is a binary, before delegating to `Dispatch.call/4`. The conversion is exposed as `CCXT.Unified.maybe_denormalize_symbol/2` so it's unit-testable in isolation. - Binance-style callers see `"BTC/USDT"` β†’ `"BTCUSDT"`; Binance linear perp `"BTC/USDT:USDT"` β†’ `"BTCUSDT"`; Deribit swap `"BTC/USD:BTC"` β†’ `"BTC-PERPETUAL"`. Common-currency reversal (Kraken `BTC`β†’`XBT`) flows through `Symbol.to_exchange_id/2`'s existing logic. - 7 new test cases in `test/ccxt/unified_test.exs` cover: spot/swap happy paths, Deribit PERPETUAL shorthand, no-symbol-key no-op, non-binary-value no-op, empty-`symbol_patterns` graceful no-op, and pass-through of other params. **Why:** - Task 79's first run produced 9 pass / 32 flunk, with the overwhelming majority of failures coming from unified `"BTC/USDT"` reaching Binance/OKX/Bybit/Kucoin/HTX as a raw query param. `CCXT.Symbol.denormalize/2` and `to_exchange_id/2` had been ported from bak but were never wired into the unified dispatch path β€” in bak, the only caller was the WebSocket layer. **Key decisions:** - **Insertion point: `Unified.call/5`, not `Dispatch.call/4`.** Dispatch also serves raw generated endpoint callers (e.g., `CCXT.Bybit.public_get_v5_market_tickers/3`) who are expected to pass exchange-native symbols. Denormalizing inside Dispatch would double-convert for raw callers. `Unified.call/5` is the semantic boundary where `"BTC/USDT"` *means* a CCXT unified symbol; below it, everything should already be exchange-native. - **Market-type inference from the symbol shape, not endpoint metadata.** `Symbol.to_exchange_id/2` already calls `parse_extended/1` β†’ `detect_market_type/1` internally: `"BTC/USDT"` β†’ `:spot`, `"BTC/USDT:USDT"` β†’ `:swap`, `"BTC/USDT:USDT-260327"` β†’ `:future`, `"BTC/USD:BTC-260112-84000-C"` β†’ `:option`. No new metadata threading needed. - **Graceful degradation.** Exchanges without `symbol_patterns` data in spec get `symbol_patterns: %{}`; `to_exchange_id/2` falls through to the input unchanged. No flunks on exchanges that don't have pattern classification. - **Narrow scope by design.** Only singular `"symbol"`. No `:symbols` plural β€” verified not present as a required param in any `method_defs/0` entry. No param-key renaming (`symbol` β†’ `pair`/`instId`/`instrument_name`). Key-renaming is a separate concern belonging to the endpoint config's param mapping (upstream Phase 11 / Task 57 when it lands). **Verification:** - Full suite green (`mix test.json --quiet --summary-only` β†’ 1095 pass / 0 fail / 116 excluded). - T79 probe re-run deferred to next integration session; offline unit coverage confirms the conversion executes for the Binance-/Deribit-style patterns that drove the original T79 breakage count. **Follow-ups on deck:** - Task 87 (candidate): wire `:symbols` plural for batch methods (`fetch_tickers`) if a future probe surfaces breakage. - Task 88 (candidate): param-key renaming β€” parked behind upstream Phase 11 request-building contract. --- ### 2026-04-17: Task 79 β€” Symbol-required public probe **What was done:** - `CCXT.Test.Generator.SymbolPublicEndpointProbe` β€” compile-time generator sibling to `PublicEndpointProbe`, covering `fetch_ticker` (`[symbol]`) and `fetch_ohlcv` (`[symbol, "1m"]`) per exchange. - Symbol resolution pulls the full spec at compile time via `CCXT.Spec.load!/1` (markets live under `runtime.markets.markets`, which the lean `__spec__` strips). Preference list: `BTC/USDT` β†’ `ETH/USDT` β†’ `BTC/USD` β†’ `BTC/USDC` β†’ `ETH/USDC` β†’ `BTC/USDT:USDT` β†’ `ETH/USDT:USDT` β†’ `BTC/USDC:USDC` β†’ `BTC/USD:BTC` β†’ `ETH/USD:ETH`, falling through to first-spot / first-swap / first-market. Futures-only exchanges (binanceusdm, binancecoinm, kucoinfutures) now pick liquid perps instead of arbitrary first entries. - Consumer test module `CCXT.SymbolPublicEndpointProbeTest` β€” `@moduletag :network` + `:symbol_public_probe` emitted from inside the generator's `__using__/1` quote (same retroactivity concern as `PublicEndpointProbe`). - Selection skip conditions: no compiled module, no matching unified endpoint, any config with `authenticated: true`, no resolvable symbol. **Why:** - Task 40's `PublicEndpointProbe` covered only zero-arg methods (`fetch_time` / `fetch_currencies` / `fetch_markets`). Symbol-scoped endpoints are a distinct correctness surface: symbol denormalization, per-pair path interpolation, and pair-specific response shapes. T79 closes the gap with spec-driven symbol selection β€” new exchanges added via `--exchange ` automatically get probe coverage without hand-maintained config. **Evidence captured (first run):** - 9 of 41 emitted tests pass: `coinbaseexchange.fetch_ohlcv`, `deribit.fetch_{ohlcv,ticker}`, `gate.fetch_{ohlcv,ticker}`, `gateio.fetch_{ohlcv,ticker}`, `kraken.fetch_ohlcv`, `lighter.fetch_ohlcv`. - 32 fail in tight clusters that point at a single root cause (surfaced as a new task β€” see below): - **Binance family, OKX family, Bybit, HTX/Huobi, Kucoin:** unified `"BTC/USDT"` handed directly to the exchange API as a raw param. Binance returns `-1121 Invalid symbol` on `fetch_ticker`; `-1102 Mandatory parameter X sent empty` on `fetch_ohlcv`. OKX returns `50014`. Bybit `10001`. Kucoin `200003` / `300000`. Symbol denormalization is wired in `CCXT.Symbol` but not called anywhere in `CCXT.Dispatch` or `CCXT.Unified`. - **Kraken, lighter, derive:** response-shape mismatches. Kraken returns a map of tickers keyed by native symbol (`XXBTZUSD`); the helper's `has_price_field?` check expects price fields at the top level. Lighter's `fetch_ticker` returns `{"code", "order_book_details", ...}` β€” the spec/endpoint mapping likely resolves to a non-ticker endpoint. - **Bitfinex:** 404 on both β€” likely URL/prefix bug specific to the symbol-scoped path. - **Hyperliquid.fetch_ohlcv:** `:exchange_error` with `code: nil` β€” overlaps with T81 (POST-body shape for hyperliquid). **Key decisions:** - Spec-driven > hand-maintained symbol map. Tested option (a) informally against option (c) "hybrid" and rejected: duplicating spec data creates a drift surface and fails exactly when upstream adds a new exchange. - `:custom` signing patterns NOT filtered (public pipeline bypasses signing; matches Task 40's rationale). - `"1m"` baked in for `fetch_ohlcv` β€” every priority-tier spec declares `"1m"` in `runtime.describe.timeframes` (verified). Per-exchange timeframe override deferred until a spec breaks this assumption. **Follow-up:** - New **Task 86** added to ROADMAP for the symbol denormalization fix β€” the dominant breakage class. Until that lands, most T79 `fetch_ticker`/`fetch_ohlcv` tests are expected to flunk on the Binance/OKX/Bybit/HTX/Kucoin families. The probe itself is a correctness gate; the failures are real bugs, not test defects. ### 2026-04-17: Task 85 β€” `@spec_dir` adopts ccxt_extract's split read/write layout **What was done:** - `CCXT.Spec.@spec_dir` changed from `"priv/specs/json"` to `"priv/specs/json/output"` β€” both the compile-time attribute and the `:code.priv_dir/1` runtime fallback in `spec_dir/0`. - `git mv` relocated all 28 files from `priv/specs/json/` into `priv/specs/json/output/`: 24 exchange specs (tier1 + tier2 + dex + aliases) and four metadata files (`_manifest.json`, `_base_methods.json`, `_validation_report.json`, `_contract_test_report.json`). Content unchanged β€” pure relocation. - Moduledoc and doc-example path strings updated in `lib/ccxt/spec.ex` and `lib/ccxt/exchanges.ex` to reflect the new location. **Why:** - Upstream `ccxt_extract` shipped REFACTOR Item 9 (2026-04), splitting `Paths.priv/1` (read) from `Paths.out/1` (write). Consumers that pass `--output ` now receive files at `/output/.json`, not `/.json`. Before this change, the next `mix ccxt_extract.update --output ../ccxt_client/priv/specs/json` would have silently written to a new `output/` subdirectory while the client kept reading stale flat files β€” a latent trap with no surface-level error. **Verification:** - Full suite green (1088 passed, 75 excluded) β€” Registry, generator, manifest loading, and the 24 test spec files all resolve correctly against the new path. **Key decisions:** - Option A chosen (relocate in place) over Option B (re-run `ccxt_extract.update`). Relocating preserves exact spec content so the change is byte-identical for every downstream consumer β€” no risk of unrelated regressions leaking in from an intervening upstream extraction diff. Users can re-run `mix ccxt_extract.update` whenever they want fresh data; that's now decoupled from the layout fix. ### 2026-04-16: Task 80 (htx + huobi subset) β€” Per-section hostname overrides **What was done:** - `CCXT.Exchange.new/2` now threads `urls["hostnames"]` through URL resolution. `resolve_urls/3` overlays per-section hostname overrides onto the `urls.api` map when the override entry is a string keyed by the same section. Nested-map entries (htx's `status` page by market type) are left on the default hostname β€” no consumer today, and market-type-aware lookup needs a separate design. - `compute_url_prefixes/2` pulls `urls["hostnames"]` from the spec and passes it through so url_prefix path extraction runs against the correct per-section base URL. - Unit tests in `exchange_test.exs` assert `htx.base_urls["contract"] == "https://api.hbdm.vn"` and the huobi alias inherits the same override. **Bug fixed:** - T40 probe flunked `htx.fetch_time` and `huobi.fetch_time` with HTML 404. htx's `urls.api` uses `"https://{hostname}"` uniformly, but the actual `contract` section lives on `api.hbdm.vn` β€” that hostname is captured in `urls.hostnames.contract` which no code read. Every section was collapsing to the `describe.hostname` default (`api.huobi.pro`), so contract requests hit the wrong host. Post-fix, `contract` resolves to `api.hbdm.vn` and url_prefix path extraction works because the base URL now prefix-matches the absolute `url_prefix` from `runtime.url_templates`. **Scope:** - Priority-tier subset only (htx + huobi). The remainder of Task 80 covers tier3 exchanges with different URL/prefix bugs that aren't compiled in the default Registry β€” deferred until a consumer surfaces need. - Only htx and huobi populate `urls.hostnames` across the 23 priority-tier specs (verified), so the new code path is a no-op for every other exchange. - T40 probe priority-tier failure count: 3 β†’ 1 (hyperliquid/T81 remains). **Key decisions:** - Overrides live in the consumer (not upstream ccxt_extract) because `urls.hostnames` is already correctly extracted. This is pure interpretation-gap territory β€” add the field to the `resolve_urls` pipeline, done. No three-strikes or config-over-inference question: the spec data is authoritative and structured. - Nested-map override values deliberately unsupported. When a future task needs htx/huobi status-page endpoints, market-type routing is a separate design problem; adding a half-measure now would seed inference debt. ### 2026-04-15: Task 84 β€” Surface `exchange.tier` on Exchange struct **What was done:** - `:tier` field added to `%CCXT.Exchange{}` defstruct + typespec. Populated by `Exchange.new/2` from `spec["exchange"]["tier"]` (nil-safe via bracket access β€” older specs without the field return `nil`). - Public `Exchange.tier/1` accessor for ergonomic struct access. - `__tier__/0` introspection function emitted on every generated exchange module at compile time, mirroring the `__id__`/`__name__`/`__signing__` pattern. Extracted once in `prepare_generate_data/1`, rendered in `build_introspection_ast/1`. - `CCXT.exchange/2` Descripex `returns` description now documents that the result carries `:tier` with enumerated values (`"tier1"` / `"tier2"` / `"dex"` / `"tier3"` / `"unclassified"` / `nil`). **Key decisions:** - **Scope collapsed from original task (2026-04-15 pre-implementation).** The `@moduletag tier:` gating half of Task 84 is moot now that `CCXT.Registry` is already scoped by extraction (`--tier1 --tier2 --dex`) β€” test generators iterate only priority-tier exchanges without any tag filtering. Only the consumer-query surface (struct + helper + introspection + Descripex) was implemented. - Struct access preferred over `__tier__/0` for runtime callers with a `%CCXT.Exchange{}` in hand; `__tier__/0` remains for compile-time-only or module-level queries. - Credo cyclomatic-complexity disable on `build_introspection_ast/1` β€” the quote block now emits one more `def` and tipped Credo's counter to 10. The function is a pure AST constructor; each `def` is not a decision point in the semantic sense. ### 2026-04-15: Tier-rebase executed β€” Registry shrunk to 23 exchanges **Operational follow-up to the tier-scope policy decision below.** Ran `mix ccxt_extract.update --tier1 --tier2 --dex --output ../ccxt_client/priv/specs/json` from ccxt_extract. Result: 87 tier3 / unclassified spec files pruned; `priv/specs/json/` now holds tier1 (5) + tier2 (6) + dex (4) + aliases = 23 compiled exchange modules. **Test fallout (adjusted in-repo):** - `test/support/test_exchange.ex` β€” `CCXT.TestExchange.BingX` swapped to `CCXT.TestExchange.HTX` (tier2 β€” 7 top-level API sections, depth 6; stresses the recursive endpoint-flattening path more than BingX did). Follow-up edits to `exchange_generator_test.exs` + `signing/classifier_test.exs` to match. - Deleted 10 tests whose unique canary was a pruned tier3 exchange: krakenfutures prefix inheritance, apex + bitget passphrase classification, Ascendex camelCase section mapping, bitbank error_code_fields fallback, bit2c null hostname, MEXC nested URL, bithumb "0000" success sentinel, edge-case IDs with digits (bit2c/p2b), dYdX opaque symbol patterns, tokocrypto dispatch-gate pair. Also removed the Kucoin `fetch_uta_ohlcv` regression canary (upstream `fetchUTAOHLCV` is gone from the priority-tier specs; the `js_to_atom` canonical mapping still handles the underlying `Macro.underscore` mangling, just no current spec stresses it). - `spec_test.exs` "loads a small exchange" swapped bit2c β†’ deribit. - `registry_test.exs` hardcoded `> 100` / `"aftermath"` / `"zonda"` asserts replaced with `>= 20` + `Enum.sort(exchanges) == exchanges` β€” resilient to scope changes. **CLAUDE.md scope refresh:** - Removed three stale `/110` numerics (endpoint path templates, signing auto-classification rate, Config-Over-Inference dataset bullet) β€” replaced with qualitative claims. - Canonical deep-nesting example reference updated: "(BingX/Gate)" β†’ "(HTX/Gate)". **ROADMAP reprioritization:** - T69 / T70 / T72 transitioned ⬜ β†’ πŸ”Ά **dormant**. Post-rebase T67 + T40 network probes (19 pass / 3 fail / 20 inconclusive) reproduce zero T69/T70/T72 symptoms. The only remaining priority-tier failures are T80 (htx/huobi fetch_time 404) and T81 (hyperliquid fetch_time JSON body). Dormant rows carry reactivation note ("reactivate via `--exchange ` if a consumer surfaces one"). - Tier-rebase note rewritten from forward-looking prediction β†’ verified outcome. **Verification:** - `time mix compile --force` β€” 13s, clean. - `mix test.json --quiet` β€” 1081 pass, 0 fail, 75 excluded (network + fixture-replay gates). - `mix test.json --quiet --include network test/ccxt/invalid_creds_probe_test.exs test/ccxt/public_endpoint_probe_test.exs` β€” 19 pass, 3 fail (T80 Γ— 2 + T81 Γ— 1), 20 inconclusive. ### 2026-04-15: Tier-scoped extraction adopted as routine policy **Decision:** Switched routine spec extraction from `mix ccxt_extract.update --all` (110 exchanges) to `mix ccxt_extract.update --tier1 --tier2 --dex` (~20 exchanges). Matches upstream ccxt_extract's derivation-scope decision exactly (canonical `priv/priority_tiers.json`). **Mechanics:** Scoped runs prune out-of-scope files from `priv/specs/json/` (ccxt_extract Task 2 behavior). `CCXT.Registry` is generated from the `priv/specs/json/*.json` glob, so it auto-shrinks to ~20 exchange modules. No runtime gating needed β€” Registry *is* the filter. A specific tier3 / unclassified exchange can be added on demand via `--exchange ` (scoped runs are additive, don't prune priority tiers). **Knock-on roadmap changes:** - Vision statement (ROADMAP.md, CLAUDE.md) updated from "110 exchange client library" to "priority-tier client library (~20 exchanges)". - CLAUDE.md Quick Commands updated with four variants: routine `--tier1 --tier2 --dex`, skip-setup variant, on-demand `--exchange `, full-sweep `--all`. - Tier-Scope Policy section in ROADMAP.md reframed β€” tier3 / unclassified are now "not compiled by default" rather than "compiled but test-gated." - T84 collapsed from "Surface `exchange.tier` + tier-gate test generators" to "Surface `exchange.tier` on Exchange struct" (D:1/B:3/U:4 β†’ Eff:3.5). Test-tag + `test_helper.exs` opt-in portion becomes redundant because Registry already filters. - T69 / T70 / T72 flagged as likely dormant post-rebase β€” every affected exchange in those tasks is tier3. Bug classes are still real defensive-coding issues but drop to "proactive hardening" priority. Re-run T67 / T40 post-rebase to confirm. - T80 (htx/huobi tier2) + T81 (hyperliquid dex) priority subsets move to top of queue β€” tier3 subsets become on-demand via `--exchange `. **Rationale:** - Baseline bug-discovery from tier3 exchanges (T69/T70/T72) is already captured. The bug classes are in shared dispatch/construction code β€” future regressions will surface through priority-tier exchanges anyway. - Compile time + noise + MCP tool-list surface all shrink. - Consumer friction for tier3 is low (`--exchange ` is a one-liner). - "Full CCXT replacement" framing was aspirational β€” tier-scoping matches the project's actual derivation coverage honestly. **Not done:** the spec rebase itself (`mix ccxt_extract.update --tier1 --tier2 --dex --output ...`) is a separate operational step, not part of this docs change. Run it when ready to shrink Registry. ### 2026-04-14: Non-Unified Track triage β€” defer upstream-superseded work **Decision:** Narrowed the Active Track to consumer-only bugs + integration-evidence gathering. Moved T68 and T73–T78 (fixture-replay signing patches), T71 (error-classifier override seeding), and the "Custom signing modules" backlog into a new **⏸ Deferred β€” Awaiting Upstream Signing/Request Contract** section in ROADMAP.md. **Why:** - ccxt_extract's critical path (Phases 9–11, bundles 🎁 **9-pipeline** / **10-core** / **10-HMAC** / **11-shape**) is shipping spec-driven signing recipes, request-building contract, and override provenance. - Consumer-side tasks T54 (retire `Signing.Classifier`), T56 (spec-driven pattern modules), T57 (adopt request-building contract), and T64 (error overrides) already sit in Upstream-Blocked β€” the fixture-replay patches would either be retired by those upstream deliverables or duplicate the eventual data model. - T74 explicitly fronts a ccxt_extract schema change (timestamp format) β€” no consumer work is possible until upstream ships. - Three-strikes rule already cited on T73 classifier Tier-2. **What stays active:** - T69 / T70 / T72 β€” consumer-only bugs unrelated to upstream (construction crashes, endpoint-mapping privacy filter, xt header leak). - T39 / T79 / T80 / T81 / T82 / T83 β€” integration evidence gathering. Highest leverage for deciding unified normalization strategy. **Re-entry trigger:** Pick the deferred tasks back up when the relevant upstream bundle lands, or earlier if T39 integration evidence surfaces a correctness blocker that can't wait. ### Task 82: IntegrationHelper hardening β€” Cloudflare split + empty-body βœ… **Scope:** Resolve two IntegrationHelper issues surfaced by the Task 40 first run that are independent of any exchange bug, and undo the over-broad `:access_restricted` inconclusive fix that was applied post-T40. **What was done:** - New error atom `:cloudflare_challenge` in `CCXT.Error` (added to the `error_type` union, `@non_recoverable_types`, `@moduledoc` listing, and a dedicated `cloudflare_challenge/1` factory built on the existing `build/3` helper). No change to `@spec_class_mapping` β€” CF detection is purely heuristic, not spec-class driven. - New `classify_html_response/3` in `CCXT.HTTP` replaces the previous `build_access_restricted_error/3`. Routes HTML responses to `:cloudflare_challenge` when the page title matches `~r/just a moment/i` or `~r/attention required/i`, or the body preview contains `cf-chl-bypass`, `challenge-platform`, or `cf-browser-verification`. All other HTML (wrong URL, 404 landing, auth-challenge HTML, unknown providers) stays `:access_restricted`. Markers are a module attribute β€” easy to extend as new CF variants appear. - Empty/whitespace-only 2xx body (status β‰  204) now returns `:network_error` at the HTTP layer via a new `empty_body_error/2` helper, called from `handle_success_body/4`. Prevents the `{:ok, %{status: 200, body: ""}}` tuple from escaping the HTTP boundary and fixes `cryptocom fetch_currencies` flunking in per-method shape validation. - `test/support/integration_helper.ex`: removed `:access_restricted` from `@inconclusive_public` (T80 canary restored β€” non-CF HTML responses flunk again), added `:cloudflare_challenge` to both `@inconclusive_public` and `@inconclusive_private_auth`. `:access_restricted` stays on `@inconclusive_private_auth` because a valid-creds call landing on a geo/IP block isn't a code bug. - `test/ccxt/http_test.exs`: 4 new cases β€” Cloudflare title match β†’ `:cloudflare_challenge`, CF body-marker match β†’ `:cloudflare_challenge`, generic HTML (title "Not Found") stays `:access_restricted` (T80 canary regression guard), empty + whitespace-only 200 bodies β†’ `:network_error`. - Refactored 2xx handler: `handle_response/4` now delegates to `handle_success_body/4`, keeping both handler clauses grouped and flattening nesting depth. **Why the split:** Codex correctly flagged that the post-T40 `:access_restricted` β†’ `@inconclusive_public` fix masked genuine T80 URL/prefix bugs (bitso 404, yobit 200 HTML landing, alpaca 401 HTML, bullish 403 landing, htx/huobi/latoken 404) alongside the legitimate btcbox Cloudflare case. The probe lost its canary function for future T80-class regressions. Splitting CF challenges off preserves both: btcbox stays inconclusive with an actionable log; real URL bugs flunk. **Out of scope:** - T80 URL fixes themselves β€” this task restores the probe's ability to surface them, doesn't address individual exchange URL bugs. - HTTP 204 No Content handling β€” no probe exchange hits it; address if/when it surfaces. - Telemetry events for HTML detection β€” existing `log_inconclusive` in the helper remains the hook. **Files:** `lib/ccxt/error.ex`, `lib/ccxt/http.ex`, `test/ccxt/http_test.exs`, `test/support/integration_helper.ex`, plus this CHANGELOG + ROADMAP updates. **Post-review fixes (Codex 2026-04-15):** - **HEAD regression in empty-body guard.** `handle_success_body/4` treated any non-204 empty 2xx as `:network_error`, misclassifying HEAD responses (body-less by definition) even though `:head` is declared as supported in `build_request/5`. Latent β€” no HEAD callers exist today β€” but the contract was inconsistent. `handle_response` now threads `method` through and `handle_success_body/5` skips the empty-body check when `method == :head`. New regression test in `test/ccxt/http_test.exs` locks this in. - **Contract coverage gap for `:cloudflare_challenge`.** Added `cloudflare_challenge/1` factory + `@non_recoverable_types` assertion in `test/ccxt/error_test.exs`. Added public/private path split coverage in `test/ccxt/integration_helper_test.exs` β€” CF inconclusive on both paths, `:access_restricted` flunks on public (T80 canary) but inconclusive on private (geo/IP block with valid creds). ### Task 40: Public-Endpoint Pipeline Probe βœ… **Scope:** narrower than the original Task 40 description. The "raw endpoints from `__endpoints__/0` with `path_param_defaults` and `:dangerous` write-path tagging" surface is retained as Task 83 in ROADMAP.md; what landed here is the zero-arg unified-method probe only. **What was done:** - New compile-time test generator `CCXT.Test.Generator.PublicEndpointProbe` iterates `CCXT.Registry.exchanges/0` and emits one ExUnit test per exchange for which a zero-arg public unified method is available. - Candidate methods checked in order: `:fetch_time`, `:fetch_currencies`, `:fetch_markets`. An endpoint qualifies only if every config returned by `module.__unified_endpoint__(method)` has `authenticated: false` β€” the probe stays strictly on the public code path. - Each generated test builds `%CCXT.Exchange{}` via `CCXT.Exchange.new!/1` (no credentials), invokes the selected method through `apply(CCXT, method, [exchange])`, and delegates classification to `CCXT.IntegrationHelper.assert_public_response/3` (rate-limit / network / exchange-unavailable / geo-block treated as inconclusive; structural issues and 4xx/5xx flunk). - Consumer file `test/ccxt/public_endpoint_probe_test.exs` uses the generator; `@moduletag :network` + `@moduletag :public_probe` are emitted from **inside the generator's `__using__/1` quote** so they register before the test blocks and survive Styler's `use`-grouping rewrites in the consumer file. Default runs skip the suite. - Offline helper `__collect_for_inspection__/0` exposes the compile-time selection so the generated exchange set can be inspected before any network traffic. **Post-review fixes (Codex 2026-04-14):** - **Probe was leaking into the default test run.** Original version declared `@moduletag :network` in the consumer file *after* `use ...PublicEndpointProbe`, so the tags never attached to the already-registered tests (ExUnit reads `@ex_unit_moduletag` at each `test` macro expansion, not retroactively). Empirical evidence: `mix test.json` on the probe file reported `excluded: 0, total: 104`; full-suite failures dropped from 25 to 0 under `--exclude public_probe`. Moved moduletag emission into the generator's quote β€” default runs now correctly report `excluded: 104`. - **`:access_restricted` was flunking on the public path.** `IntegrationHelper.@inconclusive_public` only covered `:rate_limit_exceeded / :network_error / :exchange_not_available`, contradicting the docstring and the ROADMAP promise that geo-blocks log inconclusive rather than flunk. Added `:access_restricted` to the list so Cloudflare challenges, VPN-required responses, and geo-blocks behave as infrastructure issues instead of code bugs. **Key decisions:** - `:custom` signing patterns are NOT filtered out (unlike T67) β€” public endpoints bypass signing, so custom-signing exchanges still exercise a meaningful public pipeline. - `:fetch_ticker` deliberately excluded from v1 β€” it requires a per-exchange symbol. Follow-up tracked as Task 79 (symbol-aware public probe). - Delegates validation to existing `IntegrationHelper.assert_public_response/3` rather than reimplementing classification β€” single source of truth for public inconclusive rules. **First-run evidence (seeded follow-up tasks):** - Extends T69's enumeration: `luno` crashes in `Access.get/3` during `Exchange.new!` β€” same shape as `independentreserve`/`lbank`/`mercado`/`paymium`. - T73 validated: `ascendex` "Invalid Http Request Input" (100004) matches the classifier Tier-2 misroute bucket. - **T80** (new): ~9 exchanges return HTML instead of JSON on public endpoints (`alpaca`, `bitso`, `bullish`, `coinbaseinternational`, `htx`, `huobi`, `latoken`, `yobit`, plus `btcbox` Cloudflare) β€” URL resolution / prefix-inheritance bugs on the public path. - **T81** (new): 5+ exchanges where the URL is right but the request shape is wrong (`hyperliquid` wants JSON POST body, `aftermath`/`grvt`/`tokocrypto`/`oxfun`/`deepcoin` missing params, `paradex`/`ndax` possibly mislabeled as public). - **T82** (new): `IntegrationHelper.validate_body!/2` flunks on empty-string body (should be inconclusive); Cloudflare challenges should get a dedicated classification. ### T65 fixture replay triage (no code change) 145 of 146 classified exchanges fail byte-equal signing parity against CCXT JS fixtures. Decomposed into six root-cause buckets and added T73–T78 to the Active Track. Bucket E (Binance POST body/URL placement) already tracked as T68. - **T73** Classifier Tier-2 fallback misroutes unclassified sha256 exchanges to Bybit `X-BAPI-*` config β€” blocker; masks real counts of T74–T77. - **T74** Timestamp format hardcoded per pattern (ms/s/ISO/Β΅s/nonce) β€” fronts a ccxt_extract schema change (no timestamp-format field today). - **T75** Param insertion order not preserved through Dispatch β†’ Signing β€” structural fix. - **T76** Body encoding + extra headers not spec-driven β€” `:body_encoding` + `:extra_headers` config keys. - **T77** Signature encoding (base64 vs hex) not consistently spec-resolved. - **T78** CI gate on fixture replay green count β€” do last. Fixtures verified authoritative (e.g. `apex.json` contains literal `"APEX-TIMESTAMP": "1700000000000"` ms epoch matching CCXT JS; ccxt_client emits ISO instead). ### Task 67: Invalid-Credentials Private-Pipeline Probe βœ… **What was done:** - New compile-time test generator `CCXT.Test.Generator.InvalidCredsProbe` iterates `CCXT.Registry.exchanges/0`, skips `:custom` signing patterns and modules that expose neither `:fetch_balance` nor `:fetch_accounts` on their unified mapping, and emits one ExUnit test per remaining exchange. - Each test builds `%CCXT.Credentials{}` with a random `INVALID_` api_key (avoids any exchange-side caching of known-bad keys), a valid-base64 zero-filled secret, and placeholder `password`/`uid` (harmless when unused), then invokes the picked unified method and classifies the response: - `{:error, %CCXT.Error{type: :authentication_error | :permission_denied}}` β†’ pass. - `:rate_limit_exceeded` / `:network_error` / `:exchange_not_available` / `:access_restricted` β†’ logged as `⚠️ INCONCLUSIVE`, does not flunk. - `{:ok, _}` β†’ flunk (signature accepted with bogus key β€” real bug). - Any other shape β†’ flunk with the raw response for diagnosis. - `test/ccxt/invalid_creds_probe_test.exs` β€” thin shell with file-level `@moduletag :network` + `:invalid_creds`, `async: false` to serialize real-exchange traffic. Both tags are added to the default `ExUnit.start(exclude: ...)` list in `test/test_helper.exs` β€” module tags alone don't exclude; the runner must know them. Run with `mix test --include network`. - ROADMAP entries added for Task 67 (probe) and Task 68 (HmacSha256Query POST body/URL placement β€” real signing divergence surfaced by T65 fixture replay). **Why:** Task 41 proves our signing output is structurally correct. Task 65 proves the bytes match CCXT JS. Neither proves the **full private pipeline** (DNS β†’ URL resolution β†’ signing β†’ transport β†’ response parsing β†’ error classification) actually reaches a real exchange and gets correctly interpreted. The invalid-credentials probe fills that gap: if the pipeline produces an auth-shaped error, every link worked except the key itself. No credential story required β€” hence `@moduletag :network` is the only gate. Run with: mix test.json --quiet --only network test/ccxt/invalid_creds_probe_test.exs **First-run findings (seeding the T69–T72 backlog):** - **T69 β€” `Exchange.new!/2` crashes (5 exchanges):** `digifinex` raises `no function clause matching in CCXT.Error.from_spec_class/1`; `independentreserve`, `lbank`, `mercado`, `paymium` raise `no function clause matching in Access.get/3`. These modules are currently unusable for any private call. - **T70 β€” Endpoint privacy routing (5 exchanges):** `bit2c`, `bitbank`, `coinone`, `exmo`, `krakenfutures` returned `{:ok, _}` with bogus creds β€” almost certainly `__unified_endpoint__(:fetch_balance)` resolves to a public endpoint on these. Fix: filter by `endpoint_config.authenticated` when dispatching unified methods expected to be private. - **T71 β€” Error classifier gaps (~14 exchanges):** clear auth failures misclassified as `:exchange_error` / `:bad_request` / `:exchange_not_available`. Examples: `bithumb` "Invalid Apikey", `poloniex` "Invalid Apikey or Signature", `coinsph` -2018 "API-key does not exist", `indodax` "Sign not found in header", `bittrade` "login-required", `blofin` 152400 "access-nonce cannot be empty", plus `apex` / `bitfinex` / `whitebit` / `hashkey` / `tokocrypto` / `zebpay` / `woo` / `oxfun`. Pair with T49/T64 override track; the probe output is the override seed list. - **T72 β€” `xt` query fragment landing in headers:** `Req.HTTPError{reason: {:invalid_header_name, "&xt-validate-recvwindow="}}` β€” a URL query pair is being written as a header name. Distinct from T68 (body/URL placement); this is query/headers placement. ### Task 65: Byte-Equal Signing Parity Tests βœ… **What was done:** - New compile-time test generator `CCXT.Test.Generator.FixtureReplay` enumerates `../ccxt_extract/priv/fixtures/signing/*.json` at compile time and emits one ExUnit test per private case, tagged `:fixture_replay`. Registers each fixture file + `_manifest.json` as `@external_resource` so regeneration triggers recompile. - Supporting pure helpers in `CCXT.Test.Generator.RequestBuilder`: fixture-credential β†’ `%Credentials{}` mapping, URL split (`scheme://host[:port]` vs path vs query), sorted-header normalization, first-byte-diff error messages. - Determinism threaded through `CCXT.Signing` via config-map overrides (`:timestamp_ms_override`, `:nonce_override`) β€” no change to `sign/4` arity. New helpers `timestamp_ms_from_config/1`, `timestamp_seconds_from_config/1`, `timestamp_iso8601_from_config/1`, `nonce_from_config/2`. All 8 non-custom pattern modules updated. - `test/ccxt/signing_fixture_replay_test.exs` β€” thin shell using the macro. - Suite excluded from default `mix test` via `:fixture_replay` tag (follows `:integration`/`:dangerous` pattern). Run with `mix test --include fixture_replay`. - **Scope:** generator skips `:custom`-classified exchanges at compile time β€” no byte-equal contract exists for them against the generic pattern modules. Extending coverage to `:custom` signers is follow-up work (paired with the 19 custom-signing modules backlog). Missing `../ccxt_extract/priv/fixtures/signing` no longer raises at compile time β€” emits a warning via `Mix.shell().error/1` and produces zero tests, so fresh checkouts without the sibling repo still compile. **Why:** Task 41 verifies *structural* signing shape from exchange introspection (header presence, signature length, encoding); T65 verifies *byte-equal cryptographic parity* against CCXT JS truth. Shape passing β‰  bytes matching. Comparison strategy is path-only: fixture URL is parsed into `scheme://host + path + query`, `request.path` fed to signing is the full fixture path (including `url_prefix`), and `scheme_host <> signed.url` is compared to the fixture URL verbatim. This keeps the signal scoped to signing correctness β€” Dispatch URL resolution is tested separately (T52). **Findings on first run (follow-up backlog):** 146 private cases / 145 failures surface real divergences between our signing patterns and CCXT JS β€” e.g. Apex misclassified as `:hmac_sha256_iso_passphrase` (CCXT uses ms timestamp), Apex POST body serialized as JSON instead of urlencoded form, Arkham classified with Bybit-style headers but upstream uses `Arkham-Expires` microseconds + base64. Each is a standalone classifier-or-pattern fix, not a test-infra issue. Existing 360 signing_verification tests remain green β€” `sign/4` API is unchanged. ### Task 66: sha384_payload variant classifier βœ… **What was done:** - `CCXT.Signing.Classifier.extract_payload_config/1` now detects Gemini vs Bitfinex variant by probing extracted headers for a `payload` header. When present, emits `variant: :gemini` + `payload_header`. When absent, emits `variant: :bitfinex` + `nonce_header`. - Regenerated specs: Gemini and Bitopro now classify as `:gemini` variant (were falling through to Bitfinex branch); Bitfinex proper keeps `:bitfinex`. **Why:** `CCXT.Signing.HmacSha384Payload.sign/3` dispatches on `Map.get(config, :variant, :bitfinex)`. Without explicit `:variant` in the spec, Gemini-family exchanges were silently signed with the Bitfinex algorithm β€” a real signing correctness bug masked by Task 41 shape tests that had inferred the variant from emitted headers (circular). Task 41 helper now reads `:variant` authoritatively and flunks if absent, surfacing any future regression. ### Upstream: signing fixtures now available (enables Task 65) ccxt_extract shipped `mix ccxt_extract.signing_fixtures` β€” a frozen-input / frozen-output fixture extractor that calls CCXT JS's `exchange.sign()` under deterministic credentials, timestamps, nonces, and `randomBytes`. Fixtures live at `../ccxt_extract/priv/fixtures/signing/.json` (107 exchanges, byte-deterministic across runs). Each fixture captures CCXT's exact `sign()` output for up to three cases (`public_get_ticker`, `private_get_balance`, `private_post_order`) with preserved wire-contract casing (`X-BAPI-SIGN`, `OK-ACCESS-TIMESTAMP`, etc.). This unblocks **Task 65: Byte-Equal Signing Parity Tests** β€” a higher-bar companion to Task 41's shape tests. Task 41 verifies format correctness from exchange introspection; Task 65 will verify byte-equal cryptographic parity against CCXT truth. Shape β‰  correctness. ### Task 41: Signing Verification Tests βœ… **What was done:** - Added `CCXT.Test.Generator.SigningTests` (`test/support/test_generator/signing_tests.ex`) β€” compile-time `__using__/1` macro that iterates `CCXT.Registry.exchanges/0`, resolves each module via `Registry.module_for/1`, reads `__signing__/0`, and emits a `describe` block per exchange with 4 tests: basic sign succeeds, signature format matches pattern, required headers present, timestamp/nonce format correct. - Added `CCXT.Test.Generator.SignatureHelper` (`test/support/test_generator/signature_helper.ex`) β€” ported from `../ccxt_client_bak` with pattern-default encoding fixes. Validates hex vs base64 signatures at SHA256/384/512 lengths, required header keys per pattern, ms/seconds/iso8601/nonce timestamp formats, and the Deribit `deri-hmac-sha256 id=...,ts=...,sig=...,nonce=...` authorization envelope. - Added `test/ccxt/signing_verification_test.exs` β€” single consumer: `use CCXT.Test.Generator.SigningTests`. - Tests tagged `:signing` (offline; run by default). Per-exchange `exchange_{id}` tag for spot checks. **Key decisions:** - `:custom` exchanges are skipped at generation time β€” they intentionally bypass the shared pattern contract. Run `mix ccxt.classify_signing` for the list. - Default signature encoding is pattern-derived (e.g., `hmac_sha256_iso_passphrase` β†’ base64, `hmac_sha384_payload` β†’ hex) when the spec config omits `:signature_encoding`. Aligns with what each signing module actually produces. - Default header names mirror each signing module's fallbacks (`@pattern_default_headers`) β€” helper asserts against what the signer actually emits rather than silently skipping when spec config omits a header-name key. Unresolvable keys `flunk` with an actionable message. Closes the silent-skip hole flagged in post-merge review. - `:hmac_sha384_payload` variant read authoritatively from `signing[:variant]` (populated by Task 66 classifier). Helper `flunk`s if absent β€” refuses to infer from emitted headers, which would mask classifier gaps. - `:hmac_sha512_nonce` nonce validated from signed request body (per `body_encoding`) rather than silently skipped β€” Kraken-style signers put the nonce in the body, not a header. - Header-timestamp validation falls back to the signer's default header name (e.g., `X-BAPI-TIMESTAMP` for `:hmac_sha256_headers`) when spec config omits `:timestamp_header`, instead of silently skipping. - Dummy secret is valid base64 (`dGVzdHNlY3JldGJ5dGVzZm9yc2lnbmluZ3Rlc3QxMjM0NTY=`) so Kraken-style patterns that base64-decode the secret don't fail on the dummy value. - No changes to `lib/`, `mix.exs`, or `test_helper.exs` β€” the generator consumes already-exposed introspection (`__signing__/0`, `Registry.exchanges/0`). - Foundation for T39/T40: same compile-time iteration pattern will extend to unified-method and raw-endpoint integration test generation. ### Task 52: Section Visibility Config βœ… **What was done:** - Regenerated specs from ccxt_extract schema 1.7.1 β€” 39 exchanges gained non-empty `structure.authenticated_sections` (inheritance propagation, alternate-branch AST walker, 14 per-exchange overrides upstream). Zero regressions. - `CCXT.Exchange.build_endpoint_configs/3` now takes `authenticated_sections` and sets `:authenticated` (boolean) on each endpoint config. The top-level section of each endpoint's `sections` path is checked against the spec list. - `CCXT.Dispatch.call/4` reads `endpoint_config[:authenticated]` instead of running a case-insensitive substring match on section names. Deleted `private_endpoint?/1`. - `@type endpoint_config` in `CCXT.Dispatch` gained `authenticated: boolean()`. - Tests: existing dispatch signing tests updated to set the flag explicitly; new tests in `exchange_generator_test.exs` cover build-time flag population and a real-spec Bybit assertion (private section authenticated, public not). **Key decisions:** - Top-level section check (not per-section-path membership): all authenticated sections in observed specs are top-level API tree keys. Matches ccxt_extract's extraction contract. - Substring fallback **removed** (originally planned as transitional). With schema 1.7.1 every exchange that has private endpoints reports them β€” no residual heuristic. - Resolves the "Private endpoint detection" bullet under CLAUDE.md's Config Over Inference β€” heuristic replaced with spec data. Closes the three-strikes loop (commit 154f6f6 patch #1 β†’ this replacement). ### Task 38: Integration Test Helpers βœ… **What was done:** - Added `CCXT.IntegrationHelper` (`test/support/integration_helper.ex`) β€” shared assertion primitives for the Phase 4b test generators (T39/T40/T41). - `require_credentials!/2` returns a `%CCXT.Credentials{}` from the `CCXT.Testnet` registry or `flunk/1`s with the exact env var names (`{EXCHANGE}[_{SANDBOX}]_TESTNET_API_KEY`/`_API_SECRET` + optional `{EXCHANGE}_PASSPHRASE`) to set. - `build_exchange/2` wraps `CCXT.Exchange.new!/2` and splits a Credentials struct into the keyword opts the constructor expects; supports `:sandbox`, `:hostname`, `:options`. - `assert_public_response/3` and `assert_private_response/3` implement the inconclusive-vs-flunk contract: rate-limit / network / exchange-not-available errors log a warning and return `:ok`; private-endpoint auth/permission/access errors are also inconclusive (likely testnet or geo issue, not code bug); `:allow_not_found` / `:allow_invalid_order` / `:allow_no_position` opt-in allow specific exchange errors. - Two-stage response detection: parsed unified struct (Phase 5) takes the `validate_struct/2` path (stub returning `:ok` until Task 42 lands validators); raw `%{status, body}` envelopes validate 2xx + method-specific shape (price-like field for tickers, bids/asks for order books, list/map shape for trades/ohlcv/balance). - `unwrap/1` walks one level into `result` / `data` / `response` envelopes so shape checks work against unparsed exchange responses. **Key decisions:** - Helpers are functions, not macros β€” `flunk/1` gives an actionable message without macro ceremony. - Inconclusive classifications never hide failures: they log an `⚠️ INCONCLUSIVE` warning and return `:ok`. Every other error class flunks with a multi-line message including type/code/message. - `validate_struct/2` is intentionally a stub; when Task 42 lands it will delegate to `CCXT.StructValidators` and every T39 test activates stronger assertions automatically β€” no test-file churn. ### Task 49: Error Code Field Config β€” πŸ”Ά PARTIAL **Status:** Patch landed but two regression classes remain. Marked partial in ROADMAP; no further in-session patches per CLAUDE.md "Three-Strikes Heuristic Rule" (this is patch #4 on the same heuristic family). **Regressions found:** - `exchange.ex` `usable_error_body_check?` excludes any entry with `"error_message"` in roles, which drops mixed-role fields where one field simultaneously carries the error code and a human-readable message (observed on Coinmate, Indodax) - `http.ex` `evaluate_sentinel_value` gates `===` sentinel error branches on `:error_code in check.roles`, so pure `:status_sentinel` entries (Hyperliquid) always fall through to `:unknown` and never flag an error **Why not upstream ask #5:** This is the 4th ccxt_extract round on body-error semantics. Three prior rounds have refined role classification but not converged on a contract consumers can apply without a role-interpretation layer. That non-convergence is itself a signal the boundary is genuinely interpretive β€” consistent with the Three-Strikes Heuristic Rule. Prefer narrow per-exchange overrides as the bridge. **Deferred resolution path (separate task):** 1. Add an `error_overrides` field to `%CCXT.Exchange{}` keyed by exchange id for Coinmate, Indodax, Hyperliquid (and any siblings surfaced by integration tests) 2. Override carries explicit `{field, semantics}` where semantics is `:code_and_message` (treat as error when non-empty/non-zero, regardless of role filtering) or `:status_sentinel_only` (treat `!==` as success, `===` as error without requiring `:error_code` role) 3. Runtime checks override first, falls back to current spec-derived `error_body_checks` for the 100+ exchanges where the heuristic is correct 4. Overrides live in ccxt_client (not ccxt_extract) β€” they are consumer-side role interpretation, not extraction gaps **What was done (original task β€” still applies to non-regressing exchanges):** - Re-extracted specs to schema 1.7.0 (from 1.4.0) β€” adds `structure.handle_errors.error_code_fields` with role classification - Added `error_code_fields: [String.t()]` to `%CCXT.Exchange{}` β€” per-exchange list of response fields to probe for error codes - Added `error_body_checks` to `%CCXT.Exchange{}` β€” top-level `handleErrors()` checks with role metadata and sentinel values for runtime body-error detection - `Exchange.new/2` builds the list from `structure.handle_errors.error_code_fields`, keeping top-level `response` entries whose `roles` include `"error_code"` and do not include `"error_message"`, then collecting the `field` + `field2` (safeString2) names, deduped in spec order - Non-string fields (e.g., Bitfinex's integer array indices) are filtered out β€” current runtime only probes map bodies - Hardcoded fallback `~w(code ret_code retCode error_code)` applies when the spec has no usable entries (or handleErrors is absent) - `CCXT.HTTP` now evaluates top-level sentinel checks before falling back to the legacy exact-code probe, preventing false positives on success payloads like KuCoin `code: "200000"` and Bithumb `status: "0000"` - `CCXT.HTTP.extract_error_code/2` now takes the per-exchange field list instead of hardcoding four names **Key decisions:** - **Config over inference, per the CLAUDE.md design principle.** ccxt_extract 1.5.0+ classifies each `safe*` call in `handleErrors()` by how its value is used (`error_code` vs `error_message` vs `status_sentinel`). For the current runtime, only top-level `response` fields with an `error_code` role and no `error_message` role are relevant for exact-match lookup; message fields belong to the broad-pattern path. - **Sentinel checks stay narrow.** The runtime now consumes top-level `status_sentinel` data for body-level success/error detection, but it does not attempt nested `object_path` traversal or full handler-routing logic from upstream Phase 13. - Kept the fallback list for exchanges without `handle_errors` (would otherwise silently miss errors). Fallback covers the most common conventions β€” per-exchange config takes precedence when available. - Did not attempt to traverse `object_path` for nested fields (e.g., OKX's `error.sCode` at path `[response, data, 0]`). Current `check_body_error/2` only inspects the top-level body; nesting traversal is a larger change out of scope here. ### Task 37: Testnet Credential Registry **What was done:** - Added `CCXT.Testnet` β€” ETS-backed credential registry under Application supervision - Supports multi-API exchanges via per-sandbox keys (`:default`, `:futures`, `:coinm`) - `register_from_env/2,3` reads `{EXCHANGE}[_{SANDBOX}]_TESTNET_API_KEY|_API_SECRET` and optional `{EXCHANGE}_PASSPHRASE` - `register_all_from_env/1` batch registration from `test_helper.exs` - `creds/1,2` returns `%CCXT.Credentials{}` or `nil`; `creds!/2` raises with actionable env var names - `sandbox_key_from_url/1` derives the sandbox key from a URL (path `/dapi` β†’ `:coinm`, host containing `future` β†’ `:futures`) - `test_helper.exs` now excludes `:integration` and `:dangerous` tags β€” integration tests never run by accident **Key decisions:** - ETS (not Agent as in `../ccxt_client_bak`) β€” lock-free reads align with existing `RateLimiter.State` pattern and scale to the 110-exchange integration test matrix - GenServer owns the `:named_table`, `:public`, `read_concurrency: true` table β€” restart-safe, any process can read - Unregistered `creds!/2` raises with the exact env vars to set (e.g. `BINANCE_FUTURES_TESTNET_API_KEY`), matching the "fail loudly on missing credentials" rule from CLAUDE.md ### Task 21b: Unified Type Schemas **What was done:** - Added `JSONSpec.schema()` definitions to all 34 unified type structs - Each struct gets a `@json_schema` module attribute (compile-time) and `schema/0` public function - Schemas include `doc:` descriptions derived from each struct's `@moduledoc` Fields section - Meta-test validates all 34 schemas: exports, properties match struct fields, atomize roundtrip **Key decisions:** - Module attribute pattern (`@json_schema schema(...)` + `def schema, do: @json_schema`) β€” avoids name collision between `def schema` and the imported `schema/2` macro, and pre-computes the map at compile time - Nested struct refs (e.g., `CCXT.Fee.t()` in Order) β†’ `map()` β€” JSONSpec doesn't support cross-module struct references; the nested struct's own `schema/0` is available for deeper validation - Dynamic-key maps (e.g., Balance's `%{String.t() => number()}`) β†’ `map()` β€” JSONSpec handles fixed-key maps only - OrderBook's `[[number()]]` bids/asks use the correct nested array schema β€” JSONSpec supports recursive list conversion ### URL Prefix: Config Over Inference (Codex Review Fix) **What was done:** - Rewrote `compute_url_prefixes/2` to read `url_prefix` from specs (schema 1.2.0) instead of deriving via heuristics - Deleted `strip_sample_path/2` β€” the hostname collision bug that corrupted btcmarkets prefix - New approach: ccxt_extract provides full URL prefix, ccxt_client strips base URL (from `urls.api`) to get path prefix - Added `path_from_url_prefix/2`, `section_base_url/2`, `navigate_to_url/2` helpers - Re-extracted specs to pick up ccxt_extract's Bug 2 fix (Gate null resolved_url sections) **Bugs fixed:** - btcmarkets: `strip_sample_path("https://api.btcmarkets.net/v3/markets", "markets")` split on "markets" in hostname, producing wrong prefix. Now reads `/v3/` directly from spec. **Key decisions:** - Config over inference for URLs β€” URLs are finite, verifiable against exchange docs, and easy to get wrong with heuristics. ccxt_extract probes runtime, ccxt_client consumes the answer. - URI path fallback when section has no matching base URL in `urls.api` (e.g., OKX uses "rest" key) ### URL Prefix Integration (Phase 4) **What was done:** - Integrated `runtime.url_templates` from ccxt_extract into endpoint URL construction - Each endpoint config carries a `url_prefix` field (e.g., `/api/v5/`, `/spot/`, `/api/v3/`) - `Dispatch.call/4` prepends `url_prefix` instead of hardcoded `"/"` β€” fixes ~50% of exchanges **Key decisions:** - Prefix computed at compile time (zero runtime cost) β€” stored on each endpoint config map - Private sections inherit prefix from matching public sibling (e.g., `private.spot` β†’ `public.spot`) - Default `"/"` preserves backwards compatibility for exchanges without url_templates - Known limitation: url_templates is per-section by design (one sample per section). Exchanges with mixed API versions within a section (e.g., KuCoin v1/v3 in `public`) get the sampled version's prefix for all endpoints. Correct for the majority of endpoints; a few may get the wrong version prefix. ### Prefix Inheritance Fix (Codex Review) **What was done:** - Fixed `maybe_inherit_prefix/2` β€” namespaced private sections (e.g., `contract.private`, `utaPrivate`) no longer fall back to top-level `"public"` prefix - Added `build_private_candidates/1` with constrained candidate generation: sibling replacement first, then stripped suffix, no bare `"public"` fallback for namespaced sections **Bugs fixed:** - HTX `contract.private` inherited `/v1/` from top-level `"public"` when `contract.public` was filtered by hostname mismatch β€” produced `/v1/api/v1/...` double prefixes - KuCoin `utaPrivate` inherited `/api/v3/` from `"public"` instead of `/api/ua/v1/` from `uta` (no `utaPublic` exists) - Finding 3 (CoinEx mixed paths) not applicable β€” no exchange has endpoint paths with embedded version prefixes **Verified live:** OKX (`/api/v5/`), Gate.io (`/spot/`), Bybit, Binance β€” all returning 200 with correct data ### Roadmap: Porting Analysis + New Tasks (44-48) **What was done:** - Compared current ccxt_client vs old ccxt_client_bak to validate the rewrite decision - Identified which old modules are "exchange domain knowledge" (port) vs "spec format plumbing" (leave behind) - Added Task 44: Response Shape Transformers (Phase 5) β€” the old `ResponseTransformer` with 12 transform types for non-standard exchange response shapes, missing from the roadmap - Added Phase 8: Trading Utilities with Tasks 45-48 β€” Multi-exchange parallel fetch, Order Builder, Order Sanity, Method Emulation - Updated porting reference in ROADMAP.md header with specific port/leave-behind guidance **Key decisions:** - Rewrite was the right call β€” spec loading pipeline, type system, Exchange struct, and unified API generation are architecturally different - WebSocket system (Phase 6) should port the old code β€” 14 sub patterns, 9 auth patterns, ~80 per-exchange modules encode format-independent domain knowledge - Response shape transformers are a Phase 5 prerequisite β€” without them, parsers receive wrong shapes for exchanges like BitMEX - Phase 8 tasks are lower priority but capture proven patterns worth preserving ### Task 43: Missing Unified Type Structs **Completed** | Phase 4b | [D:3/B:8/U:9 β†’ Eff:2.83] **What was done:** - Audited all 241 unified methods in `CCXT.Unified.method_defs/0` and identified every implied return type - Created 27 new unified type structs (34 total), covering all standardized CCXT return types - Tier 1 (core): OrderBook, Position, Currency, Transaction, LedgerEntry, FundingRate, DepositAddress, TransferEntry, TradingFee - Tier 2 (derivatives): Leverage, OpenInterest, Liquidation, Greeks, OptionData, LeverageTier, MarginMode, MarginModification, ADLRank - Tier 3 (analytics/account): LongShortRatio, FundingHistory, FundingRateHistory, Conversion, Account, LastPrice, DepositWithdrawFee, BorrowRate, BorrowInterest - Added helpers: OrderBook (`best_bid/1`, `best_ask/1`, `spread/1`), Position (`long?/1`, `short?/1`, `profitable?/1`), Transaction (`deposit?/1`, `withdrawal?/1`, `pending?/1`) - ~30 methods with exchange-specific/opaque returns identified as not needing structs (e.g., `fetch_time`, `set_leverage`, `fetch_status`) **Key decisions:** - `OptionData` not `Option` β€” avoids confusion with Elixir concept - Unified `BorrowRate` for cross+isolated β€” nil `symbol` means cross; avoids two near-identical structs - Flat namespace maintained β€” 34 struct files at `lib/ccxt/*.ex` is manageable - All fields `| nil` except OrderBook's `bids`/`asks` (default `[]`), Currency's `networks` (default `%{}`), DepositWithdrawFee's `networks` (default `%{}`) - `Transaction` covers both deposits and withdrawals (CCXT pattern); `fee` field is `CCXT.Fee.t()` not plain number - `TradingFee` is the fee schedule per symbol (maker/taker rates); distinct from `CCXT.Fee` (fee on a single trade) - Post-review fixes: `OrderBook.best_bid/1`+`best_ask/1` return price (number), not `[price, amount]` pair (matching bak contract). `LeverageTier.tier` typed as `integer()`. `networks` typespec on Currency/DepositWithdrawFee removed `| nil` (default `%{}`). Added 3 missing structs found by Codex: ADLRank, BorrowInterest, FundingRateHistory. ### Phase 4b: Integration Tests β€” Tasks Added to Roadmap **Planned** | Tasks 37-42 Added 6 new tasks for macro-generated integration tests covering unified methods, raw endpoints, signing verification, and Phase 5 struct validators. Design informed by `../ccxt_client_bak/` test infrastructure patterns (credential registry, tag hierarchy, response validators). Key decisions: ETS-backed `CCXT.Testnet` registry for credential management, compile-time macro generation from exchange introspection functions, two-stage response detection (raw vs parsed structs) so tests automatically become parser acceptance criteria when Phase 5 lands. ### Task 18: Rate Limiting **Completed** | Phase 4 | [D:4/B:7/U:7 β†’ Eff:1.75] **What was done:** - Built `CCXT.RateLimiter` β€” per-credential weighted rate limiter with sliding window algorithm, ported from `ccxt_client_bak` - Built `CCXT.RateLimiter.State` β€” ETS-backed store for rate limit status from response headers (`:public` table, lock-free reads) - Built `CCXT.RateLimiter.Info` β€” normalized rate limit status struct with `should_wait?/2`, `usage_percent/1`, `wait_time/1` - Built `CCXT.RateLimiter.Headers` β€” response header parser supporting 3 exchange patterns: Binance (`x-mbx-used-weight-1m`), Bybit (`x-bapi-limit`), Standard (`x-ratelimit-*`) - Created `CCXT.Application` supervision tree with RateLimiter + State as children - Integrated rate limiting into `CCXT.HTTP.request/4` and `CCXT.HTTP.signed_request/4` β€” pre-request weight check between circuit breaker and HTTP execution - Wired endpoint weight from `CCXT.Dispatch.call/4` through to HTTP via `:endpoint_weight` opt - Wired response header parsing (`Headers.parse/3` β†’ `State.update/2`) into `execute_request/4` β€” ETS state store populated with real-time rate limit pressure from every response - Added rate limiter defaults to `CCXT.Defaults` (enabled, cleanup interval, max age) - Added `[:ccxt, :rate_limiter, :throttled]` telemetry event with delay and cost measurements - Phase 4 is now complete **Key decisions:** - GenServer with Application supervision (not process-free like CircuitBreaker) β€” rate limiting needs serialized check-and-record + periodic cleanup - Rate key is `{exchange_id, api_key | :public}` β€” per-credential isolation for multi-user scenarios - `rate_limit_ms` is milliseconds between requests (CCXT convention); converted to max requests per period via `trunc(period / rate_limit_ms)` (float-safe; e.g., Bybit 20ms β†’ 3000 req/min, Binance 50ms β†’ 1200 req/min) - Monotonic timestamps prevent clock-jump issues - Rate limiter disabled when `rate_limit_ms` is 0 (exchange has no configured limit) - Key eviction after 24h idle prevents memory growth from many API keys ### Task 28: Discoverable + MCP Tools **Completed** | Phase 7 | [D:2/B:7/U:8 β†’ Eff:3.75] **What was done:** - Added Descripex `api()` declarations to all 241 unified methods + `exchange/2` constructor on the `CCXT` module, replacing `@doc false` with structured metadata (params, returns, errors) - Enhanced `CCXT.Unified.method_defs/0` from 3-tuples to 4-tuples with description strings β€” curated descriptions for ~40 key methods, auto-generated for the rest - Wired `Descripex.Discoverable` on `CCXT` β€” progressive disclosure via `describe/0-2` across CCXT, Exchange, Symbol, HTTP modules - Created `CCXT.MCP` module β€” thin wrapper for MCP tool generation via `Descripex.MCP.tools/1` - Added `api()` declaration to `exchange/2` constructor (bang variant `exchange!/2` excluded β€” raising functions are wrong for agent/MCP consumption) **Key decisions:** - Direct attribute emission instead of `api()` macro for compile-time loops β€” the macro's `preprocess_schemas/1` can't iterate over compile-time variables (they're AST at macro expansion time). Calling `Descripex.generate_doc/2` and `Descripex.build_hints/2` directly is equivalent and avoids this limitation - Bang variants keep `@doc false` β€” they're convenience wrappers, not primary API surface - No `schema:` on params yet β€” JSON Schema integration deferred to Task 21b (Unified Type Schemas) - Exchange param declared with `kind: :value` on every method β€” agents need to know the first arg is always `%CCXT.Exchange{}` **Discovered tasks:** - Task 36: Add `Descripex.emit_api/3` upstream helper for compile-time loop usage ### Task 16 Post-Review Fix: Unified API Bug Fixes (Codex Review) **Completed** | Phase 4 **What was done:** - Fixed `fetchUTAOHLCV` naming mismatch β€” `Macro.underscore("fetchUTAOHLCV")` produces `:fetch_utaohlcv` but `method_defs/0` declares `:fetch_uta_ohlcv`. Wired up the existing `js_to_atom` canonical mapping in `Exchange.build_unified_method_mapping/2` so `method_defs/0` is the authoritative source - Fixed required params being silently overridden by opts β€” `build_params/3` used `Map.put` which let extra opts overwrite positional args (dangerous for trading calls like `create_order`). Changed to `Map.put_new` so positional args always win - Fixed transport opts (`:headers`, `:base_url`) leaking into exchange params β€” expanded `@dispatch_opts` to include HTTP-level opts that `Dispatch.call/4` β†’ `HTTP.request/4` understand - Replaced weak uniqueness test with atom-match canary test that documents known `Macro.underscore` divergences - Added regression test verifying `:fetch_uta_ohlcv` resolves on Kucoin - Added test verifying HTTP-level opts are separated from exchange params **Key decisions:** - `method_defs/0` is the single source of truth for JSβ†’atom mapping, not `Macro.underscore` - Known divergences (where `Macro.underscore` mangles consecutive acronyms) are handled by the `js_to_atom` lookup and documented in the canary test's `known_divergences` set ### Task 16 Post-Review Fix: Unified Dispatch Corrections **Completed** | Phase 4 **What was done:** - Removed overly strict `check_capability` gate from `Unified.call/5` β€” dispatch now uses endpoint mapping as sole gate, matching CCXT JS runtime behavior where method overrides bypass the base class `has` check - Removed dead `fetch_uta_markets_alt` entry β€” `Macro.underscore` maps both `fetchUTAMarkets` and `fetchUtaMarkets` to the same atom, so the `_alt` variant's `__unified_endpoint__` always returned `[]` - Added cross-exchange regression tests using `Exchange.new/2` with real specs (tokocrypto, ascendex) - Added `Macro.underscore` uniqueness test to prevent future naming collisions - Improved `resolve_endpoint` error messages to include exchange ID and capability name - Replaced hardcoded counts in tests with structural assertions derived from `method_defs/0` **Key decisions:** - `Exchange.has?/2` stays as public API for consumer introspection but is no longer in the dispatch path - Endpoint mapping presence is the authority on whether a method is callable β€” matches CCXT's actual runtime behavior for exchanges with method overrides - Investigated via ccxt_extract: `has` is static metadata from `describe()`, not runtime introspection. Exchanges with `has=false` + method overrides work in CCXT JS because overrides bypass the base class gate. ### Task 16: Unified API Functions **Completed** | Phase 4 | [D:5/B:10/U:10 β†’ Eff:2.0] **What was done:** - Built `CCXT.Unified` β€” internal dispatch helper with data-driven method definitions list covering unified methods across 110 exchanges - Generated functions on `CCXT` module (regular + bang variants) via compile-time `for` comprehension over `CCXT.Unified.method_defs/0` - Each method: resolves exchange module β†’ selects endpoint β†’ dispatches through `Dispatch.call/4` - Added `:module` field to `%CCXT.Exchange{}` struct β€” populated at construction via `Registry.module_for/1` for O(1) module resolution - Converted `CCXT.Error` from `defstruct` to `defexception` with `message/1` callback β€” enables clean bang variant raising - Method definitions categorized by 14 signature patterns covering all CCXT JS unified method signatures - Both `fetchUTAMarkets` and `fetchUtaMarkets` map to `:fetch_uta_markets` via `Macro.underscore/1` β€” single entry suffices **Key decisions:** - Data-driven generation over handwritten functions β€” single source of truth, adding a method = adding one tuple - All optional CCXT params (since, limit, price, symbol when optional, etc.) go in `opts` keyword list β€” idiomatic Elixir - Dispatch-level opts (`:endpoint_index`, `:timeout`, `:plug`) separated automatically, never sent to exchange - First-in-list endpoint selection with `:endpoint_index` override β€” simple, correct for 95% of cases, refinable later - Raw `{:ok, map()}` responses β€” Phase 5 (parsers) adds struct creation - Descripex `api()` macro integration deferred to Task 28 β€” needs testing with compile-time generation - `@doc false` on generated functions β€” unified API docs will be added via Descripex in Task 28/29 ### Task 16b: Compile-Time Exchange Generation **Completed** | Phase 2 | [D:3/B:9/U:9 β†’ Eff:3.0] **What was done:** - Built `CCXT.Exchanges` β€” reads spec manifest at compile time and generates all 110 exchange modules via `Module.create/3`, eliminating individual stub files - Refactored `CCXT.Exchange.__generate__/1` to extract `build_module_body/2` and `prepare_generate_data/1` as shared functions β€” both the `use CCXT.Exchange` macro path and `Module.create` path use the same pipeline - Extracted introspection functions into `build_introspection_ast/1` helper for cleaner separation - Added `.doctor.exs` to ignore macro DSL modules (`CCXT.Exchange`, `CCXT.Exchanges`) that Doctor can't meaningfully check - Display names use spec's `name` field (e.g., "BTC Markets", "Crypto.com") instead of `Macro.camelize(id)` - Updated sobelow skips for shifted line numbers **Key decisions:** - `Module.create/3` over individual stub files β€” new exchange = new spec file, literally zero Elixir code - `build_module_body/2` accepts `:moduledoc` option β€” `CCXT.Exchanges` provides per-exchange docs while the macro path leaves it to the caller - Spec `name` field for display over `Macro.camelize` β€” preserves brand-specific formatting (spaces, dots, casing) ### Task 15: Unified Method Mapping **Completed** | Phase 4 | [D:5/B:8/U:9 β†’ Eff:1.7] **What was done:** - Built `CCXT.UnifiedMethod` β€” compile-time mapping of unified method names (e.g., `fetchTicker`) to raw endpoint configs (e.g., `%{name: :public_get_v5_market_tickers, ...}`) - Forward-conversion approach: compute JS interface names from endpoint configs via `endpoint_config_to_js_name/3`, build lookup, resolve unified method references through it - Wired into generator macro via `prepare_generate_data/1` β€” stores `@ccxt_unified_mapping` as module attribute - Generated `__unified_endpoints__/0` (full mapping) and `__unified_endpoint__/1` (per-method lookup) introspection functions on each exchange module - Required ccxt_extract enhancement (separate project): new `UnifiedEndpoints` OXC extractor walks exchange `.ts` ASTs to find which `this.publicGet*()` / `this.privatePost*()` methods each unified method calls - Re-extracted 110 specs with new `structure.unified_endpoints` field **Key decisions:** - Forward-conversion over `Macro.underscore/1` β€” `build_function_name/3` preserves section casing (`:dapiPublic_get_ticker_24hr`) while `Macro.underscore` lowercases it (`:dapi_public_get_ticker24hr`). Forward conversion from endpoint configs to JS names avoids this mismatch - `upcase_first/1` over `String.capitalize/1` β€” capitalize lowercases remaining chars, breaking camelCase paths like `algoOrder` β†’ `Algoorder`. Only uppercasing the first char preserves original casing - Multiple endpoints per method is correct β€” `fetchOHLCV` on Bybit calls 4 different kline endpoints depending on price type. Stored as list for Task 16 to select at runtime - Graceful degradation β€” specs without `unified_endpoints` (old specs, child exchanges with empty API) produce empty mapping **Bugfix (code review run 1):** - Fixed `sections_to_js_prefix/1` using `String.capitalize/1` instead of `upcase_first/1` β€” lowercased the tail of camelCase section names like `accountCategory` β†’ `Accountcategory`, breaking 23 unified method lookups on Ascendex. Added regression test. Found by Codex rescue agent. **Bugfix (code review run 2):** - Added `@` to `@path_separator_pattern` in `endpoint_config_to_js_name/3` β€” BigOne's `depth@{symbol}/snapshot` path produced `Depth@SymbolSnapshot` instead of `DepthSymbolSnapshot`, causing `fetchOrderBook`'s contract endpoint to silently miss the mapping. Added regression test. Found by Codex rescue agent. **Discovered bugs:** - Task 35: `build_endpoint_configs/1` treats `"options"` as HTTP OPTIONS method instead of Gate.io's API section name β€” 28 options endpoints silently skipped ### Task 17b: Derivative Symbol Conversion **Completed** | Phase 4 | Extension of Task 17 **What was done:** - Added `to_exchange_id/2` and `from_exchange_id/3` to `CCXT.Symbol` β€” pattern-based conversion for all market types (spot, swap, future, option) - Added `classify_pattern/2` β€” maps spec JSON fields to pattern atoms at Exchange construction time - Added `symbol_patterns` field to `%CCXT.Exchange{}` struct, populated from `runtime.symbol_patterns` in specs via `build_symbol_patterns/1` - 4 exchange pattern families: Binance (no-sep + underscore dates), Deribit (dash + DDMMMYY, base-only for USD), OKX (dash + suffix), Bybit (concat + DDMMMYY + settle suffix) - Pattern atoms for dispatch: 12 spot, 4 swap, 4 future, 4 option patterns - Forward/reverse currency alias support (unified↔exchange codes) using `common_currencies` - Bang variants `to_exchange_id!/2` and `from_exchange_id!/3` with `CCXT.Symbol.Error` - Graceful fallback to spot `denormalize/normalize` when pattern config missing **Bug fixes (code review + Codex):** - Fixed `classify_pattern/2` rejecting `id_structure: "opaque"` entries (dYdX swap IDs like "MOG-USD" are not mechanically derivable from unified symbols) - Fixed `reverse_future` missing `:yyyymmdd` date format handler β€” was silently returning raw exchange ID - Fixed `normalize/3` ignoring `:quote_currencies` option β€” `find_and_split` was hardcoded to default list - Aligned settle heuristic to `["USD", "USDC"]` across `reverse_future_yymmdd`, `reverse_future_yyyymmdd`, and `reverse_swap` **Key decisions:** - Classify once at Exchange construction, dispatch by atom at call time β€” mirrors signing pattern architecture - Deribit "base-only" heuristic: omit quote currency when separator is "-" and quote is "USD" (applies to swaps, futures, options) - Binance future separator split: `"_"` separates pair from date, but pair itself has no separator (`BTCUSDT_260327`) - Tests use hardcoded pattern configs β€” no dependency on spec extraction having live market data - `symbol_patterns` from specs require `runtime.symbol_patterns` which needs `loadMarkets()` live API calls; only 1/110 specs currently have this data (dYdX, opaque). Most exchanges will get patterns when `ccxt_extract` is enhanced to call `loadMarkets()` per exchange ### Task 17: Symbol Normalization **Completed** | Phase 4 | [D:3/B:8/U:9 β†’ Eff:2.83] **What was done:** - Built `CCXT.Symbol` β€” bidirectional symbol normalization between unified (`"BTC/USDT"`) and exchange-native formats (`"BTCUSDT"`, `"BTC-USD"`, `"btcusd"`) - Ported from `ccxt_client_bak/lib/ccxt/symbol.ex`, adapted to current project architecture - Format-based conversion: `normalize/3` (exchange β†’ unified) and `denormalize/2` (unified β†’ exchange) with configurable separator and case - Symbol parsing: `parse/1`, `parse!/1`, `parse_extended/1` for spot, swap, future, and option symbols - Symbol building: `build/2-3` for constructing unified symbols from components - Currency aliases: `apply_alias/2` and `reverse_aliases/1` for `commonCurrencies` mapping (XBTβ†’BTC, etc.) - Prefix handling: `strip_prefix/1` for Kraken X/Z currency prefixes and KrakenFutures contract prefixes (PI_, PF_, FI_) - WebSocket format conversion: `denormalize_ws/2` for 4 WS channel formats - Date conversion: `convert_date/3` between `:yymmdd`, `:ddmmmyy`, `:yyyymmdd` formats (derivative expiry dates) - Market type detection: `detect_market_type/1` from parsed extended symbol components - Added `common_currencies` field to `%CCXT.Exchange{}`, populated from spec `commonCurrencies` data - Built `CCXT.Symbol.Error` exception with constructors for 5 error types **Key decisions:** - Format as explicit parameter (not stored on Exchange) β€” keeps Symbol module pure, composable, and testable in isolation - Longest-match-first quote currency detection β€” sorted by length descending to match "USDT" before "USD" in no-separator symbols - Advanced pattern-based conversion (to_exchange_id/from_exchange_id for derivatives) not yet wired β€” requires `symbol_patterns` data not in current specs (needs `ccxt_extract` enhancement or runtime market loading) - `@type` moved after `defstruct` in Exchange to fix Elixir 1.20-rc.4 compilation (validates struct fields in typespec) ### Task 14: Sign Dispatch + Integration **Completed** | Phase 3 | [D:3/B:8/U:9 β†’ Eff:2.83] **What was done:** - Wired `CCXT.Signing.sign/4` into `CCXT.Dispatch.call/4` for private endpoint authentication - Private endpoint detection via case-insensitive `"private"` substring match in sections β€” handles standard `"private"` and 18 non-standard variants (`"fapiPrivate"`, `"privateTrading"`, `"v2Private"`, etc.) across 15 exchanges - Credential and signing pattern validation returns `{:error, :authentication_error}` with actionable messages - Refactored `CCXT.HTTP` to extract shared execution pipeline (`execute_request/4`) from `do_request/6` - Added `CCXT.HTTP.signed_request/4` for pre-signed requests β€” uses same telemetry, circuit breaker, and response handling as unsigned requests - Updated existing tests that used `sections: ["private"]` without credentials to use `["public"]` **Key decisions:** - Signing integrates as a step within `Dispatch.call/4`, not a separate argument β€” consistent with `call/4` not `/5` design decision - `HTTP.signed_request/4` is a separate function (not flags on `request/4`) β€” explicit contract, no behavior-changing options - `signed.body` passes through without re-encoding β€” signing patterns already handle JSON serialization - `signed.url` is relative; base URL prepended by HTTP, not by Signing ### Task 13: Pattern Classification **Completed** | Phase 3 | [D:5/B:8/U:9 β†’ Eff:1.7] **What was done:** - Built `CCXT.Signing.Classifier` β€” two-tier heuristic classification of signing patterns from spec `structure.sign_method` AST data - Tier 1: Header-name matching (e.g., `X-BAPI-*` β†’ `:hmac_sha256_headers`, `OK-ACCESS-*` β†’ `:hmac_sha256_iso_passphrase`) β€” most reliable signal - Tier 2: Hash algorithm fallback (sha384/sha512/sha256 identifiers + structural cues like passphrase literals) - Parent inheritance for 12 child exchanges without `sign_method` (e.g., binancecoinm β†’ binance, gateio β†’ gate) - Config extraction: header names (api_key, timestamp, signature, passphrase) extracted from AST string literals - Added `signing_pattern` and `signing_config` fields to `%CCXT.Exchange{}` struct, populated at both compile time (generator macro) and runtime (`new/2`) - Generated `__signing__/0` introspection function on all exchange modules - `mix ccxt.classify_signing` Mix task for reporting classification across all 110 exchanges **Key decisions:** - Data-driven rules list instead of large `cond` β€” avoids cyclomatic complexity while maintaining ordered evaluation - Header-name heuristics as primary signal over hash algorithm β€” many patterns share SHA-256, but header names are unique per exchange family - Passphrase detection checks both AST identifiers AND header literals β€” some exchanges (apex, coinbase) embed "passphrase" only in header names, not as JS variable names - 91 exchanges auto-classified into 8 specific patterns, 19 classified as `:custom` (exotic auth: ed25519, StarkNet, basic auth, etc.) - Classification runs at compile time in generator macro AND at runtime in `Exchange.new/2` β€” both paths use the same `Classifier.classify/1` ### Task 11: Signing Behaviour + Task 12: Pattern Modules **Completed** | Phase 3 | [D:2/B:7/U:8 β†’ Eff:3.75] + [D:3/B:9/U:9 β†’ Eff:3.0] **What was done:** - Built `CCXT.Signing` β€” behaviour definition, function-head dispatcher routing 9 pattern atoms to modules, shared crypto helpers (HMAC-SHA256/384/512, SHA256/512, hex/base64 encoding, sorted URL encoding) - Ported 9 signing pattern modules from `ccxt_client_bak`: HmacSha256Query (Binance, ~40 exchanges), HmacSha256Headers (Bybit, ~30), HmacSha256Iso (OKX, ~10), HmacSha256Kucoin (~3), HmacSha512Nonce (Kraken, ~3), HmacSha512Gate (Gate.io), HmacSha384Payload (Bitfinex/Gemini, two variants), Deribit (custom Authorization header), Custom (escape hatch) - Each module implements `CCXT.Signing.Behaviour` callback `sign/3` - Custom pattern includes `validate_module/1` for runtime module validation - Switched Credo to GitHub `release/1.7` branch to fix `SpaceAroundOperators` crash with Elixir 1.20-rc.4 sigils **Key decisions:** - `sign/4` dispatcher (not `/5`) β€” pattern atom is first arg, consistent with ROADMAP design - Shared helpers are `@doc false` β€” internal API for pattern modules, not public - `urlencode/1` sorts alphabetically β€” Binance and others require deterministic param ordering for signature verification - Kraken's secret is base64-encoded at the exchange β€” `HmacSha512Nonce` decodes before HMAC - KuCoin v2 API keys HMAC-sign the passphrase itself β€” unique among all patterns - HmacSha384Payload supports two variants (`:bitfinex` and `:gemini`) via config - No dispatch integration yet (Task 14) β€” signing modules are ready, wiring into `Dispatch.call/4` is a separate task ### Task 7: Endpoint Generation + Task 8: Dispatch Module **Completed** | Phase 2 | [D:6/B:9/U:10 -> Eff:1.58] each **What was done:** - Built `CCXT.Dispatch` β€” shared stateless dispatcher for all generated endpoint functions - Path interpolation replaces `{param}` templates in endpoint paths (41/110 exchanges use these) - Base URL resolution navigates `exchange.base_urls` using endpoint sections β€” handles 4 URL patterns: flat (Bybit), flat-distinct (Binance), nested (Gate/MEXC), and early-stop (BingX) - Generated one function per REST endpoint in each exchange module via `build_endpoint_functions/1` - Functions support arity 1/2/3 with defaults: `(exchange)`, `(exchange, params)`, `(exchange, params, opts)` - Each function embeds its endpoint config as a compile-time literal β€” zero runtime lookup - Pattern match on `%CCXT.Exchange{}` catches type errors at call site - Phase 2 macro/dispatch capability is now complete β€” exchange modules can be generated with callable endpoint functions (production modules not yet wired) **Key decisions:** - Only `{curly}` path style needed β€” no `:colon` style found across 110 exchange specs - Missing path params preserved as-is (exchange returns descriptive error); explicit nil values raise `ArgumentError` (fail-fast prevents mutated paths hitting wrong endpoints) - `@doc false` on all generated functions β€” unified API (Phase 4) is the public interface - `Dispatch.call/4` not `/5` β€” signing integrates in Phase 3 as a step within call, not a separate arg - `build_endpoint_functions/1` extracted from macro to keep `__generate__/1` complexity under Credo threshold ### Task 10: Exchange Registry **Completed** | Phase 2 | [D:3/B:7/U:8 -> Eff:2.5] **What was done:** - Built `CCXT.Registry` β€” compile-time lookup mapping exchange IDs to module names - Function-head dispatch for O(1) lookup (`:bybit` β†’ `CCXT.Bybit`) - Supports both string and atom IDs via atomβ†’string canonicalization - `loaded?/1` distinguishes "registered" from "module compiled" via `Code.ensure_loaded?/1` - Tracks `@external_resource` on manifest for automatic recompilation when specs change - Provides lookup infrastructure for exchange module resolution (integration into `CCXT.exchange/2` deferred to Task 16) **Key decisions:** - Function heads over `Map.get/2` β€” Erlang optimizes multi-clause dispatch, and per-exchange clauses are extensible - Module naming via `Macro.camelize/1` β€” verified correct for all IDs including edge cases (bit2c, p2b) - Atomβ†’string conversion in `lookup/1` β€” single canonical path through string-matched heads ### Task 6: Generator Macro + Task 9: Introspection Functions **Completed** | Phase 2 | [D:7/B:10/U:10 -> Eff:1.43] + [D:2/B:6/U:7 -> Eff:3.25] **What was done:** - Built two-stage generator macro in `CCXT.Exchange` (`__using__/1` β†’ `__generate__/1`) - `use CCXT.Exchange, spec: "bybit"` loads spec at compile time, stores lean spec as `@ccxt_spec` - Pre-computes flat `@ccxt_endpoint_configs` from nested API tree (ready for Task 7) - Recursive traversal handles all 3 spec patterns: standard 2-level, deep nesting (BingX/Gate/MEXC), array endpoints (Alpaca/Bitflyer) - Normalizes complex weight objects (Binance's `{byLimit: [...], cost: N}`) to numbers - Tracks spec files via `@external_resource` for automatic recompilation - Wires `use Descripex` into every generated module β€” `api()` macro ready for Task 7 - Generates 5 introspection functions: `__id__/0`, `__name__/0`, `__spec__/0`, `__endpoints__/0`, `__features__/0` - Added `elixirc_paths` to mix.exs for `test/support/` modules - Test exchange modules: Bybit (standard), Binance (mixed sections + complex weights), BingX (deep nesting) **Key decisions:** - Macro lives in `CCXT.Exchange` alongside struct β€” matches Phoenix/Ecto patterns - Descripex wired from day one (Task 5b intent) β€” no Phase 7 retrofitting needed - Two layers of introspection: descripex for function-level (`__api__/0`), custom for exchange-level metadata - `build_endpoint_configs/1` is a regular function (not macro) β€” simpler, directly testable - `:sections` list replaces `:visibility` string β€” captures full nesting path (e.g., `["spot", "v1", "private"]`) - HTTP method names (`get`, `post`, etc.) used as sentinel keys to detect terminal nodes in recursive traversal - `String.to_atom/1` at compile time is safe β€” bounded by spec, max ~800 per exchange ### Code Review Fixes (Tasks 4 & 5) **What was done:** - Fixed nested `base_urls` (Gate, MEXC) breaking `default_base_url/1` β€” now traverses into nested maps - Separated `exceptions.broad` from `exceptions.exact` β€” broad patterns now substring-matched against error messages instead of dead code - Removed `content-type: application/json` from GET/HEAD/DELETE requests (no body) - Added defensive `error_code?/1` guard against non-coercible code values - Replaced 13 identical `build_typed_error/4` function heads with `@known_error_types` MapSet + `apply/3` - Added `broad_error_patterns` field to `%Exchange{}` struct - Clarified ROADMAP Task 4 rate limiting deferral - Added tests: POST body verification, CB disabled path, CB telemetry emission, Defaults config override, broad error pattern matching ### Task 5: Error Types **Completed** | Phase 1 | [D:2/B:6/U:7 -> Eff:3.25] **What was done:** - Built `CCXT.Error` struct with 17 error type atoms covering all 34 CCXT spec exception classes - Factory functions for each type with consistent `build/3` helper - Inline recoverability classification β€” `recoverable?/1`, `recoverable_types/0` - `from_spec_class/1` bridges spec `"__function:AuthenticationError"` strings to atoms - `spec_class_mapping/0` exposes the full mapping for introspection **Key decisions:** - 34 spec classes collapsed to 17 atoms (many-to-one: e.g., InvalidNonce β†’ :authentication_error) - No Hints submodule β€” `hints` field populated by callers directly (simpler) - Exchange field is `String.t()` (not atom) matching new project's string exchange IDs ### Task 4: HTTP Client **Completed** | Phase 1 | [D:4/B:7/U:9 -> Eff:2.0] **What was done:** - Built `CCXT.HTTP` β€” Req wrapper with full error handling pipeline - Built `CCXT.CircuitBreaker` β€” per-exchange circuit breakers using `:fuse` Erlang library - Built `CCXT.Telemetry` β€” 6 telemetry events (3 request + 3 circuit breaker) - Built `CCXT.Defaults` β€” centralized config defaults (recv_window, timeout, retry policy) - Updated `CCXT.Exchange` β€” added `error_codes` and `http_exceptions` fields pre-processed from spec - Manual query encoding (signing needs raw params before URL encoding) - Safe retry policy (GET/HEAD only, never POST β€” prevents duplicate orders) - Body-level error detection (non-zero code fields in HTTP 200 responses) - HTML response detection (geo-blocks, Cloudflare return HTML instead of JSON) - HTTP status normalization (429 β†’ rate_limit, 401/403 β†’ auth_error) - Base client caching via `:persistent_term` - Added deps: `req`, `fuse`, `telemetry`, `plug` (test only) **Key decisions:** - Adapted from backup's proven 963-line implementation, not written from scratch - No signing yet (Phase 3) β€” private endpoints pass through unsigned - No rate limiting yet (Task 18) β€” circuit breaker provides protection - Circuit breaker uses lazy fuse installation β€” no application supervision needed - `should_melt?/1` only trips on 500+ and transport errors, not 429/4xx ### Task 3: Exchange Constructor **Completed** | Phase 1 | [D:3/B:8/U:8 -> Eff:2.67] **What was done:** - Built `CCXT.Exchange` struct β€” config object holding everything needed for API calls - Constructor `new/2` / `new!/2` loads spec, resolves URLs, builds credentials - URL resolution handles 3 patterns: nested with hostname template (Gate, MEXC), flat with hostname template (Bybit, OKX), absolute with null hostname (Bit2c) - Sandbox mode switches to testnet URL set with `{hostname}` interpolation - Credentials optional (public endpoints), built from opts or passed as pre-built struct - `has?/2` helper for checking exchange capabilities - `CCXT.exchange/2` / `CCXT.exchange!/2` convenience functions on top-level module - Lean spec storage β€” strips `api` endpoints (large) from stored describe data **Key decisions:** - String keys in `has`/`required_credentials` β€” matches spec format, avoids unsafe atom conversion - No process β€” pure data struct; rate limiting deferred to Task 4/18 - Sandbox resolved from explicit opt > credentials.sandbox > false - `spec` field stores lean `runtime.describe` minus `api` β€” full spec via generated modules (Phase 2) ### Task 2: Core Types **Completed** | Phase 1 | [D:3/B:8/U:9 -> Eff:2.83] **What was done:** - Defined 8 unified data structs in flat `CCXT.*` namespace: Fee, Credentials, Ticker, Trade, Order, Balance, OHLCV, Market - `CCXT.Credentials` has `new/1` / `new!/1` with validation β€” explicit credentials, never from ENV - `CCXT.Order` has helper predicates: `open?/1`, `closed?/1`, `canceled?/1`, `filled?/1` - `CCXT.Balance` has `get/2` (per-currency lookup) and `currencies/1` helpers - `CCXT.OHLCV` has `from_list/1` for `[timestamp, o, h, l, c, v]` arrays - All types except `CCXT.OHLCV` and `CCXT.Fee` include `info` field for raw exchange responses **Key decisions:** - Exchange config struct deferred to Task 3 β€” avoids namespace collision with `CCXT.Exchange` generator macro (Phase 2) - No `from_map/1` on types β€” parser layer (Phase 5) handles construction via field mappings - Status/side/type fields are strings, not atoms β€” normalization belongs in parser layer - `CCXT.Fee` extracted as separate module β€” referenced by both Trade and Order ### Task 1: Spec Loader **Completed** | Phase 1 | [D:3/B:9/U:10 -> Eff:3.17] **What was done:** - Built `CCXT.Spec` module β€” compile-time JSON spec loader for 110 exchange specs - Public API: `load!/1`, `load_manifest!/0`, `exchanges/0`, `spec_path/1`, `manifest_path/0` - Dead weight stripping removes `structure.interface_signatures` and `structure.methods` (~110KB per large exchange) - String keys throughout β€” no atom conversion for safety with arbitrary exchange field names - Added `jason` as explicit runtime dependency - Updated CCXT module with proper project moduledoc - Re-extracted specs to CCXT 4.5.46 **Key decisions:** - No caching in Spec module β€” generator macro loads once per exchange at compile time; runtime access via introspection functions - `@external_resource` tracking deferred to generator macro (Phase 2), not Spec - `spec_dir/0` resolves via `:code.priv_dir/1` with fallback for compile-time - Exchange ID validation (`^[a-z0-9_]+$`) prevents path traversal in `load!/1` --- ## Phase 0: Project Setup ### Task 0: Initial Setup **Completed** | 2026-04-02 **What was done:** - Initialized Elixir project with dev tooling (Credo, Dialyxir, ExDoc, Doctor, Styler, Tidewave) - Imported 110 exchange specs from ccxt_extractor (CCXT 4.5.45) - All specs validated against schema