Testing, fakes and isolation

Run the conformance suite

defmodule DpExchange.YourVenue.ContractTest do
  use DpExchange.Core.AdapterContract,
    venue: DpExchange.YourVenue,
    symbol_format: DpExchange.YourVenue.SymbolFormat,
    sample_pairs: ~w(BTC-USD ETH-USD),
    credentials: %{api_key: "test", api_secret: "test"}
end

Fifteen assertion groups, listed in DpExchange.Core.AdapterContract.assertions/0. It ships inside the Hex tarball, so you get it as a dependency and run it against yourself. Prose in six CLAUDE.md files drifts; a suite running in five CI pipelines cannot.

The four verification tiers, and only two ever run unattended

1. In-process fakes — every CI run. The default.
2. Live public endpoints — per venue, by hand, during its extraction. Never on a schedule: a venue that sees a package polling it on a timer will rate-limit or block.
3. Authenticated, read-only — needs credentials a public repo must never hold. Runs where the keys already are, which is a consumer's machine, not this repo's CI.
4. Money-moving — never a test. It is answered in production by a consumer trading live, and that is what moves an endpoint to :proven.

What a green CI run does not prove: that the venue behaves as your fake does. Tier 1 proves your code is self-consistent. Nothing below tier 3 proves it is right.

No mocking

Use real fixtures and real processes. A mock asserts that your code called what you expected, which is a restatement of the code rather than a test of it.

Your fake is a real implementation of the facade that happens to answer from memory. It runs the same conformance suite as the real adapter.

Two rules for your fake, from thirteen real bug reports

Eleven of thirteen bugs found in a comparable in-process fake were the fake diverging from the real client. Six were loud — the fake rejecting what the real thing accepts — which cost time and nothing else. Three were silent, and those are the ones to design against:

1. Less capable is allowed. Differently capable is not. Where the fake cannot answer, it returns an error. It never returns an empty success for something unsupported — the original answered {:ok, []} to an unsupported query and silently dropped clauses it could not parse, so callers got plausible wrong answers rather than failures.
2. Never rewrite a value the caller supplied. The original discarded the caller's timestamp and substituted the current clock, landing points written 900 seconds apart microseconds apart.

Every divergence found becomes a new assertion in the shared suite. A gap fixed only in your fake is a gap the next venue reintroduces. That ratchet is why a comparable suite grew to over a thousand lines and why none of those thirteen bugs ever reached an external user.

Isolate the fake per process — this is a hard requirement

Your consumer runs async: true. Application.put_env/3 is node-wide. Any seam a consumer's tests need to vary must be resolvable per process, or your package is unusable in their suite.

# In a test, for its own process tree only:
DpExchange.Core.Config.put_override(:your_venue_fake, MyFake)

# In your package, at call time — never compile time:
DpExchange.Core.Config.get(:dp_exchange_core, :your_venue_fake, RealClient)

Resolution order: this process's dictionary, then the $callers ancestor chain, then application env.

The $callers walk is the step people omit, and omitting it works in simple tests and fails in exactly the concurrent ones the seam exists for. ExUnit propagates $callers to spawned Tasks.

Crossing a process boundary needs one more step

A GenServer runs in its own process and is not in the caller's $callers chain, so it will not find the override at all. Snapshot in the caller and put it in the message:

snapshot = DpExchange.Core.Config.snapshot([:your_venue_fake])
GenServer.call(server, {:fetch, symbol, snapshot})

Resolving inside the server is too late, and it fails in the direction that looks like it works: production is unaffected, so only your consumer's async suite breaks.

Why this is non-negotiable

Seven tests once set a global rate-limiting flag for their duration, and for that duration every other async test on the node was metered against a one-request bucket. An unrelated WebSocket test came back {:error, {:rate_limited, 1}} while asserting on connection errors. The failure was silent, intermittent and seed-dependent.

A consumer will want venue A faked while venue B is real, and two tests will want the same fake to behave differently — one returning 429, one succeeding. Global config cannot express that at all.

Making a fake fail on demand, and skipping its credential check

Every venue's Fake is wired to DpExchange.Core.FakeInjection — a deterministic seam for exercising your own retry/circuit-breaker/alerting code against a Venue implementation, and for testing pure dispatch/decode logic without constructing valid-looking credentials for every call.

# Make the next call fail once, then resume normal fake behaviour:
DpExchange.Core.FakeInjection.queue_failures(:coinbase, [{:error, :timeout}])
DpExchange.Coinbase.Fake.get_price("BTC-USD", credentials: creds)  #=> {:error, :timeout}
DpExchange.Coinbase.Fake.get_price("BTC-USD", credentials: creds)  #=> {:ok, %Quote{}}

# Make every call for one symbol fail, indefinitely — every other symbol is untouched:
DpExchange.Core.FakeInjection.fail_always(:coinbase, "ETH-USD", {:refused, :not_listed})

# Skip a venue-faithful credential refusal, for a wiring-only test. Robinhood's `Fake`
# gates every call on `credentials:` (the real venue signs everything); this bypasses
# that refusal without changing it for anyone who doesn't opt in:
DpExchange.Core.FakeInjection.bypass_credentials(:robinhood)
DpExchange.Robinhood.Fake.get_price("BTC-USD", [])  #=> {:ok, %Quote{}}, no credentials needed

Not every venue's Fake gates credentials centrally, and this does not add a check that was never there. Coinbase's Fake, for instance, has no single credential refusal to bypass — most of its functions never inspected credentials in the first place, and bypass_credentials/1 has nothing to do there. Check the venue package's own Fake moduledoc for whether it applies.

There is no error_rate-shaped knob and there will not be one. Every outcome is queued explicitly and consumed in order — a test that fails 30% of the time on its own schedule is not more useful than one that never fails. queue_failures/2,3 pops one entry per matching call; fail_always/2,3 never pops, for "every call fails" until you call FakeInjection.reset/1.

Per-symbol targeting is real isolation, not a convention to remember. A symbol-specific override can never be satisfied by, or interfere with, a call for a different symbol — the same rule this family applies everywhere a batch could otherwise let one bad member take down the rest.

Built on the same Config process-scoped override machinery as everything else on this page, so it inherits the identical async: true / $callers isolation guarantee — and the identical limitation: injection configured in your test only reaches a Fake function called from your test's own process or a Task it spawns, not from inside a separately-supervised GenServer.

Not every function is wired. A Fake function that takes a list of symbols in one call — a venue's own bulk subscribe, say — is deliberately left out: whole-call injection can pick the outcome one call returns, but it cannot express "this one symbol in the batch fails, the rest succeed." Check the venue package's own Fake moduledoc for which functions are wired.

Tests must be deterministic

Three things worth stating because they have each cost a debugging session here:

• function_exported?/3 returns false for a module that is merely not loaded. Call Code.ensure_loaded!/1 first, or the assertion tests something other than what it says.
• Never assert on node-global counters — :erlang.system_info(:atom_count) and friends move because of other async tests. Assert the local property instead.
• Synchronise on events, not on elapsed time. A Process.sleep/1 before an assertion races whatever it is waiting for. Widening the sleep converts a visible flake into a slow one.

Which tier each capability group can actually be verified at

The surface grew to 88 callbacks. Most of the new ones cannot be verified above tier 1 here, and saying which is which is the point of this table — a group that CI cannot reach is a group whose correctness rests on the reference documentation and a consumer's production use, and a package author should know that before shipping it.

Gemini's demo environment is the one place this changes. Order placement, cancellation and balances move from tier 4 to tier 3 there — real venue machinery, fake money — which is why the authenticated endpoints exist in that package at all. See usage-rules/environments.md.

A refresh test spends a real token

Schwab's refresh token is one-time use and every refresh mints a replacement. A test that calls Auth.refresh/2 against the live venue consumes the credential it was given and returns a new one that must be persisted or the account is lost until a person logs in again.

So the refresh path is tested at tier 1 against a fake token endpoint, and the tier-3 version is run by hand, once, by someone holding the credential and ready to write the result down. That is not a gap in the suite — it is the honest cost of an at-most-once operation, and it is why refresh/2 forces retries off rather than trusting a test to remember.

Fakes grow with the surface, and a stale fake is worse than a missing one

Every callback a package implements needs a fake clause. A fake that has not kept up answers {:error, :not_supported} for something the real adapter now serves — which is the false-negative defect, reproduced inside the test suite, and it will make a correct implementation look broken.

Two rules that fall out of that:

• Add the fake clause in the same change as the real one. The conformance suite runs against both, so a missing clause fails immediately; a wrong one does not.
• A fake never returns an empty success for something unsupported. {:ok, []} for a query it could not parse is the exact silent divergence this guide's thirteen bug reports were mostly made of.

Coverage is a floor, not a finish line

The threshold is 90 and CI enforces it. Two things it does not measure, both of which have bitten this family:

• A default-argument head counts as a line. An unused \\ [] head is counted as missed and can drop a package below the threshold while nothing is actually untested. Delete the head rather than writing a test for it.
• Coverage says nothing about whether the fake resembles the venue. Tier 1 proves your code is self-consistent. Nothing below tier 3 proves it is right.