Testing Erlang ADK

This guide defines the release gates and how to interpret them. Run commands from the repository root unless a section changes directory.

Toolchains

• Core: Erlang/OTP 27; the verified and minimum production patch is OTP 27.3.4.14.
• Phoenix: Elixir 1.17 or newer on OTP 27; verified with Elixir 1.19.5.
• Browser assets/tests: Node.js; verified with Node 24.3.0.
• Use the repository's ./rebar3 for core commands.

Test source organization

Erlang tests mirror the production ownership hierarchy under test/; the complete map and placement rules are in TEST_LAYOUT.md. The recursive test root is enabled only in Rebar3's test profile, so EUnit and Common Test discover nested modules while ordinary builds and packages do not compile test helpers. Explicit Common Test commands must use each suite's full path; EUnit module selection remains path-independent.

Complete deterministic Erlang gate

./rebar3 do clean, compile, eunit, ct, dialyzer

The final 2026-07-16 v0.7 run passed:

• 1,176 EUnit tests;
• six deterministic Common Test cases;
• Dialyzer over 210 project files with no warnings; and
• both 1,000-operation concurrency/resource stress scenarios.

Those are historical v0.7 numbers, not v0.8 evidence.

The final 2026-07-17 v0.8 command exited zero and passed:

• 1,414 EUnit tests;
• six deterministic Common Test cases; and
• Dialyzer analysis over 235 source modules with no warnings.

Common Test intentionally skipped 22 paid-provider cases because their opt-in flags were absent from this deterministic command. Those skips are expected and are not included in the six passing deterministic cases.

Some HTTP and protocol fixtures open loopback listeners. A restricted build environment must allow local sockets; otherwise a permission failure is an environment failure, not passing test evidence.

Deterministic line coverage gate

./scripts/coverage.sh

The script resets all previously exported Cover data, performs a clean build, instruments the complete EUnit and Common Test suites, aggregates both exports, and fails below 74% executable-line coverage. The HTML summary and per-module reports are written under _build/test/cover/. Do not run rebar3 cover against exports from an earlier source tree: rebar3 clean does not remove stale Cover data.

./rebar3 cover --verbose only aggregates previously exported .coverdata; it does not execute or instrument tests. Therefore No coverdata found is the expected result on a clean tree or immediately after cover --reset. Use ./scripts/coverage.sh to produce both the EUnit and Common Test exports before rendering the report.

Paid Gemini cases remain explicitly skipped without their opt-in flags, and Phoenix/ExUnit coverage is a separate project concern. The 74% floor therefore measures the deterministic Erlang release contract only; it must be raised as new deterministic behavior becomes covered and must not be weakened to hide a regression. The final 2026-07-16 aggregate is 73.88% across 210 production modules. That is historical v0.7 evidence. The final 2026-07-17 v0.8 coverage script exited zero at 74.17% against the enforced 74% floor, while repeating 1,414 EUnit and six deterministic Common Test passes.

Path to 100% deterministic coverage

Coverage work must protect behavior, not optimize a number. Add tests for supported feature and contract branches first: documented public API preservation, structural failure/cancellation paths, concurrency ownership and cleanup, persistence/recovery, security boundaries, and provider-neutral wire behavior. Never remove, deaden, merge away, or make unreachable a supported feature merely to raise coverage.

Ratchet the merged deterministic Erlang report in reviewed stages:

Each stage merges fresh EUnit and deterministic Common Test line-hit data from the same source revision; percentages from separate reports must never be averaged or added. Phoenix ExUnit/Node results and paid-provider evidence stay separate because they do not instrument the Erlang source report. A module may be excluded only when it is generated code or a behaviour-only declaration with no executable product behavior; every exclusion must be narrow, documented, and release-reviewed. Adapters, error branches, unsupported-input guards, and concurrency code are not exclusion candidates merely because they are difficult to exercise.

A release is accepted only when all deterministic gates pass, aggregate and per-module floors do not regress, the public feature ledger remains intact, and any exclusion list is unchanged or explicitly approved. Once a stage is reached on the release branch, raise the enforced floor and do not lower it to accommodate a later regression. The 100% milestone requires every eligible production line to be reached by behavior-asserting tests; remote credentials, quota, or provider availability are never substitutes for deterministic contract coverage.

README and focused v0.8 gates

./rebar3 eunit --module=readme_examples_test
./rebar3 eunit --module=readme_workflow_examples_test

erlc -Werror -pa _build/default/lib/erlang_adk/ebin -o /tmp \
  examples/readme_weather_tool.erl \
  examples/readme_live_weather_executor.erl \
  examples/readme_stateful_counter_plugin.erl

./rebar3 eunit \
  --module=adk_live_media_test,adk_live_gemini_codec_test,adk_live_gun_transport_test,adk_live_public_api_test,adk_live_session_test,adk_live_tool_execution_test,adk_live_observability_test,adk_live_voice_protocol_test,adk_live_voice_bridge_test,adk_plugin_pipeline_test,adk_plugin_runner_integration_test,adk_plugin_builtin_test,adk_plugin_stateful_test,adk_trace_context_test,adk_observability_v2_test,adk_observability_runner_test,adk_otlp_json_test,adk_otlp_http_json_exporter_test,adk_eval_criteria_test,adk_eval_v2_test,adk_eval_llm_judge_test,adk_eval_dev_view_test,adk_dev_v07_http_test,adk_cli_test

./rebar3 eunit \
  --module=adk_provider_credential_test,adk_provider_profile_test,adk_provider_profile_snapshot_test,adk_provider_registry_test,adk_provider_registry_live_test,adk_provider_request_options_test,adk_provider_capabilities_test,adk_model_http_headers_test,adk_model_gun_transport_test,adk_model_sse_decoder_test,adk_llm_openai_test,adk_openai_responses_content_test,adk_openai_responses_codec_test,adk_openai_responses_stream_test,adk_llm_anthropic_test,adk_llm_anthropic_content_test,adk_llm_anthropic_request_test,adk_llm_anthropic_stream_test,adk_llm_compatible_test,adk_llm_compatible_content_test,adk_llm_compatible_request_test,adk_llm_compatible_stream_test,adk_live_openai_codec_test,adk_live_openai_gun_transport_test,adk_live_session_multi_frame_test,adk_live_session_profile_test,adk_live_voice_protocol_test,adk_live_voice_bridge_test,adk_llm_test,adk_cli_test

./rebar3 ct --suite test/runtime/invocations/adk_concurrency_stress_SUITE.erl
./rebar3 ct --suite test/integrations/stress/adk_v05_stress_SUITE.erl

The historical v0.7 evidence is 29 README tests, four workflow tests, warning-as-error compilation/runtime smoke for all three example modules, and 193 focused Live/plugin/observability/evaluation/developer tests. The stress suites cover 1,000 correlated stable runs, 1,000 isolated artifact/memory writes, and 128 cache acquisitions collapsing to four exact-scope provider lifecycles. The stable-run stress assertions correlate every response with its exact session and invocation, require unique run IDs, verify supervisor cleanup, and leave the test-process mailbox stable after each bounded concurrent batch.

The second focused EUnit command isolates the 0.8 additions. The final 2026-07-17 focused provider/profile/Realtime run passed 244/244. It remains diagnostic; the clean complete gate is authoritative. A deterministic HTTP fixture or WebSocket state-machine test does not prove that a paid remote provider accepted the request.

The final README gates passed 30 README and four workflow tests (34/34), and all three example modules compiled with warnings as errors. The v0.7 totals in the preceding paragraph remain historical rather than being silently updated.

After the final audit repairs, the seven-module targeted regression set passed 67/67. It covers multi-frame priority ordering, Anthropic max_tokens >= 1 validation, and synchronous/streaming Gun header and trailer limits.

Every current README recipe and sanity command is mapped to its prerequisites and validation in README_EXAMPLE_COVERAGE.md. The focused suites directly exercise deterministic core, workflow, Live, plugin, evaluation, retry, memory, and artifact examples. Features that need dedicated startup state or transport fixtures—such as HITL, Mnesia, authenticated developer startup, the project-specific HTTP endpoint, and Gemini wire behavior—are asserted by their owning test modules; a focused README smoke may check that those modules remain present without replacing their complete tests.

CLI and package smoke gate

./rebar3 xref
./rebar3 escriptize
_build/default/bin/adk doctor
_build/default/bin/adk config validate examples/agent.json
./rebar3 ex_doc
./rebar3 hex build
./scripts/verify_hex_package.sh

For v0.8, adk doctor must report application version 0.8.0, OTP 27, the REST default gemini-3.1-flash-lite, required dependencies, and whether a Gemini, OpenAI, or Anthropic key is configured without exposing any key. The configuration validator must accept configured binary profiles through their safe aliases without projecting credential sources or values. The checked agent configuration must validate with the same REST model. Xref checks undefined and deprecated calls/functions without treating a library's public exports as unused errors. ExDoc must be warning-free. The non-publishing Hex build and artifact verifier must prove required contents, excluded caches/secrets, and a clean compile from the extracted package.

On 2026-07-17, xref, escript assembly, adk doctor reporting 0.8.0, checked configuration validation, ExDoc, the 0.8.0 Hex build, and compilation from the verified extracted package all passed.

Paid Gemini REST gate

Export the key and opt-in flag in the shell that starts Common Test:

export GEMINI_API_KEY="your_api_key_here"
ERLANG_ADK_GEMINI_REST=1 ./rebar3 ct \
  --suite test/readme/readme_live_gemini_SUITE.erl

Despite the historical suite filename, this is REST GenerateContent/SSE using gemini-3.1-flash-lite; it is not Gemini Live. It exercises 17 cases and roughly 39 provider requests, including text, function/tool rounds, Search grounding, multimodal generation/streaming, explicit context-cache creation, artifact/memory tools, orchestration, continuation, Mnesia, telemetry, evaluation, and the HTTP endpoint.

The suite spaces request starts by 4.2 seconds by default. On projects with sufficient quota, pacing can be disabled explicitly:

ERLANG_ADK_GEMINI_REST=1 \
ERLANG_ADK_GEMINI_REST_INTERVAL_MS=0 \
./rebar3 ct --suite test/readme/readme_live_gemini_SUITE.erl

The test transport caps each wait at 15 seconds and retries one non-streaming transport timeout. A non-streaming HTTP 429 receives one bounded backoff of at least 10 seconds; a second 429 fails explicitly. Test agent/direct-turn worker timeouts are 120 seconds, while production defaults remain 60 seconds. These settings pace only the paid suite and do not change production concurrency or retry behavior.

The final 2026-07-15 v0.7 run completed all 17 cases with no skips: 15 passed; google_search_grounding and context_cache each failed with HTTP 429 after one bounded retry. Preserve those results as quota failures, not product passes or skips. The rubric judge and artifact/memory cases passed.

The 2026-07-17 v0.8 attempt reached Google, but the provider rejected the configured credential shape with HTTP 401 UNAUTHENTICATED and reason ACCESS_TOKEN_TYPE_UNSUPPORTED. The suite therefore did not produce a paid REST pass. Record this as an external credential failure, not a pass, skip, or Erlang ADK product regression; deterministic Gemini REST/header evidence remains separate.

ERLANG_ADK_LIVE_GEMINI and ERLANG_ADK_LIVE_GEMINI_INTERVAL_MS remain compatibility aliases for this historical REST suite. New automation should use the unambiguous REST names.

Paid Gemini Live gate

export GEMINI_API_KEY="your_api_key_here"
ERLANG_ADK_GEMINI_LIVE=1 ./rebar3 ct \
  --suite test/models/gemini/gemini_live_SUITE.erl

This suite uses gemini-3.1-flash-live-preview and covers five cases: text-to-audio plus transcription, 16 kHz PCM input, PNG input, a synchronous tool round trip, and owner-bound browser framing/ACK behavior. The complete 2026-07-15 v0.7 recorded suite passed 5/5. It skips unless both the opt-in flag and key reach the Common Test process.

The paid suite is provider-integration evidence. Deterministic codecs, transport state, ownership, backpressure, reconnect, tool, observability, and voice-bridge tests remain the release contract when network access or quota is unavailable.

No v0.8 paid Gemini Live pass is recorded by the 2026-07-17 validation. The focused Gemini Live broker/transport evidence remains deterministic and does not establish remote-provider success. The v0.7 5/5 result above remains historical and must not be carried forward as v0.8 evidence.

OpenAI, Anthropic, and compatible provider evidence

The 0.8 repository currently provides deterministic injected-transport and codec coverage for OpenAI Responses, Anthropic Messages, compatible Chat Completions, and OpenAI Realtime. It does not currently provide a first-party opt-in paid Common Test suite for those providers. Therefore:

• OPENAI_API_KEY or ANTHROPIC_API_KEY being present is not itself a test;
• deterministic fixture success must be reported as deterministic evidence, not remote-provider success;
• a manual OpenAI/Anthropic smoke must record the exact profile alias, concrete model/version in deployment-owned evidence, date, operation, and structural result without recording prompts, outputs, or credentials; and
• every OpenAI-compatible endpoint needs its own evidence because the adapter does not certify optional vendor semantics.

If a repository paid suite is added later, it must be opt-in, skip cleanly without both its explicit flag and credential, use a release-owned account, and report skips/provider/quota failures separately from passes.

Phoenix, LiveView, and browser gate

cd examples/phoenix_adk_ui
mix deps.get
mix assets.setup
mix precommit
MIX_ENV=prod mix assets.deploy
MIX_ENV=prod mix release
mix hex.audit
elixir ../../scripts/verify_phoenix_hex_audit.exs
../../scripts/smoke_phoenix_release.sh proxy 4101
../../scripts/smoke_phoenix_release.sh tls 4443

For v0.7, mix precommit checked formatting, compiled with warnings as errors, built assets, ran 31 dependency-free Node browser/audio tests, and ran 101 ExUnit tests. Production assets, release assembly, and both loopback smoke modes passed. Those counts are historical. On 2026-07-17 the v0.8 mix precommit gate passed with 103 ExUnit and 40 dependency-free Node tests. Production assets, release assembly, and both trusted-proxy and direct-TLS loopback smokes also passed.

mix hex.audit is expected to remain non-zero at the current lock only for the two Cowlib 2.18.0 advisories documented in SECURITY.md. The wrapper enforces that exact exception and fails for new or stale findings; it does not make the underlying audit a pass. The root Rebar3 project has no equivalent rebar3 hex audit command; do not describe this Mix result as a root dependency audit.

For the recorded v0.8 gate, raw mix hex.audit was non-zero only for those same two advisories, and the exact-exception verifier passed.

The companion's local-auth mode is for interactive development, but its authorization, CSRF, session, gateway, socket, LiveView, static-asset, and audio tests use deterministic fake providers and spend no Gemini quota.

Result policy

Report each category separately:

• pass — the command/case completed its assertions;
• skip — an explicit optional prerequisite was absent;
• provider failure — the live request ran but the provider, network, account, or quota rejected it;
• environment failure — the build runner could not provide required local sockets, tools, certificates, or dependency access; and
• product failure — the implementation or assertion failed under its documented prerequisites.

Never turn a skip, HTTP 429, dependency advisory, or sandbox restriction into a pass. Record command, date, toolchain, model, pass/fail/skip counts, and the bounded structural reason without copying secrets or model content.