Testing & interop

How Stevedore is tested, why the suite is shaped the way it is, and how to run each slice. This guide is for contributors (what to run, what each layer proves) and for anyone evaluating the library (how thoroughly its claims are verified).

The short version: mix test is fast, hermetic, and offline; everything that touches a network, a real registry, an external binary, or a Go toolchain is gated behind an ExUnit tag and excluded by default. A machine missing a tool still gets a clean run — the affected cases skip, they never fail.

The core idea: asymmetric interop

Stevedore is both a registry client and a registry server, both a producer and a consumer of OCI artifacts. That symmetry is a trap for testing: if you push with Stevedore and pull with Stevedore, a shared encoding bug passes in both directions and the test stays green. Round-tripping against yourself proves consistency, not correctness.

So the high-value tests are asymmetric — produce with one tool, consume with another:

• Write an oci: layout with Stevedore, load it with skopeo / crane / podman; and read layouts those tools wrote.
• Build an image with Stevedore.Build, push it, then actually podman run it — running the output is the only true test that the layers and config are well-formed.
• Sign with Stevedore, verify with real cosign; sign with cosign, verify with Stevedore.
• Compare Stevedore.Analyze's merged filesystem against crane export for the same image.

Those external tools are oracles: independent implementations, written to the same specs, free of our blind spots. Self round-tripping (Stevedore → Stevedore) is a fallback, not the goal.

The runtime invariant still holds. Stevedore never shells out to skopeo, cosign, crane, tar, or openssl at runtime (see AGENTS.md). These tools appear only in test/, as oracles — never in lib/. Likewise, running a registry:2 / zot container in CI is a test-time dependency, not a runtime one: the library itself talks to registries and on-disk layouts directly.

The testing surface

Each surface is tested against a third-party oracle chosen to expose bugs a self-test can't:

Tags & how to run each slice

The default exclude list lives in test/test_helper.exs:

ExUnit.configure(exclude: [:external, :conformance, :interop])

mix test                          # fast, hermetic, offline — the PR gate (unit + schema validity)
mix test --include external       # client vs real registry:2 + zot (needs the compose stack up)
mix test --include conformance    # boots the server + runs the OCI distribution-spec suite (needs Go)
mix test --include interop        # produce/consume vs skopeo/crane/cosign/podman/oras/regctl

The four layers, fastest first:

1. fast (untagged, hermetic). The pure core — formats, digests, parsing, the Store/copy logic — plus format-validity tests that schema-validate every manifest, index, config, and descriptor Stevedore emits against the upstream image-spec schemas (vendored under test/support/schema/). No network, no binaries. This is the required gate.

2. :external. The docker:// client and copy against two real registries that diverge on edge cases (referrers fallback, cross-repo mount, error envelopes): registry:2 (the CNCF reference implementation, no native Referrers API) and zot (strict, OCI-native). Bring them up first:

   docker compose -f docker-compose.test.yml up -d
   mix test --include external
   docker compose -f docker-compose.test.yml down
   

3. :conformance. The official OCI distribution-spec conformance.test (a Go/Ginkgo binary) run against a live Stevedore.Server, asserting zero failures across pull / push / content-discovery / content-management. The support module clones the pinned spec tag (v1.1.0) and go test -cs it, caching the binary under _build/conformance/. Needs a Go toolchain; skips cleanly without one. (Results can be self-submitted for the OCI conformance badge at https://github.com/opencontainers/oci-conformance.)

4. :interop. The asymmetric produce/consume matrix above, against the external oracle tools.

Tooling

The interop and conformance slices drive real binaries. Exact versions are pinned in CI (.github/workflows/ci.yml) so a green run today is green next month; that workflow is the source of truth. The oracles:

Skip, don't fail, when a tool is absent. Stevedore.TestTools provides tool_test/3 (skips with a clear "missing tools: …" reason when a binary isn't found) and registry_test/3 (skips when the registry isn't reachable). oras and regctl are usually go installed into ~/go/bin, which may not be on PATH; TestTools.find/1 falls back to ~/go/bin, so the suite works whether or not your shell rc exports it.

What the suite has caught

The point of asymmetric interop is to find bugs unit tests structurally can't. It has:

• 3 registry-server bugs (distribution-spec conformance, fixed): out-of-order chunked-blob PATCH now returns 416; a manifest PUT with a subject now echoes the OCI-Subject response header; the referrers index now propagates per-referrer annotations.
• Confirmed cosign interop both ways with real cosign — Stevedore's signatures verify under cosign verify, and cosign-made signatures verify under Stevedore.Verify (wrong key fails closed).
• Confirmed built images run under both podman and docker with the expected Env / WorkingDir / User observed inside the running container.

Known limitations (current scope)

Documented deliberately so the boundaries are honest:

• Signing format. Stevedore implements cosign's legacy simple-signing format (the sha256-<hex>.sig tag, dev.cosignproject.cosign/signature annotation). cosign 3.x's default sign now emits a Sigstore DSSE bundle as an OCI referrer, which Stevedore.Verify does not read yet. cosign verify remains backward-compatible with our signatures. (The interop test pins the legacy format with --registry-referrers-mode=legacy and friends.)
• Referrers on no-API registries. On a registry without the Referrers API (e.g. registry:2), Referrers.attach/4 pushes the artifact with its subject but does not yet maintain the <algo>-<hex> tag-schema fallback index that Referrers.list/3 reads — so the referrer is undiscoverable there. Modern targets with the native API (zot, GHCR, Docker Hub, ECR, GAR, Harbor, distribution v3) work. The :external suite asserts strong behavior on zot and carries a tripwire on registry:2 that flips when this lands.

CI

Four jobs match the tag taxonomy, fastest first (see .github/workflows/ci.yml):

References

• OCI Distribution Spec + conformance — https://github.com/opencontainers/distribution-spec
• OCI Image Spec (+ schema/) — https://github.com/opencontainers/image-spec
• cosign SIGNATURE_SPEC — https://github.com/sigstore/cosign/blob/main/specs/SIGNATURE_SPEC.md
• registry:2 (distribution) — https://github.com/distribution/distribution · zot — https://github.com/project-zot/zot
• skopeo — https://github.com/containers/skopeo · crane — https://github.com/google/go-containerregistry · cosign — https://github.com/sigstore/cosign

See also REFERENCES.md (specs mapped to modules) and AGENTS.md (design boundary and conventions).