Benchmarks

The public benchmark runners provide reproducible host-side and AtomVM-compatible workloads without publishing private audit artifacts or device logs.

Run locally

scripts/bench.sh

The remediation benchmark compares representative decode, encode, sequence, malformed-input, partial, and deterministic-map workloads. Lower latency is better. Results vary by CPU, OTP version, architecture, scheduler state, and power policy, so compare versions on the same system and report methodology with every result.

Release notes may contain independently validated host and hardware summaries. These are historical measured context, not universal latency guarantees and not claims that the implementation is leak-free or secure against every possible workload.

The public release gate reads the candidate version from VERSION, considers only SemVer tags reachable from the candidate commit, normalizes the optional historical v prefix, excludes the current version, and selects the greatest lower SemVer as the baseline. Missing or equal-precedence duplicate baselines fail closed. The log records the selected tag and peeled commit SHA, and result files include both baseline and candidate version/SHA identities. This avoids a stale hard-coded baseline while keeping the comparison reproducible and auditable. APIs absent from the selected baseline are reported as N/A rather than inferred.

CI always runs public hygiene and changed-path policy first. A newly opened PR is classified against its base; later synchronize events use the previous exact PR head, so a documentation or workflow-policy follow-up does not replay an unchanged runtime suite. Deleted paths are classified, unknown executable paths fail closed into the broad validation class, and a tag, manual run, or missing comparison history selects the complete chain.

When runtime or benchmark paths select performance, that gate runs alone after hygiene. Only after the unchanged +5% regression limit passes do the other selected OTP, coverage, AtomVM, ESP-IDF, and package jobs fan out across the two self-hosted runners. Documentation/release-note changes select the package job without reserving unrelated compiler or firmware capacity. The final job checks that every selected job passed and every unselected job was actually skipped; an empty, ambiguous, or inconsistent routing result fails closed.

Each baseline/current measurement starts a fresh Erlang VM with the same single normal scheduler plus one dirty CPU and one dirty I/O scheduler. Five runs alternate baseline-first and current-first order, then aggregate the run-level statistics by median. This removes scheduler migration as a dominant source of sub-microsecond p95 noise without relaxing the +5% limit or hiding an individual failure. Baseline capabilities are probed from the selected tag, so APIs absent from an old baseline are N/A; future baselines are not forced through a release-specific compile flag.

Publication has a separate fail-closed lineage check. The release tag must match VERSION, resolve to the checked-out commit, be reachable from origin/main, and have a successful full tag-push release-gate run for the same tag name and commit SHA. The protected hex-production environment is reached only after those checks and the complete package/coverage/docs preflight pass. This prevents an unmerged but previously green PR commit, a moved tag, or an unrelated successful check from becoming a publishable release identity.

See the 0.2.0 validation report for the matched host, ESP32-S3, and RP2040 evidence.