Benchmarks

The public benchmark runners provide reproducible host-side and AtomVM-compatible workloads without publishing private audit artifacts or device logs.

Run locally

scripts/bench.sh

The remediation benchmark compares representative decode, encode, sequence, malformed-input, partial, and deterministic-map workloads. Lower latency is better. Results vary by CPU, OTP version, architecture, scheduler state, and power policy, so compare versions on the same system and report methodology with every result.

Release notes may contain independently validated host and hardware summaries. These are historical measured context, not universal latency guarantees and not claims that the implementation is leak-free or secure against every possible workload.

The public release gate reads the candidate version from VERSION, considers only SemVer tags reachable from the candidate commit, normalizes the optional historical v prefix, excludes the current version, and selects the greatest lower SemVer as the baseline. Missing or equal-precedence duplicate baselines fail closed. The log records the selected tag and peeled commit SHA, and result files include both baseline and candidate version/SHA identities. This avoids a stale hard-coded baseline while keeping the comparison reproducible and auditable. APIs absent from the selected baseline are reported as N/A rather than inferred.

CI always runs public hygiene and changed-path policy first. A newly opened PR is classified against its base; later synchronize events use the previous exact PR head, so a documentation-only follow-up does not replay an unchanged runtime suite. Workflow-definition changes are the exception: they select the complete chain so a changed gate proves its own scheduling and conclusions. Deleted paths are classified, unknown executable paths fail closed into the broad validation class, and a tag, manual run, or missing comparison history selects the complete chain.

When runtime or benchmark paths select performance, that gate runs after hygiene in parallel with the OTP 25, 27, and 29 compatibility matrix. OTP validation is therefore reported even when the OTP 29 performance comparison fails. Only after the unchanged +5% regression limit passes do the other selected coverage, AtomVM, ESP-IDF, and package jobs fan out across the two self-hosted runners. Documentation/release-note changes select the package job without reserving unrelated compiler or firmware capacity. The final job checks that every selected job passed and every unselected job was actually skipped; an empty, ambiguous, or inconsistent routing result fails closed.

Trusted same-repository release heads run the performance gate on the canonical isolated public-performance host with the digest-pinned OTP 29 container already present on that runner; the workflow never pulls a mutable image. Fork code cannot execute on that runner and therefore cannot satisfy the exact-head release gate directly.

Each baseline/current measurement starts a fresh Erlang VM with the same single normal scheduler plus one dirty CPU and one dirty I/O scheduler. Five runs alternate baseline-first and current-first order, then aggregate the run-level statistics by median. This removes scheduler migration as a dominant source of sub-microsecond p95 noise without relaxing the +5% limit or hiding an individual failure. Baseline capabilities are probed from the selected tag, so APIs absent from an old baseline are N/A; future baselines are not forced through a release-specific compile flag.

Publication has a separate fail-closed lineage check. The release tag must match VERSION, resolve to the checked-out commit, be reachable from origin/main, and have a successful full tag-push release-gate run for the same tag name and commit SHA. The protected hex-production environment is reached only after those checks and the complete package/coverage/docs preflight pass. This prevents an unmerged but previously green PR commit, a moved tag, or an unrelated successful check from becoming a publishable release identity.

Hardware release comparisons follow the same rules: compare exact public versions on the same physical device, runtime, firmware, clock configuration, payload, harness, warmup, and iteration count. Run at the device's documented supported clock unless the report explicitly records another operating point. Never relabel historical evidence with a different clock or calculate a cross-device percentage.

Valid retained captures are the source for release tables. Do not rerun a hardware benchmark merely to regenerate documentation or reformat a report. A new capture is justified only when required evidence is absent or invalid, the code or benchmark changes, or the controlled hardware/runtime configuration changes. Preserve the old result as historical evidence when a new operating point is introduced.

For an exact-tag documentation-only patch, the host performance gate compares the selected previous SemVer commit with the candidate across the runtime modules, options header, and benchmark harness. It reuses the previous exact evidence only when every one of those paths is byte-identical and records the combined source-set SHA-256. Any changed byte follows the normal measured gate with the same 5% median and p95 thresholds.

Changes to the performance orchestration are checked by public hygiene and local pre-PR validation. The restricted performance runner executes them at the automatic exact-tag gate. Runtime or benchmark-harness changes still select performance validation before the release tag.

See the 0.3.1 validation report for the current package and retained exact evidence, and the 0.3.0 validation report for the original exact-public-tag, same-device comparison against 0.2.0 on the ESP32-S3 N16R8, WaveShare N32R16V, and RP2040 E462 boards at 240 MHz, 240 MHz, and 133 MHz respectively. The 0.2.0 validation report retains the earlier 160 MHz and maximum-bound evidence unchanged.