SafeRPC keeps performance checks reproducible and separate from correctness tests. Benchmark results depend on the host, scheduler count, OTP version, power state, and competing workloads; compare revisions on the same otherwise-idle machine instead of treating one run as a universal number.
Serial compatibility benchmark
bench/safe_rpc_bench.exs measures one-shot calls, a persistent client, and a sharded client pool with multiple payload sizes and parallelism levels. It reports throughput, latency including the 99th percentile, and caller-process allocations.
Run the current checkout with fixed scheduler count:
ELIXIR_ERL_OPTIONS="+S 8:8" mix run bench/safe_rpc_bench.exs
Compare the current checkout with the last release using the same benchmark script and VM settings:
scripts/benchmark_compare.sh v0.1.16 /tmp/safe-rpc-benchmark-results
The comparison script creates a temporary detached worktree, runs the common benchmark against both revisions, and stores Benchee suites plus console output in the selected results directory. It does not modify either revision.
Runtime-foundation benchmark
bench/runtime_foundations_bench.exs compares serialized and concurrent dispatch with a 5 ms handler at parallelism 1, 4, 16, and 64. It also compares an echo call with no telemetry handlers against the same call with a no-op request telemetry handler attached.
ELIXIR_ERL_OPTIONS="+S 8:8" mix run bench/runtime_foundations_bench.exs
At parallelism greater than one, interpret Benchee's per-invocation latency together with the configured parallelism. Serialized dispatch queues the calls; concurrent dispatch should keep latency close to one handler duration until a configured in-flight bound is reached.
Configuration
Both scripts accept environment overrides:
| Variable | Default | Purpose |
|---|---|---|
SAFERPC_BENCH_WARMUP | 1 | Warmup seconds per scenario |
SAFERPC_BENCH_TIME | 3 | Measurement seconds per scenario |
SAFERPC_BENCH_MEMORY_TIME | 1 | Caller-allocation measurement seconds |
SAFERPC_BENCH_PARALLEL | 1,4,16 | Parallelism for the serial compatibility suite |
SAFERPC_BENCH_PAYLOADS | small,1kb,64kb | Payloads for the serial compatibility suite |
SAFERPC_BENCH_DELAY_MS | 5 | Simulated handler time in the runtime suite |
SAFERPC_BENCH_TELEMETRY | true | Run the telemetry comparison |
The runtime suite defaults to parallelism 1,4,16,64 when SAFERPC_BENCH_PARALLEL is unset.
Benchee's memory figure covers allocations attributed to the benchmark caller. Use the stress suite to validate server-side bounds and mailbox behavior rather than interpreting that figure as total system memory.
Stress suite
Run the opt-in stress tests with:
SAFERPC_STRESS=1 mix test --include stress
Increase the request count when validating a release candidate:
SAFERPC_STRESS=1 SAFERPC_STRESS_COUNT=10000 mix test --include stress
The suite covers persistent-client floods, asynchronous requests, concurrent cancellation and slot recovery, bounded serialized connection mailboxes, and sharded client pools.