mix bench.heap (PtcRunner v0.14.0)

Copy Markdown View Source

Measures the heap cost of the PtcRunner embedding units and compares them against bench/baselines/heap.json.

mix bench.heap                  # measure, compare, fail on regression
mix bench.heap --write-baseline # re-record the committed baseline
mix bench.heap --report         # measure and print, never fail

mix bench.check calls report/1 to print the same table informationally alongside the eval-reduction gate. This task is the gating entry point; run it deliberately, at release time or from the scheduled Soak workflow.

Rows

rowwhat it sizes
floor_no_mix_idleresident cost of a started :ptc_runner with no Mix on the path
floor_no_mix_warmthe same child after one Lisp.run/2, so lazy loading has happened
lisp_trivialthe cheapest possible Lisp.run/2 unit
lisp_collection_hofsallocation-heavy builtins
lisp_large_binaryrefc-binary construction and its residual
compile_bundle_1per-artifact resident cost of a compiled bundle
compile_bundle_20whether bundle cost scales linearly in components
kernel_runthe real embedding unit
kernel_run_capabilities_eventsthe same unit with a capability and an event sink
concurrency_Nheap per concurrent Kernel.run, for N in 1/8/32/64/128

Metrics

  • unit_bytes — the workload's own heap at return, before any GC. For the lisp_* rows this is step.usage.memory_bytes, the sandbox child's Process.info(self(), :memory) (sandbox.ex). For the compile_* and kernel_* rows it is the same reading taken in a dedicated driver process. PtcRunner.Kernel.run/2 spawns its own sandbox children, so the kernel_* driver figure excludes themconcurrency_* is the row that sees the whole system.
  • retained_bytes — the retained size of the durable artifact the row produces, from PtcRunner.Lisp.RetainedSize, or null when that module answers :oversized.
  • residual_bytes_per_iter — repeated-batch residual. K equal batches of N iterations with a system-wide GC at each boundary; the reported rate is (endpoint_K - endpoint_1) / ((K - 1) * N) over :erlang.memory(:total). Batch 1 is deliberately excluded: it carries lazy module loading and first-use interning, and a single before/after delta cannot tell that one-shot cost apart from a linear leak. At these iteration counts the metric has no resolution below roughly half a kilobyte per iteration — see @metric_floors. It is here to be read as a trend; the soak suite is where residual growth is hard-asserted, because only there is the workload repeated enough for a slope to outrun the VM's own drift.
  • worker_heap_bytes_total / worker_heap_bytes_medianconcurrency_* only. Each concurrent worker reports its own heap at return, so these are deterministic and are what this row gates on. Like the kernel_* rows they exclude the sandbox children.
  • peak_total_bytes / peak_bytes_per_runconcurrency_* only, from a sampling process reading :erlang.memory(:total). This is the only sample that sees the sandbox children, and it is diagnostic, never compared: :erlang.memory/0 walks every allocator, so the sampler lands ~10 reads on a sub-millisecond run and the figure moved by more than 2x between runs of the unchanged workload. samples_observed records that coverage.

Every figure is a heap figure, not RSS. Allocator carrier fragmentation and resident-set growth are invisible here by construction; the plan puts both in the diagnostic tier, and neither is sampled at all by this task.

Summary

Functions

Measures every row and prints the comparison table without failing.

Functions

report(opts \\ [])

@spec report(keyword()) :: :ok

Measures every row and prints the comparison table without failing.

This is what mix bench.check calls: the heap table is informational there, because a re-baseline reflex is the failure mode a byte gate invites, and the hard assertions live in the soak suite where the workload is repeated enough for a slope to mean something.