v0.3.0 benchmark report

Local microbenchmarks of the SDK's message and framing paths, comparing v0.2.1 (protox 2.0.9) with v0.3.0 (protox 2.1.0). No network access: every workload runs against in-memory fixtures, so these numbers describe the CPU cost of the generated encoders/decoders and the wire framer, not a Longbridge round trip.

What differs between the two trees

v0.3.0 moves protox 2.0.9 → 2.1.0 and regenerates lib/longbridge/_protos.ex. The .proto sources are byte-identical in both trees, so any difference below comes from the generated code and the protox runtime, never from a schema change.

Method

v0.2.1 was checked out in a git worktree; v0.3.0 was the release commit. Both ran on the same machine and toolchain: Elixir 1.20.4, Erlang/OTP 29.0.5, 10 schedulers.
scripts/bench_proto.exs (standard library only — no benchmark dependency) ran three times per tree, interleaved.
Each workload warms up, then auto-calibrates a batch size large enough to take at least 20 ms, then reports the median of 7 samples of that batch.
Reported per operation:
- time — wall clock via :timer.tc/1
- reductions — via Process.info(self(), :reductions)
- allocation — words reclaimed per operation, via :erlang.statistics(:garbage_collection) around a full collection
Every workload reports a checksum of the value it produced. All six runs produced identical checksums for all eight workloads, so both trees encoded and decoded byte-identical payloads.

Reductions and allocation volume are deterministic for a given workload and are therefore the trustworthy signal here. Wall time is a median on a laptop that was also doing other work, so treat it as indicative.

Results

Medians of three runs per tree.

workload	ns/op (0.2.1 → 0.3.0)	Δ	reductions/op	Δ	allocation (words/op)	Δ	checksum
encode AuthRequest (3 fields)	440 → 337	−23.3%	114 → 81	−29.1%	106 → 77	−27.4%	34439309
encode SecurityQuote (14 fields, 3 nested)	2,467 → 1,984	−19.6%	562 → 356	−36.6%	790 → 502	−36.5%	2504037
encode SecurityDepthResponse (2 × 50 levels)	35,320 → 30,699	−13.1%	5,961 → 4,000	−32.9%	11,247 → 6,045	−46.3%	86817883
decode SecurityQuote	4,379 → 2,754	−37.1%	838 → 334	−60.1%	2,186 → 1,516	−30.6%	97823793
decode SecurityDepthResponse (2 × 50 levels)	52,633 → 32,115	−39.0%	9,538 → 2,852	−70.1%	25,982 → 14,136	−45.6%	103645140
Header.pack + unpack (11-byte request)	179 → 178	−0.4%	19 → 19	0.0%	74 → 74	0.0%	53739302
Protocol round trip (request, quote body)	379 → 378	−0.3%	30 → 30	0.0%	117 → 117	0.0%	2504037
Protocol round trip (response, gzipped depth body)	5,102 → 5,236	+2.6%	344 → 344	0.0%	142 → 142	0.0%	86817883

Reading the numbers

Encoding: 13–23% faster, 29–37% fewer reductions, 27–46% less allocation.
Decoding: 37–39% faster, 60–70% fewer reductions, 31–46% less allocation.
The last three rows are the control group. Header.pack + unpack and both Protocol.pack + unpack round trips never touch protobuf code, and their reductions and allocation are identical between the trees (−0.0% on every one), with wall time inside run-to-run variance. That is what this harness looks like when the code under it has not changed — which is what makes the deltas above credible rather than machine drift.
The gzip round trip's +2.6% is noise, not a regression: identical reductions and allocation mean it did exactly the same work, and the per-run samples overlap (5002, 5102, 5370 vs 5236, 5270, 5150 ns/op).
The decode gains exceed the protox release notes' claim (~30% less CPU, ~60% fewer reductions); the encode gains land near the claimed ~9% CPU / ~37% reductions in reductions terms, and better in wall-clock terms. Absolute numbers are specific to these fixtures and this machine.

Raw wall-time samples (ns/op, three runs per tree) for the two heaviest workloads, showing the separation is larger than the spread:

decode SecurityDepthResponse   old=[52633, 55883, 51658]   new=[32335, 32115, 32029]
encode SecurityDepthResponse   old=[35480, 35320, 34809]   new=[30974, 30699, 30485]

Reproducing

# candidate tree
mix run scripts/bench_proto.exs > /tmp/bench-new.json

# baseline tree
git worktree add --detach /tmp/lb-baseline v0.2.1
cp scripts/bench_proto.exs /tmp/lb-baseline/scripts/
cd /tmp/lb-baseline && mix deps.get && mix run scripts/bench_proto.exs > /tmp/bench-old.json

# side by side
jq -r '.results[] | [.name, .ns_per_op, .reductions_per_op, .words_per_op] | @tsv' /tmp/bench-*.json

Equal checksum values between the two output files mean both trees ran on identical data. Note that the first run in a fresh tree also prints dependency compilation to stdout, so it is worth discarding and re-running before comparing.

← Previous Page LICENSE