beam.smp SIGSEGV in enif_make_map_from_arrays — root cause found

Investigator: independent native-runtime review, 2026-08-25. Worktree: rustyjson-beam-crash-opus (based on origin/main, RustyJson 0.4.0).

Root cause: two NIFs in one BEAM each statically link mimalloc. On macOS each copy keeps its arenas and page map private to its own .so but stores the per-thread default heap in the same fixed TLS slot. One copy's mi_realloc can therefore be handed a block the other copy allocated, miss its page-map lookup, report usable_size == 0, and copy zero bytes — silently returning fresh memory and freeing the original. In parse_object that destroys keys[0] and keys[1], which reach enif_make_map_from_arrays as THE_NON_VALUE and kill the VM inside erts_cmp.

Reproduced on demand, 100% of runs, in seconds. Fixed and regression-tested.

1. Provenance

FACT. Everything below was disassembled from the binaries that actually crashed:

The production artifact was never rebuilt or replaced; it remains the control.

Embedded compiler metadata, recovered from the artifact:

No installed toolchain matches that commit, so exact codegen reproduction was not possible; it was not needed — the fault reproduces across three different ABIs and two different rustc versions (see §5).

2. The failure, verified instruction by instruction

rustyjson  .so+0x1e244   bl   map_from_term_arrays      ; parse_object, multi-entry path
rustler    .so+0x5102c   blr  x8                        ; -> enif_make_map_from_arrays
beam.smp   0x1001a4e9c   bl   erts_factory_proc_prealloc_init(&f, proc, 2*3+4 = 10 words)
beam.smp   0x1001a4eb0   bl   erts_map_from_ks_and_vs(&f, keys, values, 3)
             0x1001ee638   memcpy(hp, keys, 24)         ; copies the Rust Vec verbatim
beam.smp   0x1001ee6d8   bl   erts_validate_and_sort_flatmap(mp)
             0x1001ee748   ldp  x1, x0, [x26]           ; x1 = ks[0], x0 = ks[1]
             0x1001ee760   bl   erts_cmp                ; insertion sort, ix = 1
beam.smp   0x10012fd5c   ldur x8, [x0, #-0x2]           ; <<<< FAULT, x0 = 0

Register-derived facts (identical in all three reports; only heap addresses differ): mp->size = 3, ix = 1, x0 = x1 = 0. That is the first comparison of the insertion sort, so keys[0] and keys[1] were both THE_NON_VALUE. Nothing is knowable about keys[2] — had it been the bad one, ix would be 2.

Two traps worth recording: beam.smp contains four distinct static _erts_cmp symbols (the crash used 0x10012fc70, not the first one nm reports), and parse_value tail-calls parse_object, which is why no parse_value frame appears and why the crashing map is the root object.

Why the fault address is 0xfffffffffffffffe

boxed_val(t) = t - TAG_PRIMARY_BOXED = t - 2. erts_cmp screens operands with tbnz w0, #0 — bit 0 only — so a word with primary tag 0b00 reaches the load. With a = 0, the address is 0 - 2 = 0xFFFFFFFFFFFFFFFE, matching far and esr = 0x92000004 (level-0 translation fault, read). THE_NON_VALUE == 0 is confirmed: enif_make_badarg and enif_raise_exception both compile to mov x0, #0x0 ; ret.

The same bit-0-only pattern appears in enif_is_binary and enif_is_number, so those are not usable as guards — verified by deliberately crashing a VM. enif_is_atom is tag-only and safe.

enif_make_map_from_arrays validates nothing: it passes the caller's pointers straight through, and erts_map_from_ks_and_vs memcpys and sorts them.

3. Who zeroed the terms

The reallocation

parse_object sizes its key vector with estimate_container_capacity(b'}', 30, 256) + 1. With no structural index (inputs < 256 bytes) that is (remaining / 30).clamp(1, 256) + 1. Measured, not assumed:

MCP-shaped 3-key (37 B) : cap_before=2 -> 3rd push realloc=true  (0x…630080 -> 0x…670000)
tiny 3-key       (19 B) : cap_before=2 -> 3rd push realloc=true
3-key            (296 B): cap_before=3 -> no realloc

So a 3-key root object under 256 bytes reallocates 48 → 96 bytes on the third push. RawVec::grow_one (new_cap = max(2*cap, 4)) and finish_grow are both correct; finish_grow branches on cbz x1 (old_cap == 0) and otherwise calls mi_realloc_aligned, which must preserve the 48 bytes holding keys[0] and keys[1].

The zero-byte copy

Inside mimalloc's _mi_theap_realloc_zero:

37e4c: ldr  x8, [x8, x9]        ; page-map level-1 entry
37e50: cbz  x8, ->0x37f9c       ; MISS
37e58: ldr  x26, [x8, x9, lsl #3]
37e5c: cbz  x26, ->0x37fa0      ; MISS
...
37fa0: mov  x22, #0x0           ; <<<< usable_size = 0
...
37f3c: cmp  x22, x20
37f40: csel x2, x22, x20, lo    ; copysize = min(usable_size, newsize)
37f4c: bl   memcpy              ; min(0, 96) = 0 bytes
37f54: bl   _mi_free            ; old block freed anyway

A page-map miss yields usable_size = 0, so the copy moves nothing and the caller gets a fresh, uninitialised block — zeros on a fresh page. keys[2] is written afterwards, directly into the new buffer.

Why the lookup misses

librustycsv-v0.4.4 also statically links mimalloc. My first pass wrongly cleared it: nm -a | grep mi_ returned 0 because its mimalloc symbols are stripped. The code is unambiguously present:

Both read the per-thread heap from TPIDRRO_EL0 + 0x360 (slot 108) while each consults its own private page map (adrp 0xd2000; ldr [x8, #0xcc0]). A block allocated while that slot held the other instance's heap is invisible to this instance's page map — exactly the miss above.

Corroboration from the original crash: registers x14 = 0x3c875c004c8 and x17 = 0x3c875c003c0, which I initially dismissed as non-addresses, are mimalloc arena pointers in the same high range as the reproduction's 0x33742f60000.

4. Reproduction

The instrument was a temporary term_guard cargo feature (off by default) with three checkpoints: check_produced at key creation, check_after_push after every push, and check_arrays immediately before the ERTS call. The post-push checkpoint is the instruction-locating one, because push is the only step that relocates terms.

It was removed once the root cause was proven — it is diagnostic scaffolding, not part of the fix, and it added call sites to the hottest parse loop. It remains in git history at commit cbf9e21 if it is ever needed again.

Driver: /tmp/dual_alloc_repro.exs — interleave RustyCSV.RFC4180.parse_string and RustyJson.decode! of a small 3-key root object across all schedulers.

[rustyjson term_guard] after_push: keys: len=3 cap_before=2 bad_index=0
  ptr=0x33742f60000 prev_ptr=0x33742e50060 reallocated=true
  words=[0000000000000000, 0000000000000000, 000000010f0b0c8a]

keys[0] = keys[1] = 0, keys[2] a valid boxed term (…8a & 0b11 = 0b10), len = 3, realloc from capacity 2 — bit-for-bit the production crash.

5. Controls: ABI version is not the cause

Every ABI label below was read from the built artifact (following nif_init's constant reference), not from cargo's per-feature build dirs, which report stale values. Each cell is a fresh build plus 200k interleaved ops.

The allocator is the sole discriminant. Three different declared ABIs all corrupt with mimalloc and all survive without it, so the 2.17-artifact-on-a-2.18-VM theory is refuted rather than merely unsupported. A 2.18 rebuild passes only when the allocator is also changed — the allocator-matched counterfactual (2.18 + mimalloc) still corrupts, so rebuilding for 2.18 is not a fix.

local_dynamic_tls is also refuted: the fixed-slot accesses remain (74) because mimalloc's macOS __builtin_thread_pointer() fast path is independent of the __thread TLS model.

Can rustler_precompiled ship 2.18?

No. deps/rustler_precompiled/lib/rustler_precompiled/config.ex:33 sets @available_nif_versions ~w(2.14 2.15 2.16 2.17), and nif_versions is passed through validate_list!/3. Requesting "2.18" raises at compile time. Rustler 0.38 itself does have nif_version_2_18, so a 2.18 artifact is buildable — but only locally under FORCE_RUSTYJSON_BUILD=1, never downloaded or published. This is an independent packaging limitation, unrelated to the crash.

6. Fix and regression coverage

native/rustyjson/Cargo.toml: default = ["mimalloc"] → default = []. RustyJson no longer bundles an allocator; mimalloc remains opt-in with the hazard documented inline. Proven green by cells B and D.

Removing mimalloc from either side fixes this pair — with only one instance, the TLS slot is uncontested. RustyJson is the repo under change here, so it is the side that gives way. RustyCSV 0.4.4 should get the same treatment, since any other mimalloc-bundling NIF would re-create the hazard with it.

test/allocator_isolation_test.exs fails the build if the artifact ever links a bundled allocator again. Verified in both directions: passes on the default build, fails when one is enabled. This matters because the failure mode is silent VM-wide heap corruption that no functional test can detect.

The allocator remains available for VMs where RustyJson is the only NIF bundling one — it is worth ~20% on decode. See ALLOCATOR_SAFETY.md for the measurements, the audit procedure, and the opt-in:

RUSTYJSON_ALLOCATOR=mimalloc FORCE_RUSTYJSON_BUILD=1 mix compile

Verification: 471 Elixir tests pass, cargo clippy clean under the pinned nightly-2026-07-27 toolchain, guard never fired on the JSONTestSuite corpus while it was installed.

7. The SBR JSONProbe

Sound for what it covers — capture before, fsync, delete in after, unique :exclusive filename — but two coverage gaps: eight modules call RustyJson.decode directly and bypass it, and Tidewave decodes with Jason everywhere (plug Tidewave precedes Plug.Parsers), so MCP traffic never reaches it. Plug-level bodies are covered because dev sets plug_init_mode: :runtime.

It also distorts timing materially: fsync per decode deschedules the caller onto dirty IO, and term_to_binary of the payload can trigger a GC immediately before the decode. Given the root cause is now known, the probe can be removed along with its config block.

8. Handoff

Confidence: root cause proven. Mechanism verified by disassembly, reproduced on demand with a bit-for-bit register match, and confirmed by a 2×2 allocator × ABI control matrix plus a negative control (RustyJson alone).

What is proven about who zeroed the terms: mimalloc's _mi_theap_realloc_zero, executing a memcpy of length min(usable_size, newsize) where usable_size was 0 because the block was invisible to that instance's page map. The triggering condition is two mimalloc copies sharing macOS TLS slot 108.

What remains unknown (and does not affect the fix): the exact scheduling that flips slot ownership per thread; the crashing payload's bytes (predicted <256 B, 3-key root object — an MCP-shaped JSON-RPC request fits exactly); and whether values[] was equally corrupted (unobservable — erts_cmp only compares keys, but the values vector reallocates identically, so it almost certainly was).

Files changed

The parser itself is unchanged: native/rustyjson/src/ is byte-identical to main. The diagnostic instrumentation that found the bug lives in history at cbf9e21.

Next commands for the SBR agent

# 1. Rebuild RustyJson from this branch without the bundled allocator.
cd /path/to/sold-by-robots
FORCE_RUSTYJSON_BUILD=1 mix deps.compile rustyjson --force

# 2. Confirm the artifact no longer links mimalloc (expect 0).
strings -a _build/dev/lib/rustyjson/priv/native/*.so | grep -c "^mimalloc: error: $"

# 3. Same check for rusty_csv (expect 1 today -- it should also drop mimalloc).
strings -a _build/dev/lib/rusty_csv/priv/native/*.so | grep -c "^mimalloc: error: $"

# 4. Run the dev server normally.

# 5. Remove the probe: delete lib/cinderbase/json_probe.ex and the
#    config/config.exs block that sets :phoenix, :json_library to it.

Reproduction harness kept in the session scratchpad (zprobe.c, threads.py, abi_of_so.py, matrix.sh, my-control-crashes/). The two .ips files my own control runs produced were moved out of ~/Library/Logs/DiagnosticReports/, so the three original crash reports remain the only beam.smp entries there.