HexagonTpu.Stats (hexagon_tpu v0.1.2)

Copy Markdown

Observability for the NIF layer: native resource accounting (leak guard), dirty-scheduler utilization, and memory snapshots.

The three signals to watch in production:

  • native/0 — runtimes_alive/contexts_alive/graphs_alive must stay bounded; monotonic growth means QNN resources are being leaked (references held forever, or destructors never running).
  • dirty_utilization/1 — inference runs on dirty schedulers; sustained saturation there starves other NIFs and file I/O.
  • memory/0 — output tensors are refc binaries; growth in :binary without matching workload growth points at tensors being retained.

HexagonTpu.Monitor samples all of these periodically and emits telemetry.

Summary

Functions

dirty_utilization(sample_ms \\ 1000)

Samples dirty CPU / dirty IO scheduler utilization over sample_ms (default 1000) using microstate accounting.

memory()

Memory snapshot relevant to the NIF: refc binaries (output tensors live here) and total BEAM memory.

native()

Native counters from the NIF

sweep()

Forces a GC of all processes so NIF resource destructors for dropped references run.

with_leak_check(fun)

Runs fun, forces a full GC sweep, and returns {result, before_stats, after_stats} of the native counters. Useful in tests to assert no resources leaked across an operation.

Functions

dirty_utilization(sample_ms \\ 1000)

Samples dirty CPU / dirty IO scheduler utilization over sample_ms (default 1000) using microstate accounting.

Returns %{dirty_cpu: float, dirty_io: float, scheduler: float} where each value is 0.0..1.0 busy share for that scheduler class.

Microstate accounting is a VM-global flag: concurrent samplers (two Monitors, an operator using msacc in a remote shell) corrupt each other's windows. Run one sampler per node — HexagonTpu.Monitor is the intended one.

memory()

Memory snapshot relevant to the NIF: refc binaries (output tensors live here) and total BEAM memory.

native()

Native counters from the NIF:

  • runtimes_alive / contexts_alive / graphs_alive — QNN resources not yet destructed (leak guard: should return to baseline after GC)
  • *_created — lifetime totals
  • executes / execute_errors — inference counters
  • execute_ns_total / execute_ns_last — pure QNN graphExecute time

sweep()

Forces a GC of all processes so NIF resource destructors for dropped references run.

with_leak_check(fun)

Runs fun, forces a full GC sweep, and returns {result, before_stats, after_stats} of the native counters. Useful in tests to assert no resources leaked across an operation.