Macaw.Data.Native (macaw_data v0.9.0)

View Source

The optional native kernels (spec/04 §1.7): Zig, compiled through Zigler when the build has a Zig of the version Zigler targets, each reading a column's packed Arrow buffers in place and answering exactly what the step's Elixir path answers. Without them every step runs its Elixir path, with the same results.

  • loaded?/0 is decided once, when the application starts, and held in :persistent_term, so a step asks it for free.
  • config :macaw_data, native: :optional | :required, default :optional: :required fails the build without the toolchain, and the application's start when the built kernels don't load; :optional warns at compile time and runs the Elixir path.

  • The 1 ms rule: a call over at most sync_rows/0 rows runs on a normal scheduler, well within a millisecond; a larger one on a dirty CPU scheduler (scheduler/1).
  • Counted on normal schedulers: a call on a normal scheduler is charged a few reductions whatever its time, and one on a dirty scheduler is charged by its time, so normal_schedulers/1 makes every call in its process on a normal scheduler for a statement of work in reductions with the kernels (spec/04 §1.7, macaw_data#60).

Summary

Functions

Whether a step in this process runs its kernels: they are loaded and the process is not inside elixir/1. Each step with a kernel asks this.

Whether this build compiled the native kernels (the toolchain was present).

fun's result, every step in this process run on its Elixir path while it runs, kernels loaded or not. Reductions count no kernel's work, so a statement of work in reductions is measured here; and the Elixir path is the reference a kernel is held equal to. Not nested: the flag is cleared when fun returns.

The limit a normal group_sum call is given: sync_entries/0, past which it answers :dirty and is made again on a dirty scheduler; 0, no limit, inside normal_schedulers/1.

The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's 1 ms rule; past it the kernel runs on a dirty CPU scheduler.

Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else :dirty; :normal inside normal_schedulers/1.

Whether the native kernels are built and loaded: decided once, at the application's start.

The native: setting this build was compiled under: :optional or :required.

fun's result, every kernel call in this process made on a normal scheduler while it runs, whatever its size: scheduler/1 and finish_scheduler/1 answer :normal, and entry_limit/0 lifts the limit past which a normal group_sum call hands itself to a dirty one. The same kernel runs either way; only the scheduler changes.

Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty; :normal inside normal_schedulers/1.

The most entries, groups and their float exponents' sums, an exact sum by group (group_sum) answers on a normal scheduler, 2048: its scan is a few nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds each, so a call answering more is handed, before any term is built, to a dirty CPU scheduler, within the BEAM's 1 ms rule.

The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.

Functions

active?()

@spec active?() :: boolean()

Whether a step in this process runs its kernels: they are loaded and the process is not inside elixir/1. Each step with a kernel asks this.

built?()

@spec built?() :: boolean()

Whether this build compiled the native kernels (the toolchain was present).

elixir(fun)

@spec elixir((-> result)) :: result when result: term()

fun's result, every step in this process run on its Elixir path while it runs, kernels loaded or not. Reductions count no kernel's work, so a statement of work in reductions is measured here; and the Elixir path is the reference a kernel is held equal to. Not nested: the flag is cleared when fun returns.

entry_limit()

@spec entry_limit() :: non_neg_integer()

The limit a normal group_sum call is given: sync_entries/0, past which it answers :dirty and is made again on a dirty scheduler; 0, no limit, inside normal_schedulers/1.

finish_buckets()

@spec finish_buckets() :: pos_integer()

The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's 1 ms rule; past it the kernel runs on a dirty CPU scheduler.

finish_scheduler(buckets)

@spec finish_scheduler(non_neg_integer()) :: :normal | :dirty

Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else :dirty; :normal inside normal_schedulers/1.

loaded?()

@spec loaded?() :: boolean()

Whether the native kernels are built and loaded: decided once, at the application's start.

mode()

@spec mode() :: :optional | :required

The native: setting this build was compiled under: :optional or :required.

normal_schedulers(fun)

@spec normal_schedulers((-> result)) :: result when result: term()

fun's result, every kernel call in this process made on a normal scheduler while it runs, whatever its size: scheduler/1 and finish_scheduler/1 answer :normal, and entry_limit/0 lifts the limit past which a normal group_sum call hands itself to a dirty one. The same kernel runs either way; only the scheduler changes.

A kernel call on a normal scheduler is charged a fixed few reductions, whatever its time; one on a dirty CPU scheduler is charged by its time, about four a microsecond on the box #60 measured, so it counts more on a loaded host. A statement of work in reductions with the kernels, such as the frame's (spec/04 §1.5.1), is measured here, and Macaw.Data.Test.Work refuses a count with a dirty call in it. A call past the 1 ms rule then holds its scheduler longer than the rule allows, so this is for measuring, never for serving. Not nested: the flag is cleared when fun returns.

scheduler(rows)

@spec scheduler(non_neg_integer()) :: :normal | :dirty

Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty; :normal inside normal_schedulers/1.

sync_entries()

@spec sync_entries() :: pos_integer()

The most entries, groups and their float exponents' sums, an exact sum by group (group_sum) answers on a normal scheduler, 2048: its scan is a few nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds each, so a call answering more is handed, before any term is built, to a dirty CPU scheduler, within the BEAM's 1 ms rule.

sync_rows()

@spec sync_rows() :: pos_integer()

The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.