Macaw.Data.Native (macaw_data v0.13.0)
View SourceThe optional native kernels (spec/04 §1.7): Zig, compiled through Zigler when the build has a Zig of the version Zigler targets, each reading a column's packed Arrow buffers in place and answering exactly what the step's Elixir path answers. Without them every step runs its Elixir path, with the same results.
loaded?/0is decided once, when the application starts, and held in:persistent_term, so a step asks it for free.config :macaw_data, native: :optional | :required, default:optional::requiredfails the build without the toolchain, and the application's start when the built kernels don't load;:optionalwarns at compile time and runs the Elixir path.- The 1 ms rule: a call over at most
sync_rows/0rows runs on a normal scheduler, well within a millisecond; a larger one on a dirty CPU scheduler (scheduler/1). - Counted on normal schedulers: a call on a normal scheduler is charged a few
reductions whatever its time, and one on a dirty scheduler is charged by its time, so
normal_schedulers/1makes every call in its process on a normal scheduler for a statement of work in reductions with the kernels (spec/04 §1.7, macaw_data#60).
Summary
Functions
Whether a step in this process runs its kernels: they are loaded and the process is not
inside elixir/1. Each step with a kernel asks this.
Whether this build compiled the native kernels (the toolchain was present).
fun's result, every step in this process run on its Elixir path while it runs, kernels
loaded or not. Reductions count no kernel's work, so a statement of work in reductions
is measured here; and the Elixir path is the reference a kernel is held equal to. Not
nested: the flag is cleared when fun returns.
The limit a normal group_sum call is given: sync_entries/0, past which it answers
:dirty and is made again on a dirty scheduler; 0, no limit, inside
normal_schedulers/1.
The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal
scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's
1 ms rule; past it the kernel runs on a dirty CPU scheduler.
Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else
:dirty; :normal inside normal_schedulers/1.
Whether the native kernels are built and loaded: decided once, at the application's start.
The native: setting this build was compiled under: :optional or :required.
fun's result, every kernel call in this process made on a normal scheduler while it
runs, whatever its size: scheduler/1 and finish_scheduler/1 answer :normal, and
entry_limit/0 lifts the limit past which a normal group_sum call hands itself to a
dirty one. The same kernel runs either way; only the scheduler changes.
Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty;
:normal inside normal_schedulers/1.
The most entries, groups and their float exponents' sums, an exact sum by group
(group_sum) answers on a normal scheduler, 2048: its scan is a few
nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds
each, so a call answering more is handed, before any term is built, to a dirty CPU
scheduler, within the BEAM's 1 ms rule.
The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.
Functions
@spec active?() :: boolean()
Whether a step in this process runs its kernels: they are loaded and the process is not
inside elixir/1. Each step with a kernel asks this.
@spec built?() :: boolean()
Whether this build compiled the native kernels (the toolchain was present).
@spec elixir((-> result)) :: result when result: term()
fun's result, every step in this process run on its Elixir path while it runs, kernels
loaded or not. Reductions count no kernel's work, so a statement of work in reductions
is measured here; and the Elixir path is the reference a kernel is held equal to. Not
nested: the flag is cleared when fun returns.
@spec entry_limit() :: non_neg_integer()
The limit a normal group_sum call is given: sync_entries/0, past which it answers
:dirty and is made again on a dirty scheduler; 0, no limit, inside
normal_schedulers/1.
@spec finish_buckets() :: pos_integer()
The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal
scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's
1 ms rule; past it the kernel runs on a dirty CPU scheduler.
@spec finish_scheduler(non_neg_integer()) :: :normal | :dirty
Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else
:dirty; :normal inside normal_schedulers/1.
@spec loaded?() :: boolean()
Whether the native kernels are built and loaded: decided once, at the application's start.
@spec mode() :: :optional | :required
The native: setting this build was compiled under: :optional or :required.
@spec normal_schedulers((-> result)) :: result when result: term()
fun's result, every kernel call in this process made on a normal scheduler while it
runs, whatever its size: scheduler/1 and finish_scheduler/1 answer :normal, and
entry_limit/0 lifts the limit past which a normal group_sum call hands itself to a
dirty one. The same kernel runs either way; only the scheduler changes.
A kernel call on a normal scheduler is charged a fixed few reductions, whatever its
time; one on a dirty CPU scheduler is charged by its time, about four a microsecond on
the box #60 measured, so it counts more on a loaded host. A statement of work in
reductions with the kernels, such as the frame's (spec/04 §1.5.1), is measured here, and
Macaw.Data.Test.Work refuses a count with a dirty call in it. A call past the 1 ms rule
then holds its scheduler longer than the rule allows, so this is for measuring, never
for serving. Not nested: the flag is cleared when fun returns.
@spec scheduler(non_neg_integer()) :: :normal | :dirty
Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty;
:normal inside normal_schedulers/1.
@spec sync_entries() :: pos_integer()
The most entries, groups and their float exponents' sums, an exact sum by group
(group_sum) answers on a normal scheduler, 2048: its scan is a few
nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds
each, so a call answering more is handed, before any term is built, to a dirty CPU
scheduler, within the BEAM's 1 ms rule.
@spec sync_rows() :: pos_integer()
The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.