Macaw.Data.Native (macaw_data v0.18.0)
View SourceThe optional native kernels (spec/04 §1.7): Zig, compiled through Zigler when the build has a Zig of the version Zigler targets, each reading a column's packed Arrow buffers in place and answering exactly what the step's Elixir path answers. Without them every step runs its Elixir path, with the same results.
loaded?/0is decided once, when the application starts, and held in:persistent_term, so a step asks it for free.config :macaw_data, native: :optional | :required, default:optional::requiredfails the build without the toolchain, and the application's start when the built kernels don't load;:optionalwarns at compile time and runs the Elixir path.- The 1 ms rule: a call over at most
sync_rows/0rows runs on a normal scheduler, well within a millisecond; a larger one on a dirty CPU scheduler (scheduler/1). - Counted on normal schedulers: a call on a normal scheduler is charged a few
reductions whatever its time, and one on a dirty scheduler is charged by its time, so
normal_schedulers/1makes every call in its process on a normal scheduler for a statement of work in reductions with the kernels (spec/04 §1.7, macaw_data#60).
Summary
Functions
Whether a step in this process runs its kernels: they are loaded and the process is not
inside elixir/1. Each step with a kernel asks this.
Whether this build compiled the native kernels (the toolchain was present).
fun's result, every step in it run on its Elixir path, kernels loaded or not. The
Elixir path is the reference a kernel is held equal to, and with no kernel in it a run's
reductions count all its work.
The flags of scope, and whichever of the two the caller already holds, for the rest of
the calling process: only for a process that ends with the scope, such as a counter's
spawned run. scope/2 is the scope for any other caller, since nothing clears the flag.
The limit a normal group_sum call is given: sync_entries/0, past which it answers
:dirty and is made again on a dirty scheduler; 0, no limit, inside
normal_schedulers/1.
The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal
scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's
1 ms rule; past it the kernel runs on a dirty CPU scheduler.
Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else
:dirty; :normal inside normal_schedulers/1.
Whether the native kernels are built and loaded: decided once, at the application's start.
The native: setting this build was compiled under: :optional or :required.
fun's result, every kernel call in it made on a normal scheduler, whatever its size:
scheduler/1 and finish_scheduler/1 answer :normal, and entry_limit/0 lifts the
limit past which a normal group_sum call hands itself to a dirty one. The same kernel
runs either way; only the scheduler changes.
Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty;
:normal inside normal_schedulers/1.
fun run in a process spawned for it alone, holding scope's flag and the caller's
other one, its $callers and its minimum heap sizes: {:ok, result}, or {:error, {:down, reason}} when it raised, threw or exited, reason its exit reason ({exception, stacktrace} for a raise). The flag lives and ends with that process, so no exit from
fun can leave it set in the caller (spec/04 §1.7, macaw_data#101): a flag put in the
caller's dictionary and cleared after fun stayed set when a raise in fun was caught.
fun must not depend on the caller's mailbox or self(). Nothing outlives it but a
caller killed while it runs, which leaves it to finish alone.
As scope/2, fun's result alone; a raise, throw or exit in fun is passed on to the
caller as it ended fun's process: the exception with its stack trace, the thrown value,
or the exit reason.
The most entries, groups and their float exponents' sums, an exact sum by group
(group_sum) answers on a normal scheduler, 2048: its scan is a few
nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds
each, so a call answering more is handed, before any term is built, to a dirty CPU
scheduler, within the BEAM's 1 ms rule.
The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.
Types
Functions
@spec active?() :: boolean()
Whether a step in this process runs its kernels: they are loaded and the process is not
inside elixir/1. Each step with a kernel asks this.
@spec built?() :: boolean()
Whether this build compiled the native kernels (the toolchain was present).
@spec elixir((-> result)) :: result when result: term()
fun's result, every step in it run on its Elixir path, kernels loaded or not. The
Elixir path is the reference a kernel is held equal to, and with no kernel in it a run's
reductions count all its work.
fun runs in a process of its own (scope!/2), which holds the scope's flag and ends
with it, so no exit from fun leaves the caller's process on the Elixir path
(macaw_data#101). Its reductions are that process's: a count of them is taken inside it,
as Macaw.Data.Test.Work.reductions(fun, native: :elixir) takes it.
@spec enter(scope()) :: :ok
The flags of scope, and whichever of the two the caller already holds, for the rest of
the calling process: only for a process that ends with the scope, such as a counter's
spawned run. scope/2 is the scope for any other caller, since nothing clears the flag.
A host that counts reductions in a process of its own enters the scope there, since
elixir/1 and normal_schedulers/1 run their function in another process, whose work
the count would miss (spec/04 §1.7, macaw_data#224). Macaw.Data.Test.Work's counted
run does the same. A scope outside scope/0 is a FunctionClauseError, as at
scope/2.
iex> parent = self()
iex> spawn(fn ->
...> :ok = Macaw.Data.Native.enter(:elixir)
...> :ok = Macaw.Data.Native.enter(:normal_schedulers)
...> big = Macaw.Data.Native.sync_rows() + 1
...> send(parent, {:counted, Macaw.Data.Native.active?(), Macaw.Data.Native.scheduler(big)})
...> end)
iex> receive do: ({:counted, active?, scheduler} -> {active?, scheduler})
{false, :normal}
iex> Macaw.Data.Native.scheduler(Macaw.Data.Native.sync_rows() + 1)
:dirty
@spec entry_limit() :: non_neg_integer()
The limit a normal group_sum call is given: sync_entries/0, past which it answers
:dirty and is made again on a dirty scheduler; 0, no limit, inside
normal_schedulers/1.
@spec finish_buckets() :: pos_integer()
The most buckets a finish kernel (m4's columns, lttb's selection) reads on a normal
scheduler, 2048: a few tens of nanoseconds a bucket, well within the BEAM's
1 ms rule; past it the kernel runs on a dirty CPU scheduler.
@spec finish_scheduler(non_neg_integer()) :: :normal | :dirty
Where a finish kernel over buckets runs: :normal up to finish_buckets/0, else
:dirty; :normal inside normal_schedulers/1.
@spec loaded?() :: boolean()
Whether the native kernels are built and loaded: decided once, at the application's start.
@spec mode() :: :optional | :required
The native: setting this build was compiled under: :optional or :required.
@spec normal_schedulers((-> result)) :: result when result: term()
fun's result, every kernel call in it made on a normal scheduler, whatever its size:
scheduler/1 and finish_scheduler/1 answer :normal, and entry_limit/0 lifts the
limit past which a normal group_sum call hands itself to a dirty one. The same kernel
runs either way; only the scheduler changes.
A kernel call on a normal scheduler is charged a fixed few reductions, whatever its
time; one on a dirty CPU scheduler is charged by its time, about four a microsecond on
the box #60 measured, so it counts more on a loaded host. A statement of work in
reductions with the kernels, such as the frame's (spec/04 §1.5.1), is counted inside
this scope, and Macaw.Data.Test.Work refuses a count with a dirty call in it. A call
past the 1 ms rule then holds its scheduler longer than the rule allows, so this is for
measuring, never for serving.
fun runs in a process of its own (scope!/2), as in elixir/1, so no exit from it
leaves the caller's process on normal schedulers; a count is taken inside that process,
as Macaw.Data.Test.Work.reductions(fun, native: :normal_schedulers) takes it.
@spec scheduler(non_neg_integer()) :: :normal | :dirty
Where a kernel call over rows rows runs: :normal up to sync_rows/0, else :dirty;
:normal inside normal_schedulers/1.
fun run in a process spawned for it alone, holding scope's flag and the caller's
other one, its $callers and its minimum heap sizes: {:ok, result}, or {:error, {:down, reason}} when it raised, threw or exited, reason its exit reason ({exception, stacktrace} for a raise). The flag lives and ends with that process, so no exit from
fun can leave it set in the caller (spec/04 §1.7, macaw_data#101): a flag put in the
caller's dictionary and cleared after fun stayed set when a raise in fun was caught.
fun must not depend on the caller's mailbox or self(). Nothing outlives it but a
caller killed while it runs, which leaves it to finish alone.
As scope/2, fun's result alone; a raise, throw or exit in fun is passed on to the
caller as it ended fun's process: the exception with its stack trace, the thrown value,
or the exit reason.
@spec sync_entries() :: pos_integer()
The most entries, groups and their float exponents' sums, an exact sum by group
(group_sum) answers on a normal scheduler, 2048: its scan is a few
nanoseconds a row, but each entry of its answer is a few terms at tens of nanoseconds
each, so a call answering more is handed, before any term is built, to a dirty CPU
scheduler, within the BEAM's 1 ms rule.
@spec sync_rows() :: pos_integer()
The most rows a kernel call works on a normal scheduler, 8,192, one canonical chunk: a few nanoseconds a row, and at worst, every row its own bucket, about half a millisecond building the answer's terms, within the BEAM's 1 ms rule.