Macaw.Data.Operations.Step (macaw_data v0.12.0)
View SourceRunning a transform a chunk at a time (spec/04 §1.4): a per-chunk part and a merge, so a table's chunks are worked one by one, and a caller may keep each chunk's part across frames.
A column-native step declares, beside Macaw.Data.Transform's callbacks:
- its class (
class/1), which decides its merge. A per-row step (:row) works each chunk by itself and its merge is concatenation: the chunks it keeps whole, the runs it cuts, and the chunks it gathers, in order. - the context it reads (
reads/0): only those keys of the caller's context reach it.bind/3, when the step has it, resolves them over the whole input once (window'snow, the newest reading when the caller gives none). - the context as it resolves for a chunk (
resolve/3), given the bound context and the chunk's place::at, the row of the input its run starts at, and:rows, the input's row count. It reads the chunk's own columns, never another chunk's. - its part (
chunk/3), from the view and the context resolved for it. A per-row step's part is a selection of the view's rows::all, the view unchanged;:none;{:run, first, count}, a run of it, cut without copying; or{:rows, indexes}, the rows at those indexes, every column gathered (Macaw.Data.Table.Chunk.gather/5).
The step key of a chunk is the step's identity (the kernel's digest of its module
and checked node, as Macaw.Data.Operations.step_digest/2, or of the options its parts
depend on when it names them, keyed/1), the context as it resolved for the chunk, and
the run of the chunk the view reads when it is not the whole chunk.
So a key changes only when what the chunk's part depends on changes: window keys a
chunk wholly inside it without now. A cache keys a part by
{Chunk.id/1, Chunk.version/1, step key} (spec/01 §1.1.3); a gathered output chunk's id
is {input id, step key}.
Summary
Types
A step's fault over its input (spec/04 §3.1): the option it is at, and why. Running in
Macaw.Data.Operations.apply/4, it is the node's fault {[index, key], reason}.
One chunk's work: the view the input reads, the context as it resolved for it, and the step key its part is cached by.
A per-row step's selection of a view's rows.
Functions
What a transform's run/3 answers when it runs as a step (run/4): the table, or the
fault, {:error, {key, reason}} (Macaw.Data.Transform).
One chunk's part. A per-row step's is the views its selection keeps: none, the input's
view or a run of it, or one gathered chunk named {input id, step key}. Any other
class's is the step's own partial.
Each part's chunk/3, in the parts' order: inline, or with parallel across the
schedulers, one task a part, ordered. Either way the merge receives the same partials
in the same order, so a run's result is the same bits inline or parallel (spec/04
§1.6).
A step's fault in words, as Macaw.Data.Operations.format_error/1 words a node's.
A global step's rows gathered across the input's chunks (spec/04 §1.4): picks are
{view, row} in output order, the view its place in Table.chunks(input), repeats
allowed. The output's columns are columns ({name, type}), each a column of added
(a map from name to its values, in output order) or else the input's column of that
name, read once a view; one chunk named by key over the inputs (output/3). Every
value is read back and packed again, so a dictionary is the output's own: codes are
remapped across chunks.
The parts merged into the step's table, as its class says, given in the input's chunk
order with their step keys (parts/4)
A combine's or a global step's output: one chunk of columns ({name, type, values})
over every chunk of input, named by key over the inputs' ids and versions
(Macaw.Data.Table.Chunk.combine/3), nothing hashed.
Whether a run over table works its chunks in parallel (spec/04 §1.6): only when the
caller asks, parallel: true in the context, and the input holds at least two chunks
and parallel_rows/0 rows. Inline otherwise, the default, since a caller such as a
frame task already runs many steps at once.
The rows at or above which a run asked for parallel chunks runs them in parallel
(spec/04 §1.6): two canonical chunks' worth, 2 × Table.chunk_rows/0. Fixed, not a
setting; below it a process per chunk costs more than the chunk's work.
Each chunk's work, in order: its view, the context as it resolved for it and its step key. Nothing of a chunk's part is computed here, so a caller holding a part under its key need not compute it again.
As parts/4, raising Macaw.Data.Error naming the fault.
The step run over table: each chunk's part, merged as the step's class says
(spec/04 §1.4). The same table whatever the chunking. {:ok, table}, or bind/3's
fault, {:error, {key, reason}}, before any chunk is worked (spec/04 §3.1).
As run/4, raising Macaw.Data.Error naming the fault.
Whether module is a column-native step: it declares a class and its parts.
A combine's part of a chunk with a delta, by inverse update from its base's part
(spec/04 §1.5): base is the part the step computed for the chunk's base, under the
same step key (a cache holds it under {Chunk.id/1, Chunk.base_version/1, key}).
{:ok, part}, equal to chunk/3's, where the step's update is exact (counts, bin
counts, integer sums); :recompute where it is not, or the part's view is not its whole
chunk, or the chunk has no delta. Only the delta's rows are worked.
The table with every chunk whose digest or size is pending (a cold chunk under a
predicate delete) warmed, as every read of one is (spec/01 §1.1.4); any other table as
it is. parts/4 and run/4 warm their input so; a caller merging parts it holds warms
the input it merges them over the same way, so the output names the same versions.
Types
A step's fault over its input (spec/04 §3.1): the option it is at, and why. Running in
Macaw.Data.Operations.apply/4, it is the node's fault {[index, key], reason}.
@type part() :: %{view: Macaw.Data.Table.view(), context: term(), key: term()}
One chunk's work: the view the input reads, the context as it resolved for it, and the step key its part is cached by.
@type selection() :: :all | :none | {:run, non_neg_integer(), non_neg_integer()} | {:rows, [non_neg_integer()]}
A per-row step's selection of a view's rows.
Functions
@spec answer(module(), map(), Macaw.Data.Table.t(), map()) :: Macaw.Data.Table.t() | {:error, fault()}
What a transform's run/3 answers when it runs as a step (run/4): the table, or the
fault, {:error, {key, reason}} (Macaw.Data.Transform).
One chunk's part. A per-row step's is the views its selection keeps: none, the input's
view or a run of it, or one gathered chunk named {input id, step key}. Any other
class's is the step's own partial.
Each part's chunk/3, in the parts' order: inline, or with parallel across the
schedulers, one task a part, ordered. Either way the merge receives the same partials
in the same order, so a run's result is the same bits inline or parallel (spec/04
§1.6).
A step's fault in words, as Macaw.Data.Operations.format_error/1 words a node's.
@spec gather( Macaw.Data.Table.t(), [{non_neg_integer(), non_neg_integer()}], [{Macaw.Data.Table.name(), Macaw.Data.Table.type()}], map(), term() ) :: Macaw.Data.Table.t()
A global step's rows gathered across the input's chunks (spec/04 §1.4): picks are
{view, row} in output order, the view its place in Table.chunks(input), repeats
allowed. The output's columns are columns ({name, type}), each a column of added
(a map from name to its values, in output order) or else the input's column of that
name, read once a view; one chunk named by key over the inputs (output/3). Every
value is read back and packed again, so a dictionary is the output's own: codes are
remapped across chunks.
@spec merge(module(), map(), [term()], Macaw.Data.Table.t(), [term()], map()) :: Macaw.Data.Table.t()
The parts merged into the step's table, as its class says, given in the input's chunk
order with their step keys (parts/4):
- per row: concatenated in order, under the step's output columns (
outputs/2: the input's, but for a reshape), keeping the input's order (sorted/1) while its sorted column is kept; - a combine: folded in chunk order by the step's
combine/3, associative, so any grouping of the same partials gives the same table, then answered by itsfinish/3, given the input, the output's step key (the step's identity over the parts' keys) and the context it declares, as the parts were given it; - global: every partial, in chunk order, given to the step's
finish/3with the same.
@spec output(Macaw.Data.Table.t(), term(), [ {Macaw.Data.Table.name(), Macaw.Data.Table.type(), [term()]} ]) :: Macaw.Data.Table.t()
A combine's or a global step's output: one chunk of columns ({name, type, values})
over every chunk of input, named by key over the inputs' ids and versions
(Macaw.Data.Table.Chunk.combine/3), nothing hashed.
@spec parallel?(Macaw.Data.Table.t(), map()) :: boolean()
Whether a run over table works its chunks in parallel (spec/04 §1.6): only when the
caller asks, parallel: true in the context, and the input holds at least two chunks
and parallel_rows/0 rows. Inline otherwise, the default, since a caller such as a
frame task already runs many steps at once.
@spec parallel_rows() :: pos_integer()
The rows at or above which a run asked for parallel chunks runs them in parallel
(spec/04 §1.6): two canonical chunks' worth, 2 × Table.chunk_rows/0. Fixed, not a
setting; below it a process per chunk costs more than the chunk's work.
Each chunk's work, in order: its view, the context as it resolved for it and its step key. Nothing of a chunk's part is computed here, so a caller holding a part under its key need not compute it again.
@spec parts!(module(), map(), Macaw.Data.Table.t(), map()) :: [part()]
As parts/4, raising Macaw.Data.Error naming the fault.
@spec run(module(), map(), Macaw.Data.Table.t(), map()) :: {:ok, Macaw.Data.Table.t()} | {:error, fault()}
The step run over table: each chunk's part, merged as the step's class says
(spec/04 §1.4). The same table whatever the chunking. {:ok, table}, or bind/3's
fault, {:error, {key, reason}}, before any chunk is worked (spec/04 §3.1).
@spec run!(module(), map(), Macaw.Data.Table.t(), map()) :: Macaw.Data.Table.t()
As run/4, raising Macaw.Data.Error naming the fault.
Whether module is a column-native step: it declares a class and its parts.
A combine's part of a chunk with a delta, by inverse update from its base's part
(spec/04 §1.5): base is the part the step computed for the chunk's base, under the
same step key (a cache holds it under {Chunk.id/1, Chunk.base_version/1, key}).
{:ok, part}, equal to chunk/3's, where the step's update is exact (counts, bin
counts, integer sums); :recompute where it is not, or the part's view is not its whole
chunk, or the chunk has no delta. Only the delta's rows are worked.
@spec warmed(Macaw.Data.Table.t()) :: Macaw.Data.Table.t()
The table with every chunk whose digest or size is pending (a cold chunk under a
predicate delete) warmed, as every read of one is (spec/01 §1.1.4); any other table as
it is. parts/4 and run/4 warm their input so; a caller merging parts it holds warms
the input it merges them over the same way, so the output names the same versions.