Macaw.Data.Operations.Step (macaw_data v0.16.0)

View Source

Running a transform a chunk at a time (spec/04 §1.4): a per-chunk part and a merge, so a table's chunks are worked one by one, and a caller may keep each chunk's part across frames.

A column-native step declares, beside Macaw.Data.Transform's callbacks:

  • its class (class/1), which decides its merge. A per-row step (:row) works each chunk by itself and its merge is concatenation: the chunks it keeps whole, the runs it cuts, and the chunks it gathers, in order.
  • the context it reads (reads/0): only those keys of the caller's context reach it. bind/3, when the step has it, resolves them over the whole input once (window's now, the newest reading when the caller gives none).
  • the context as it resolves for a chunk (resolve/3), given the bound context and the chunk's place: :at, the row of the input its run starts at, and :rows, the input's row count. It reads the chunk's own columns, never another chunk's.
  • its part (chunk/3), from the view and the context resolved for it. A per-row step's part is a selection of the view's rows: :all, the view unchanged; :none; {:run, first, count}, a run of it, cut without copying; or {:rows, indexes}, the rows at those indexes, every column gathered (Macaw.Data.Table.Chunk.gather/5).

The step key of a chunk is the step's identity (the kernel's digest of its module and checked node, as Macaw.Data.Operations.step_digest/2, or of the options its parts depend on when it names them, keyed/1), the context as it resolved for the chunk, and the run of the chunk the view reads when it is not the whole chunk. So a key changes only when what the chunk's part depends on changes: window keys a chunk wholly inside it without now. A cache keys a part by {Chunk.id/1, Chunk.version/1, step key} (spec/01 §1.1.3); a gathered output chunk's id is {input id, step key}.

Summary

Types

A step's fault over its input (spec/04 §3.1): the option it is at, and why. Running in Macaw.Data.Operations.apply/4, it is the node's fault {[index, key], reason}.

One chunk's work: the view the input reads, the context as it resolved for it, and the step key its part is cached by.

A per-row step's selection of a view's rows.

Functions

What a transform's run/3 answers when it runs as a step (run/4): the table, or the fault, {:error, {key, reason}} (Macaw.Data.Transform).

One chunk's part. A per-row step's is the views its selection keeps: none, the input's view or a run of it, or one gathered chunk named {input id, step key}. Any other class's is the step's own partial.

Each part's chunk/3, in the parts' order: inline, or with parallel across the schedulers, one task a part, ordered. Either way the merge receives the same partials in the same order, so a run's result is the same bits inline or parallel (spec/04 §1.6).

A step's fault in words, as Macaw.Data.Operations.format_error/1 words a node's.

A global step's rows gathered across the input's chunks (spec/04 §1.4): picks are {view, row} in output order, the view its place in Table.chunks(input), repeats allowed. The output's columns are columns ({name, type}), each a column of added (a map from name to its values, in output order) or else the input's column of that name, read once a view; one chunk named by key over the inputs (output/3). Every value is read back and packed again, so a dictionary is the output's own: codes are remapped across chunks.

The parts merged into the step's table, as its class says, given in the input's chunk order with their step keys (parts/4)

A combine's or a global step's output: one chunk of columns ({name, type, values}) over every chunk of input, named by key over the inputs' ids and versions (Macaw.Data.Table.Chunk.combine/3), nothing hashed.

Whether a run over table works its chunks in parallel (spec/04 §1.6): only when the caller asks, parallel: true in the context, and the input holds at least two chunks and parallel_rows/0 rows. Inline otherwise, the default, since a caller such as a frame task already runs many steps at once.

The rows at or above which a run asked for parallel chunks runs them in parallel (spec/04 §1.6): two canonical chunks' worth, 2 × Table.chunk_rows/0. Fixed, not a setting; below it a process per chunk costs more than the chunk's work.

Each chunk's work, in order: its view, the context as it resolved for it and its step key. Nothing of a chunk's part is computed here, so a caller holding a part under its key need not compute it again. A node holding a term that is not data (a function, a PID, a reference, a port, an improper list) has no step key: {:error, {key, :not_data}}, the node's key holding it.

The step run over table: each chunk's part, merged as the step's class says (spec/04 §1.4). The same table whatever the chunking. {:ok, table}, or bind/3's fault, {:error, {key, reason}}, before any chunk is worked (spec/04 §3.1).

Whether module is a column-native step: it declares a class and its parts.

A combine's part of a chunk with a delta, by inverse update from its base's part (spec/04 §1.5): base is the part the step computed for the chunk's base, under the same step key (a cache holds it under {Chunk.id/1, Chunk.base_version/1, key}). {:ok, part}, equal to chunk/3's, where the step's update is exact (counts, bin counts, integer sums); :recompute where it is not, or the part's view is not its whole chunk, or the chunk has no delta. Only the delta's rows are worked.

The table with every chunk whose digest or size is pending (a cold chunk under a predicate delete) warmed, as every read of one is (spec/01 §1.1.4); any other table as it is. parts/4 and run/4 warm their input so; a caller merging parts it holds warms the input it merges them over the same way, so the output names the same versions.

Types

fault()

@type fault() :: {String.t(), term()}

A step's fault over its input (spec/04 §3.1): the option it is at, and why. Running in Macaw.Data.Operations.apply/4, it is the node's fault {[index, key], reason}.

part()

@type part() :: %{view: Macaw.Data.Table.view(), context: term(), key: term()}

One chunk's work: the view the input reads, the context as it resolved for it, and the step key its part is cached by.

selection()

@type selection() ::
  :all
  | :none
  | {:run, non_neg_integer(), non_neg_integer()}
  | {:rows, [non_neg_integer()]}

A per-row step's selection of a view's rows.

Functions

answer(module, node, table, context)

@spec answer(module(), map(), Macaw.Data.Table.t(), map()) ::
  Macaw.Data.Table.t() | {:error, fault()}

What a transform's run/3 answers when it runs as a step (run/4): the table, or the fault, {:error, {key, reason}} (Macaw.Data.Transform).

chunk(module, node, map)

@spec chunk(module(), map(), part()) :: term()

One chunk's part. A per-row step's is the views its selection keeps: none, the input's view or a run of it, or one gathered chunk named {input id, step key}. Any other class's is the step's own partial.

chunks(module, node, parts, parallel)

@spec chunks(module(), map(), [part()], boolean()) :: [term()]

Each part's chunk/3, in the parts' order: inline, or with parallel across the schedulers, one task a part, ordered. Either way the merge receives the same partials in the same order, so a run's result is the same bits inline or parallel (spec/04 §1.6).

format_error(arg)

@spec format_error(fault()) :: String.t()

A step's fault in words, as Macaw.Data.Operations.format_error/1 words a node's.

gather(input, picks, columns, added, key)

A global step's rows gathered across the input's chunks (spec/04 §1.4): picks are {view, row} in output order, the view its place in Table.chunks(input), repeats allowed. The output's columns are columns ({name, type}), each a column of added (a map from name to its values, in output order) or else the input's column of that name, read once a view; one chunk named by key over the inputs (output/3). Every value is read back and packed again, so a dictionary is the output's own: codes are remapped across chunks.

merge(module, node, partials, input, keys \\ [], context \\ %{})

@spec merge(module(), map(), [term()], Macaw.Data.Table.t(), [term()], map()) ::
  Macaw.Data.Table.t()

The parts merged into the step's table, as its class says, given in the input's chunk order with their step keys (parts/4):

  • per row: concatenated in order, under the step's output columns (outputs/2: the input's, but for a reshape), keeping the input's order (sorted/1) while its sorted column is kept;
  • a combine: folded in chunk order by the step's combine/3, associative, so any grouping of the same partials gives the same table, then answered by its finish/3, given the input, the output's step key (the step's identity over the parts' keys) and the context it declares, as the parts were given it;
  • global: every partial, in chunk order, given to the step's finish/3 with the same.

output(input, key, columns)

A combine's or a global step's output: one chunk of columns ({name, type, values}) over every chunk of input, named by key over the inputs' ids and versions (Macaw.Data.Table.Chunk.combine/3), nothing hashed.

parallel?(table, context)

@spec parallel?(Macaw.Data.Table.t(), map()) :: boolean()

Whether a run over table works its chunks in parallel (spec/04 §1.6): only when the caller asks, parallel: true in the context, and the input holds at least two chunks and parallel_rows/0 rows. Inline otherwise, the default, since a caller such as a frame task already runs many steps at once.

parallel_rows()

@spec parallel_rows() :: pos_integer()

The rows at or above which a run asked for parallel chunks runs them in parallel (spec/04 §1.6): two canonical chunks' worth, 2 × Table.chunk_rows/0. Fixed, not a setting; below it a process per chunk costs more than the chunk's work.

parts(module, node, table, context)

@spec parts(module(), map(), Macaw.Data.Table.t(), map()) ::
  {:ok, [part()]} | {:error, fault()}

Each chunk's work, in order: its view, the context as it resolved for it and its step key. Nothing of a chunk's part is computed here, so a caller holding a part under its key need not compute it again. A node holding a term that is not data (a function, a PID, a reference, a port, an improper list) has no step key: {:error, {key, :not_data}}, the node's key holding it.

parts!(module, node, table, context)

@spec parts!(module(), map(), Macaw.Data.Table.t(), map()) :: [part()]

As parts/4, raising Macaw.Data.Error naming the fault.

run(module, node, table, context)

@spec run(module(), map(), Macaw.Data.Table.t(), map()) ::
  {:ok, Macaw.Data.Table.t()} | {:error, fault()}

The step run over table: each chunk's part, merged as the step's class says (spec/04 §1.4). The same table whatever the chunking. {:ok, table}, or bind/3's fault, {:error, {key, reason}}, before any chunk is worked (spec/04 §3.1).

run!(module, node, table, context)

@spec run!(module(), map(), Macaw.Data.Table.t(), map()) :: Macaw.Data.Table.t()

As run/4, raising Macaw.Data.Error naming the fault.

step?(module)

@spec step?(module()) :: boolean()

Whether module is a column-native step: it declares a class and its parts.

update(module, node, arg3, base)

@spec update(module(), map(), part(), term()) :: {:ok, term()} | :recompute

A combine's part of a chunk with a delta, by inverse update from its base's part (spec/04 §1.5): base is the part the step computed for the chunk's base, under the same step key (a cache holds it under {Chunk.id/1, Chunk.base_version/1, key}). {:ok, part}, equal to chunk/3's, where the step's update is exact (counts, bin counts, integer sums); :recompute where it is not, or the part's view is not its whole chunk, or the chunk has no delta. Only the delta's rows are worked.

warmed(table)

@spec warmed(Macaw.Data.Table.t()) :: Macaw.Data.Table.t()

The table with every chunk whose digest or size is pending (a cold chunk under a predicate delete) warmed, as every read of one is (spec/01 §1.1.4); any other table as it is. parts/4 and run/4 warm their input so; a caller merging parts it holds warms the input it merges them over the same way, so the output names the same versions.