Macaw.Data.Transform behaviour (macaw_data v0.9.0)

View Source

The transform behavior (spec/04 §1.1): one step of a data pipeline, a table in and a table out, declared once so the validator, a drawer's forms and vocabulary, the JSON forms and the sandbox all read the same declaration (visualize#886).

defmodule MyApp.Ops.Clip do
  @behaviour Macaw.Data.Transform
  alias Macaw.Data.Operations.Option

  @impl true
  def name, do: "clip"
  @impl true
  def options, do: [{"field", %Option{type: :field, required: true}}]
  @impl true
  def needs, do: []
  @impl true
  def outputs(_node, columns), do: columns
  @impl true
  def run(node, table, _context), do: ...
end

A member is pure and deterministic: no processes, ETS, NIFs, Regex, clock or randomness. run/3 and outputs/2 are given the node checked (Macaw.Data.Operations.validate/3): its keys strings, every option present with its default filled in, an enum value its atom, a frame-bound "plot" already a number.

Summary

Types

A step's class (spec/04 §1.4), which decides its merge.

Columns in order, each with its Macaw.Data.Table type.

Context a transform may need beyond the data: a drawer's frame, projection, text.

A checked node: the op's options by their strings, each holding its value.

Callbacks

The declared context resolved over the whole input, once: a map the step's resolve/3 is given. Without it the declared context is given as it is.

One chunk's part, from its view and the context resolved for it.

A column-native step's class for the node (spec/04 §1.4): :row, each chunk worked by itself and merged by concatenation; :combine, partials merged associatively; :global, a merge over the whole input. Macaw.Data.Operations.Step runs it.

A combine's two partials merged, the first's chunks before the second's: associative, so any grouping of a table's partials gives the same merged partial.

A combine's table from its merged partial (nil for an input of no chunks), given %{input: table, key: step_key}, the key naming its output (Step.output/3).

The node's options a chunk's part depends on, when not all: what its step key holds in place of the whole node. m4 and lttb leave out their count, which only the merge reads.

The op a node names, unique in a registry.

The context the op requires beyond the data: [] for a table operation.

Every key the op reads, in order, each declared.

The columns run/3 answers for the node over input columns.

Computed once per design and shared by its marks, under the context's prepared.

The keys of the caller's context the step reads; no other key reaches it.

A reducer's count option and the rows each unit of it keeps at most (spec/04 §7).

The context as it resolves for one chunk's view, given the bound context and the view's place (:at, its first row in the input, and :rows, the input's count). Part of the chunk's step key.

The step: the table after it.

A combine's inverse update (spec/04 §1.5): a chunk's part, from its base's part and its delta's changes, {:ok, part} equal to the part the chunk as read would compute; :recompute where that is not exact. change holds removed and inserted, the parts of the base rows the delta removes and of its side rows, under the context the chunk's part resolved to; gone, the removed rows' base indexes; and base and side, where each base row and each side row lies in the chunk as read (nil for a removed row). Only counts, bin counts and integer sums have one.

Types

class()

@type class() :: :row | :combine | :global

A step's class (spec/04 §1.4), which decides its merge.

columns()

@type columns() :: [{Macaw.Data.Table.name(), Macaw.Data.Table.type()}]

Columns in order, each with its Macaw.Data.Table type.

need()

@type need() :: :frame | :projection | :text

Context a transform may need beyond the data: a drawer's frame, projection, text.

node_map()

@type node_map() :: %{required(String.t()) => term()}

A checked node: the op's options by their strings, each holding its value.

Callbacks

bind(node_map, t, map)

(optional)
@callback bind(node_map(), Macaw.Data.Table.t(), map()) :: map()

The declared context resolved over the whole input, once: a map the step's resolve/3 is given. Without it the declared context is given as it is.

chunk(node_map, view, term)

(optional)
@callback chunk(node_map(), Macaw.Data.Table.view(), term()) :: term()

One chunk's part, from its view and the context resolved for it.

class(node_map)

(optional)
@callback class(node_map()) :: class()

A column-native step's class for the node (spec/04 §1.4): :row, each chunk worked by itself and merged by concatenation; :combine, partials merged associatively; :global, a merge over the whole input. Macaw.Data.Operations.Step runs it.

combine(node_map, term, term)

(optional)
@callback combine(node_map(), term(), term()) :: term()

A combine's two partials merged, the first's chunks before the second's: associative, so any grouping of a table's partials gives the same merged partial.

finish(node_map, term, map)

(optional)
@callback finish(node_map(), term(), map()) :: Macaw.Data.Table.t()

A combine's table from its merged partial (nil for an input of no chunks), given %{input: table, key: step_key}, the key naming its output (Step.output/3).

keyed(node_map)

(optional)
@callback keyed(node_map()) :: map()

The node's options a chunk's part depends on, when not all: what its step key holds in place of the whole node. m4 and lttb leave out their count, which only the merge reads.

name()

@callback name() :: String.t()

The op a node names, unique in a registry.

needs()

@callback needs() :: [need()]

The context the op requires beyond the data: [] for a table operation.

options()

@callback options() :: [{String.t(), Macaw.Data.Operations.Option.t()}]

Every key the op reads, in order, each declared.

outputs(node_map, columns)

@callback outputs(node_map(), columns()) :: columns()

The columns run/3 answers for the node over input columns.

prepare(list, t, map)

(optional)
@callback prepare([node_map()], Macaw.Data.Table.t(), map()) :: term()

Computed once per design and shared by its marks, under the context's prepared.

reads()

(optional)
@callback reads() :: [atom()]

The keys of the caller's context the step reads; no other key reaches it.

reduces()

(optional)
@callback reduces() :: {String.t(), pos_integer()}

A reducer's count option and the rows each unit of it keeps at most (spec/04 §7).

resolve(node_map, view, map)

(optional)
@callback resolve(node_map(), Macaw.Data.Table.view(), map()) :: term()

The context as it resolves for one chunk's view, given the bound context and the view's place (:at, its first row in the input, and :rows, the input's count). Part of the chunk's step key.

run(node_map, t, map)

@callback run(node_map(), Macaw.Data.Table.t(), map()) :: Macaw.Data.Table.t()

The step: the table after it.

update(node_map, term, map, term)

(optional)
@callback update(node_map(), term(), map(), term()) :: {:ok, term()} | :recompute

A combine's inverse update (spec/04 §1.5): a chunk's part, from its base's part and its delta's changes, {:ok, part} equal to the part the chunk as read would compute; :recompute where that is not exact. change holds removed and inserted, the parts of the base rows the delta removes and of its side rows, under the context the chunk's part resolved to; gone, the removed rows' base indexes; and base and side, where each base row and each side row lies in the chunk as read (nil for a removed row). Only counts, bin counts and integer sums have one.