Macaw.Data.Table.Chunk (macaw_data v0.9.0)

View Source

One chunk of a Macaw.Data.Table (spec/01 §1.1.1): a run of rows of every column, each column packed by itself in the table's forms, Arrow's record batch. A chunk is immutable.

  • Its logical id (id/1, spec/01 §1.1.3) is where it sits, which never changes: a source's chunk {source, resolution, slot}, a chunk of a table built from rows its content digest, a step's output its input's id and the step key.
  • Its content digest (digest/1, spec/01 §1.1.2) is its contents' digest as a one-chunk table of its table's names and types, computed once, when the chunk is built from data that enters; reading it never hashes. A step's output has none until it is persisted or compared (Macaw.Data.Table.compact/1).
  • Its version (version/1) is its content digest, or a step's output's derivation digest: the kernel's digest of its inputs' versions and the step key, never of its contents. A cache keys a chunk's results by {id, version, step key}.

A table holds its chunks with the run of each one's rows it reads (Macaw.Data.Table.chunks/1); a chunk itself is always whole.

Summary

Types

A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.

t()

A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.

Functions

The value of column name at row i of the chunk, as read: one slot of its storage, for a step that searches a sorted column (spec/04 §1.4).

The version of the chunk's base (spec/04 §1.5): its content digest before its delta, the version a cache held the base's parts under, and an inverse update starts from. The chunk's version/1 when it has no delta; nil for a step's output, which has no delta.

A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them. The chunk keeps its header, id, content digest and size; it is fetched only when it is read.

A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of several chunks. Its id is {:combined, digest}, the kernel's digest of {step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of {[version, …], step_key}. Nothing of the contents is hashed.

Whether the chunk has a delta over its base (spec/01 §1.1.4).

A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing hashed. Its id is {id(input), step_key}, and its version the kernel's digest of {[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports them for columns that are not a table's.

The chunk's content digest: 64 lowercase hex characters, its contents' digest as a one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or compared. Read, never computed here.

A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes of the run of count rows from first of input, every column gathered from its storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id is {id(input), step_key} and its version the derivation digest of {[version(input)], step_key}; the step key names the run when it is not the whole chunk.

The chunk's columns, {name, type} in order.

The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.

A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of the run of count rows from first of input, repeats allowed, under header ({name, type} in order): a column of added (a map from name to its values) packed from them, any other gathered from the input's column of that name by copying its slots. Named as gather/5 names its chunk; nothing hashed.

The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil while a cold chunk's predicate deletes are pending.

The values of column name in the run of count rows from first, as read through the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a row.

The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3). A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil while the digest is pending (spec/01 §1.1.4): it matches no key.

Types

columns()

@type columns() :: [{term(), atom(), [term()]}]

A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.

t()

@opaque t()

A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.

Functions

at(chunk, name, i)

@spec at(t(), term(), non_neg_integer()) :: term()

The value of column name at row i of the chunk, as read: one slot of its storage, for a step that searches a sorted column (spec/04 §1.4).

base_version(chunk)

@spec base_version(t()) :: String.t() | nil

The version of the chunk's base (spec/04 §1.5): its content digest before its delta, the version a cache held the base's parts under, and an inverse update starts from. The chunk's version/1 when it has no delta; nil for a step's output, which has no delta.

cold(header, id, digest, size, fetch)

@spec cold([{term(), atom()}], term(), String.t(), non_neg_integer(), (-> map())) ::
  t()

A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them. The chunk keeps its header, id, content digest and size; it is fetched only when it is read.

combine(inputs, step_key, columns)

@spec combine([t()], term(), columns()) ::
  {:ok, t()} | {:error, [{list(), term()}, ...]}

A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of several chunks. Its id is {:combined, digest}, the kernel's digest of {step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of {[version, …], step_key}. Nothing of the contents is hashed.

delta?(chunk)

@spec delta?(t()) :: boolean()

Whether the chunk has a delta over its base (spec/01 §1.1.4).

derive(input, step_key, columns)

@spec derive(t(), term(), columns()) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing hashed. Its id is {id(input), step_key}, and its version the kernel's digest of {[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports them for columns that are not a table's.

digest(chunk)

@spec digest(t()) :: String.t() | nil

The chunk's content digest: 64 lowercase hex characters, its contents' digest as a one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or compared. Read, never computed here.

gather(input, first, count, indexes, step_key)

@spec gather(t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term()) ::
  t()

A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes of the run of count rows from first of input, every column gathered from its storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id is {id(input), step_key} and its version the derivation digest of {[version(input)], step_key}; the step key names the run when it is not the whole chunk.

header(chunk)

@spec header(t()) :: [{term(), atom()}]

The chunk's columns, {name, type} in order.

id(chunk)

@spec id(t()) :: term()

The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.

reshape(input, first, count, indexes, step_key, header, added)

@spec reshape(
  t(),
  non_neg_integer(),
  non_neg_integer(),
  [non_neg_integer()],
  term(),
  list(),
  map()
) :: t()

A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of the run of count rows from first of input, repeats allowed, under header ({name, type} in order): a column of added (a map from name to its values) packed from them, any other gathered from the input's column of that name by copying its slots. Named as gather/5 names its chunk; nothing hashed.

size(chunk)

@spec size(t()) :: non_neg_integer() | nil

The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil while a cold chunk's predicate deletes are pending.

values(chunk, name, first, size)

@spec values(t(), term(), non_neg_integer(), non_neg_integer()) :: [term()]

The values of column name in the run of count rows from first, as read through the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a row.

version(chunk)

@spec version(t()) :: String.t() | nil

The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3). A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil while the digest is pending (spec/01 §1.1.4): it matches no key.