Macaw.Data.Table.Chunk (macaw_data v0.18.0)

View Source

One chunk of a Macaw.Data.Table (spec/01 §1.1.1): a run of rows of every column, each column packed by itself in the table's forms, Arrow's record batch. A chunk is immutable.

  • Its logical id (id/1, spec/01 §1.1.3) is where it sits, which never changes: a source's chunk {source, resolution, slot}, a chunk of a table built from rows its content digest, a step's output its input's id and the step key.
  • Its content digest (digest/1, spec/01 §1.1.2) is its contents' digest as a one-chunk table of its table's names and types, computed once, when the chunk is built from data that enters; reading it never hashes. A step's output has none until it is persisted or compared (Macaw.Data.Table.compact/1).
  • Its version (version/1) is its content digest, or a step's output's derivation digest: the kernel's digest of its inputs' versions and the step key, never of its contents. A cache keys a chunk's results by {id, version, step key}.

A table holds its chunks with the run of each one's rows it reads (Macaw.Data.Table.chunks/1); a chunk itself is always whole.

Summary

Types

A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.

t()

A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.

Functions

The value of column name at row i of the chunk, as read: one slot of its storage, for a step that searches a sorted column (spec/04 §1.4).

The version of the chunk's base (spec/04 §1.5): its content digest before its delta, the version a cache held the base's parts under, and an inverse update starts from. The chunk's version/1 when it has no delta; nil for a step's output, which has no delta.

A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them. The chunk keeps its header, id, content digest and size; it is fetched only when it is read. id is the id id/1 gave the chunk it stands for, so it is data.

A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of several chunks. Its id is {:combined, digest}, the kernel's digest of {step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of {[version, …], step_key}. Nothing of the contents is hashed.

Whether the chunk has a delta over its base (spec/01 §1.1.4).

A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing hashed. Its id is {id(input), step_key}, and its version the kernel's digest of {[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports them for columns that are not a table's.

The chunk's content digest: 64 lowercase hex characters, its contents' digest as a one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or compared. Read, never computed here.

A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes of the run of count rows from first of input, every column gathered from its storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id is {id(input), key} and its version the derivation digest of {[version(input)], key}, key the step key named at the run's first slot (at_slot/2): the step key itself for a run from slot 0, else {step_key, {:from, first}}. So indexes, counted from the run's first row, name the input's slots they read, and two gathers share an id and a version only where they read the same slots under one step key (macaw_data#132, #145).

The chunk's columns, {name, type} in order.

The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.

The instants of :time column name in the run of count rows from first, in order (spec/01 §1.1.5): Macaw.Data.Source.Order.instant/1 of each value, integer Unix microseconds on one scale whatever the kind of time (a NaiveDateTime by its wall clock, a Date at its midnight), nil where one is missing. A packed form's buffer is read as it lies, no time made; the general form, a run of mixed kinds, value by value.

A reduced chunk of input (spec/01 §1.1.3, macaw_data#247): the rows at indexes of the run of count rows from first that a budget's reducer kept, ascending, every column gathered by its slots as gather/5 gathers them. Its id is {id(input), key} whatever rows it keeps, so a reader reducing the chunk by one reducer at one count meets the same id on every call; its version is the derivation digest of {[version(input)], {key, slots}}, slots the input's slots it holds, so it moves only when the input or the rows kept do. Nothing is hashed but that term.

A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of the run of count rows from first of input, repeats allowed, under header ({name, type} in order): a column of added (a map from name to its values) packed from them, any other gathered from the input's column of that name by copying its slots. Named as gather/5 names its chunk, at the run's first slot; nothing hashed.

The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil while a cold chunk's predicate deletes are pending.

The kind of time :time column name holds in the run of count rows from first (spec/01 §1.1.5, macaw_data#149), read from its storage form alone, nothing decoded: a chunk's column is one form, so a run's kind is its chunk's. DateTime for UTC and zoned storage, whose instants order alike; NaiveDateTime; Date; :mixed for the general form, which holds times of more than one kind. A column with no value present packs as DateTime, and is that. nil for a column the chunk does not hold, or one that is not :time, which holds no kind of time.

The values of column name in the run of count rows from first, as read through the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a row.

The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3). A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil while the digest is pending (spec/01 §1.1.4): it matches no key.

Types

columns()

@type columns() :: [{term(), atom(), [term()]}]

A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.

t()

@opaque t()

A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.

Functions

at(chunk, name, i)

@spec at(t(), term(), non_neg_integer()) :: term()

The value of column name at row i of the chunk, as read: one slot of its storage, for a step that searches a sorted column (spec/04 §1.4).

base_version(chunk)

@spec base_version(t()) :: String.t() | nil

The version of the chunk's base (spec/04 §1.5): its content digest before its delta, the version a cache held the base's parts under, and an inverse update starts from. The chunk's version/1 when it has no delta; nil for a step's output, which has no delta.

cold(header, id, digest, size, fetch)

@spec cold([{term(), atom()}], term(), String.t(), non_neg_integer(), (-> map())) ::
  t()

A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them. The chunk keeps its header, id, content digest and size; it is fetched only when it is read. id is the id id/1 gave the chunk it stands for, so it is data.

combine(inputs, step_key, columns)

@spec combine([t()], term(), columns()) ::
  {:ok, t()} | {:error, [{list(), term()}, ...]}

A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of several chunks. Its id is {:combined, digest}, the kernel's digest of {step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of {[version, …], step_key}. Nothing of the contents is hashed.

delta?(chunk)

@spec delta?(t()) :: boolean()

Whether the chunk has a delta over its base (spec/01 §1.1.4).

derive(input, step_key, columns)

@spec derive(t(), term(), columns()) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing hashed. Its id is {id(input), step_key}, and its version the kernel's digest of {[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports them for columns that are not a table's.

digest(chunk)

@spec digest(t()) :: String.t() | nil

The chunk's content digest: 64 lowercase hex characters, its contents' digest as a one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or compared. Read, never computed here.

gather(input, first, count, indexes, step_key)

@spec gather(t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term()) ::
  t()

A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes of the run of count rows from first of input, every column gathered from its storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id is {id(input), key} and its version the derivation digest of {[version(input)], key}, key the step key named at the run's first slot (at_slot/2): the step key itself for a run from slot 0, else {step_key, {:from, first}}. So indexes, counted from the run's first row, name the input's slots they read, and two gathers share an id and a version only where they read the same slots under one step key (macaw_data#132, #145).

header(chunk)

@spec header(t()) :: [{term(), atom()}]

The chunk's columns, {name, type} in order.

id(chunk)

@spec id(t()) :: term()

The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.

instants(chunk, name, first, count)

@spec instants(t(), term(), non_neg_integer(), non_neg_integer()) :: [integer() | nil]

The instants of :time column name in the run of count rows from first, in order (spec/01 §1.1.5): Macaw.Data.Source.Order.instant/1 of each value, integer Unix microseconds on one scale whatever the kind of time (a NaiveDateTime by its wall clock, a Date at its midnight), nil where one is missing. A packed form's buffer is read as it lies, no time made; the general form, a run of mixed kinds, value by value.

pick(input, first, count, indexes, key)

@spec pick(t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term()) ::
  t()

A reduced chunk of input (spec/01 §1.1.3, macaw_data#247): the rows at indexes of the run of count rows from first that a budget's reducer kept, ascending, every column gathered by its slots as gather/5 gathers them. Its id is {id(input), key} whatever rows it keeps, so a reader reducing the chunk by one reducer at one count meets the same id on every call; its version is the derivation digest of {[version(input)], {key, slots}}, slots the input's slots it holds, so it moves only when the input or the rows kept do. Nothing is hashed but that term.

reshape(input, first, count, indexes, step_key, header, added)

@spec reshape(
  t(),
  non_neg_integer(),
  non_neg_integer(),
  [non_neg_integer()],
  term(),
  list(),
  map()
) :: t()

A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of the run of count rows from first of input, repeats allowed, under header ({name, type} in order): a column of added (a map from name to its values) packed from them, any other gathered from the input's column of that name by copying its slots. Named as gather/5 names its chunk, at the run's first slot; nothing hashed.

size(chunk)

@spec size(t()) :: non_neg_integer() | nil

The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil while a cold chunk's predicate deletes are pending.

time_kind(chunk, name, first, count)

@spec time_kind(t(), term(), non_neg_integer(), non_neg_integer()) ::
  DateTime | NaiveDateTime | Date | :mixed | nil

The kind of time :time column name holds in the run of count rows from first (spec/01 §1.1.5, macaw_data#149), read from its storage form alone, nothing decoded: a chunk's column is one form, so a run's kind is its chunk's. DateTime for UTC and zoned storage, whose instants order alike; NaiveDateTime; Date; :mixed for the general form, which holds times of more than one kind. A column with no value present packs as DateTime, and is that. nil for a column the chunk does not hold, or one that is not :time, which holds no kind of time.

values(chunk, name, first, size)

@spec values(t(), term(), non_neg_integer(), non_neg_integer()) :: [term()]

The values of column name in the run of count rows from first, as read through the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a row.

version(chunk)

@spec version(t()) :: String.t() | nil

The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3). A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil while the digest is pending (spec/01 §1.1.4): it matches no key.