Macaw.Data.Table.Chunk (macaw_data v0.18.0)
View SourceOne chunk of a Macaw.Data.Table (spec/01 §1.1.1): a run of rows of every column, each
column packed by itself in the table's forms, Arrow's record batch. A chunk is immutable.
- Its logical id (
id/1, spec/01 §1.1.3) is where it sits, which never changes: a source's chunk{source, resolution, slot}, a chunk of a table built from rows its content digest, a step's output its input's id and the step key. - Its content digest (
digest/1, spec/01 §1.1.2) is its contents' digest as a one-chunk table of its table's names and types, computed once, when the chunk is built from data that enters; reading it never hashes. A step's output has none until it is persisted or compared (Macaw.Data.Table.compact/1). - Its version (
version/1) is its content digest, or a step's output's derivation digest: the kernel's digest of its inputs' versions and the step key, never of its contents. A cache keys a chunk's results by{id, version, step key}.
A table holds its chunks with the run of each one's rows it reads
(Macaw.Data.Table.chunks/1); a chunk itself is always whole.
Summary
Types
A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.
A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.
Functions
The value of column name at row i of the chunk, as read: one slot of its storage,
for a step that searches a sorted column (spec/04 §1.4).
The version of the chunk's base (spec/04 §1.5): its content digest before its delta,
the version a cache held the base's parts under, and an inverse update starts from.
The chunk's version/1 when it has no delta; nil for a step's output, which has no
delta.
A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function
answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them.
The chunk keeps its header, id, content digest and size; it is fetched only when
it is read. id is the id id/1 gave the chunk it stands for, so it is data.
A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of
several chunks. Its id is {:combined, digest}, the kernel's digest of
{step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of
{[version, …], step_key}. Nothing of the contents is hashed.
Whether the chunk has a delta over its base (spec/01 §1.1.4).
A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing
hashed. Its id is {id(input), step_key}, and its version the kernel's digest of
{[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports
them for columns that are not a table's.
The chunk's content digest: 64 lowercase hex characters, its contents' digest as a
one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or
compared. Read, never computed here.
A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes
of the run of count rows from first of input, every column gathered from its
storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id
is {id(input), key} and its version the derivation digest of {[version(input)], key},
key the step key named at the run's first slot (at_slot/2): the step key itself for a
run from slot 0, else {step_key, {:from, first}}. So indexes, counted from the run's
first row, name the input's slots they read, and two gathers share an id and a version
only where they read the same slots under one step key (macaw_data#132, #145).
The chunk's columns, {name, type} in order.
The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.
The instants of :time column name in the run of count rows from first, in order
(spec/01 §1.1.5): Macaw.Data.Source.Order.instant/1 of each value, integer Unix
microseconds on one scale whatever the kind of time (a NaiveDateTime by its wall
clock, a Date at its midnight), nil where one is missing. A packed form's buffer is
read as it lies, no time made; the general form, a run of mixed kinds, value by value.
A reduced chunk of input (spec/01 §1.1.3, macaw_data#247): the rows at indexes of
the run of count rows from first that a budget's reducer kept, ascending, every
column gathered by its slots as gather/5 gathers them. Its id is {id(input), key}
whatever rows it keeps, so a reader reducing the chunk by one reducer at one count
meets the same id on every call; its version is the derivation digest of
{[version(input)], {key, slots}}, slots the input's slots it holds, so it moves only
when the input or the rows kept do. Nothing is hashed but that term.
A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of
the run of count rows from first of input, repeats allowed, under header
({name, type} in order): a column of added (a map from name to its values) packed
from them, any other gathered from the input's column of that name by copying its
slots. Named as gather/5 names its chunk, at the run's first slot; nothing hashed.
The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil
while a cold chunk's predicate deletes are pending.
The kind of time :time column name holds in the run of count rows from first
(spec/01 §1.1.5, macaw_data#149), read from its storage form alone, nothing decoded: a
chunk's column is one form, so a run's kind is its chunk's. DateTime for UTC and zoned
storage, whose instants order alike; NaiveDateTime; Date; :mixed for the general
form, which holds times of more than one kind. A column with no value present packs as
DateTime, and is that. nil for a column the chunk does not hold, or one that is not
:time, which holds no kind of time.
The values of column name in the run of count rows from first, as read through
the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a
row.
The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3).
A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil
while the digest is pending (spec/01 §1.1.4): it matches no key.
Types
A step output's columns: {name, type, values}, as Macaw.Data.Table.new/2 takes.
@opaque t()
A chunk: its row count, each column's storage by name, its logical id, and its content digest or its derivation digest.
Functions
@spec at(t(), term(), non_neg_integer()) :: term()
The value of column name at row i of the chunk, as read: one slot of its storage,
for a step that searches a sorted column (spec/04 §1.4).
The version of the chunk's base (spec/04 §1.5): its content digest before its delta,
the version a cache held the base's parts under, and an inverse update starts from.
The chunk's version/1 when it has no delta; nil for a step's output, which has no
delta.
A cold chunk (spec/01 §1.1.4): its base held elsewhere, fetched by fetch, a function
answering the base's columns by name, as Macaw.Data.Table.chunks/1's chunks hold them.
The chunk keeps its header, id, content digest and size; it is fetched only when
it is read. id is the id id/1 gave the chunk it stands for, so it is data.
A global step's output chunk of inputs, in order (spec/01 §1.1.3): a sort, a merge of
several chunks. Its id is {:combined, digest}, the kernel's digest of
{step_key, [{id, version}, …]} over the inputs, and its version the kernel's digest of
{[version, …], step_key}. Nothing of the contents is hashed.
Whether the chunk has a delta over its base (spec/01 §1.1.4).
A per-row step's output chunk of input (spec/01 §1.1.3): columns packed, nothing
hashed. Its id is {id(input), step_key}, and its version the kernel's digest of
{[version(input)], step_key}. {:error, faults} as Macaw.Data.Table.new/2 reports
them for columns that are not a table's.
The chunk's content digest: 64 lowercase hex characters, its contents' digest as a
one-chunk table (spec/01 §1.1.2); nil for a step's output, until it is persisted or
compared. Read, never computed here.
@spec gather(t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term()) :: t()
A per-row step's output chunk (spec/01 §1.1.3, spec/04 §1.4): the rows at indexes
of the run of count rows from first of input, every column gathered from its
storage by copying its slots, nothing decoded and nothing hashed. As derive/3, its id
is {id(input), key} and its version the derivation digest of {[version(input)], key},
key the step key named at the run's first slot (at_slot/2): the step key itself for a
run from slot 0, else {step_key, {:from, first}}. So indexes, counted from the run's
first row, name the input's slots they read, and two gathers share an id and a version
only where they read the same slots under one step key (macaw_data#132, #145).
The chunk's columns, {name, type} in order.
The chunk's logical id (spec/01 §1.1.3): where it sits, which never changes.
@spec instants(t(), term(), non_neg_integer(), non_neg_integer()) :: [integer() | nil]
The instants of :time column name in the run of count rows from first, in order
(spec/01 §1.1.5): Macaw.Data.Source.Order.instant/1 of each value, integer Unix
microseconds on one scale whatever the kind of time (a NaiveDateTime by its wall
clock, a Date at its midnight), nil where one is missing. A packed form's buffer is
read as it lies, no time made; the general form, a run of mixed kinds, value by value.
@spec pick(t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term()) :: t()
A reduced chunk of input (spec/01 §1.1.3, macaw_data#247): the rows at indexes of
the run of count rows from first that a budget's reducer kept, ascending, every
column gathered by its slots as gather/5 gathers them. Its id is {id(input), key}
whatever rows it keeps, so a reader reducing the chunk by one reducer at one count
meets the same id on every call; its version is the derivation digest of
{[version(input)], {key, slots}}, slots the input's slots it holds, so it moves only
when the input or the rows kept do. Nothing is hashed but that term.
@spec reshape( t(), non_neg_integer(), non_neg_integer(), [non_neg_integer()], term(), list(), map() ) :: t()
A reshaping per-row step's output chunk (spec/04 §1.4, §4.7): the rows at indexes of
the run of count rows from first of input, repeats allowed, under header
({name, type} in order): a column of added (a map from name to its values) packed
from them, any other gathered from the input's column of that name by copying its
slots. Named as gather/5 names its chunk, at the run's first slot; nothing hashed.
@spec size(t()) :: non_neg_integer() | nil
The number of rows the chunk holds, as read through its delta (spec/01 §1.1.4); nil
while a cold chunk's predicate deletes are pending.
@spec time_kind(t(), term(), non_neg_integer(), non_neg_integer()) :: DateTime | NaiveDateTime | Date | :mixed | nil
The kind of time :time column name holds in the run of count rows from first
(spec/01 §1.1.5, macaw_data#149), read from its storage form alone, nothing decoded: a
chunk's column is one form, so a run's kind is its chunk's. DateTime for UTC and zoned
storage, whose instants order alike; NaiveDateTime; Date; :mixed for the general
form, which holds times of more than one kind. A column with no value present packs as
DateTime, and is that. nil for a column the chunk does not hold, or one that is not
:time, which holds no kind of time.
@spec values(t(), term(), non_neg_integer(), non_neg_integer()) :: [term()]
The values of column name in the run of count rows from first, as read through
the chunk's delta: what a step decides from (spec/04 §1.4). One column is read, never a
row.
The chunk's version: its content digest, else its derivation digest (spec/01 §1.1.3).
A cache's key for the chunk's results is {id(chunk), version(chunk), step key}. nil
while the digest is pending (spec/01 §1.1.4): it matches no key.