Macaw.Data.Table (macaw_data v0.13.0)
View SourceColumnar typed data: the currency of deliveries, derived sources and components (spec/01 §1.1).
A table is named columns of declared types, every column the same length:
| Type | Holds | Stored as (Arrow's form) |
|---|---|---|
:number | integers and floats | int64 when every value is an integer that fits; float64 when every one is a float; both, float64 and a bitmap of the integers (macaw.integral); both with an integer beyond ±2^53, a dense union of int64 and float64; an integer beyond int64, a list |
:time | DateTimes, NaiveDateTimes or Dates, one kind a column | all UTC, timestamp[us, UTC]; other zones, UTC microseconds and a dictionary of whole zones (macaw.zoned); naive, timestamp[us]; dates, date32; the precision kept; kinds mixed, a list |
:text | any value | strings, utf8: int32 offsets and one data buffer (Arrow binary when one is not valid UTF-8); anything else, a list |
:category | any value, an atom held as its string | strings and atoms, one dictionary of strings, any booleans beside it in side bitmaps; booleans alone, a bitmap; mixed types, a list |
:color | color terms (Macaw.Data.Schema.color?/1) | a list |
:point, :term | any value; a point is {longitude, latitude} | a list |
Packing never changes a value: every column reads back values == to those given,
but that a category's atom reads back as its string, in every form. A missing value is nil, in
every type. A packed column keeps a validity bitmap beside its buffers, least-significant
bit first as Arrow numbers bits, so a gap stays a gap: absence is never zero (spec/01 §3).
A table is an ordered list of chunks (spec/01 §1.1.1): each a run of rows of every
column, each column packed by itself, so a form and a dictionary are a chunk's. new/1
and from/2 build canonical chunks of chunk_rows/0 rows; append/2, trim/2 and
slice/3 keep whole chunks and where the table's rows start and end in them, copying
nothing; compact/1 rewrites any table into canonical chunks.
A table's identity is its contents (spec/01 §1.1.2): equal?/2 and digest/1 read
the values, never the layout, so the same rows chunked two ways are one value. t() is
opaque, and == on the struct is not a table's equality.
The packed columns live off the process heap. A binary over 64 bytes is reference-counted, so a garbage collection never copies it: a table costs its process a constant number of words a chunk, never a row.
Row lists are accepted at the edge. from/2 reads every shape data arrives in (a
list of row maps, a map or keyword list of columns, an Nx.Tensor, anything implementing
Table.Reader) into a table, and rows/1 is the row view for code that wants rows. The
row helpers (get/2, key/2, accessor/1, temporal?/1, time_columns/1,
column_types/1) read any source, as visualize's Visualize.Data.Table does; this
module's row readers are its port.
The table package and Nx are optional: without them lists and column maps still work,
and a struct source or a tensor raises UndefinedFunctionError where it is read.
Summary
Types
A column name, as the source spells it.
A source's table's origin (spec/01 §1.1.3): the source, the resolution its relation is
held at, its window field and its chunk span (each nil for row-count alignment). Given
to new/2, it may also say where the rows start among the source's (offset:, 0).
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
A column's declared type.
A chunk the table reads, with the run of its rows it reads: the first, and how many.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
The table with other's rows after its own (spec/01 §1.1.1, §1.1.3). The two have the
same names in the same order, each of the same type; another table is
{:error, {:columns, expected, given}}, each a list of {name, type}.
One increment's changes applied to the table (spec/01 §1.1.4): {table, faults, unheld}. change is %{insert, retract, replace, schema, settled, thinned}: the rows
inserted, the rows retracted and the range replacements (%{from, to, identities, rows}) of one increment, already judged for lateness, overlaps and conflicts
(spec/01 §4.2, §4.4); the source's schema (its identity and window field); the
complete_until in force; and whether the holding is thinned.
The chunks the table reads, in order, each with the run of its rows the table reads:
{chunk, first, count} (spec/01 §1.1.1). Every chunk the table references is in it, and
nothing else.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. Every value is == to the one given, integer or float, a time
in its own zone at its own precision, but that a category's atom reads back as the
string that spells it.
Each column's type, by name.
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all color terms :color (a literal or a typed
color, Macaw.Data.Schema.color?/1), all atoms or booleans :category. A string is
never a color, so a column of CSS text is :text. A mixed or empty column is left out.
A suggestion for a tool, never something a reader relies on; a table's own declared
types are column_type/2's, and a table answers those. {:ok, types}, or source/1's
{:error, faults} for a source of no shape rows/1 reads (macaw_data#161).
As column_types/1, the types alone, raising Macaw.Data.Error with the source's faults.
The table in canonical chunks, each whole, packed from its own values and hashed
(spec/01 §1.1.1): chunk_rows/0 rows each from the first, the last shorter, for a table
built from rows or of steps' outputs; a source's table on its origin's slots
(spec/01 §1.1.3). Two equal tables of one origin compact to the same struct; a table
already canonical is returned as it is.
The table's content digest (spec/01 §1.1.2): 64 lowercase hex characters, a SHA-256 over its canonical form, each value a tag, a word and bytes of its own, whatever form holds it. Over the values, never the buffers: a chunking, a column's form and a dictionary's order all keep it, and the origin is not part of it.
Whether two tables have the same contents (spec/01 §1.1.2): the same names in order, each
of the same type, and each column's values in order, compared strictly, as the digest
tells them apart (1 and 1.0 differ, and 0.0 and -0.0). How the rows are chunked
never matters. Exactly when digest/1 of the two is equal.
A table of any source rows/1 reads.
As from/2, raising Macaw.Data.Error naming every fault.
A table of chunks already built (spec/01 §1.1.3): a step's outputs (Chunk.derive/3,
Chunk.gather/5, Chunk.combine/3), in order, under header, a list of
{name, type}. Each is a chunk, read whole, or a view {chunk, first, count} reading a
run of it, as chunks/1 lists them, so a per-row step's output keeps the chunks it
passes through (spec/04 §1.4). A chunk or a run of no rows is left out. A chunk of other
columns is {:error, {:columns, header, names}}, its column names. Nothing is hashed.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
A tensor's columns, or tuple rows' elements, under names, in order.
The column names, in order.
A table of columns, each {name, type, values}, in order.
As new/2, raising Macaw.Data.Error naming every fault.
A source's table's origin (spec/01 §1.1.3), or nil for a table built from rows.
Row i (from 0) as a map, or nil past the end.
The rows of a table or of any source: {:ok, rows}, or source/1's {:error, faults}
for a source of no shape it reads (spec/01 §1.1, spec/00 §5). A table's rows are one map
per row keyed by every column name, nil where missing, and a table always answers
them. A list of rows is itself, uncopied, its tuple rows unchanged; a column map, a
tensor or a Table.Reader source is read one row at a time.
As rows/1, the rows alone, raising Macaw.Data.Error with the source's faults.
Whether chunk of the table is settled under complete_until (spec/01 §1.1.4): its
span ends at or before it. Never for a chunk of rows with no window value, and never in
a table that is not a source's in event time.
The number of rows.
The rows start to start + count - 1, as a table of the same columns. It keeps the
chunks the run spans, cut at both ends, and copies nothing (spec/01 §1.1.1).
The column the table's rows are known to be in ascending order of, nil last, or nil
for none (spec/01 §1.1.3): a source's table in event time is in its window field's order,
and a per-row step's output keeps its input's order. A step binary-searches only this
column (spec/04 §4.3).
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read. {:ok, names}, or
source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161);
a table's are its :time columns.
As time_columns/1, the names alone, raising Macaw.Data.Error with the source's faults.
The table without its oldest n rows: every chunk wholly before the cut dropped, the
first that remains read from the cut (spec/01 §1.1.1). A dropped chunk is referenced by
nothing in the table, so it is freed once nothing else references it.
The column types, in the order spec/01 §1.1 lists them.
Whether a source's rows have no names: a tensor, or tuple rows (a list whose first
row is a tuple and which is not a keyword list of columns). named/2 gives them some.
The table with every cold chunk it holds fetched once (spec/01 §1.1.4): its base in memory, and its size and content digest, pending under a predicate delete, computed as read. Every read of a table whose size is pending warms it first.
Types
A column name, as the source spells it.
@type origin() :: %{ source: term(), resolution: term(), window: name() | nil, span: Duration.t() | number() | nil }
A source's table's origin (spec/01 §1.1.3): the source, the resolution its relation is
held at, its window field and its chunk span (each nil for row-count alignment). Given
to new/2, it may also say where the rows start among the source's (offset:, 0).
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
@opaque t()
A table: equal to another by its contents (equal?/2), never by ==.
@type type() :: :number | :time | :category | :text | :color | :point | :term
A column's declared type.
@type view() :: {Macaw.Data.Table.Chunk.t(), non_neg_integer(), pos_integer()}
A chunk the table reads, with the run of its rows it reads: the first, and how many.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
@spec append(t(), t()) :: {:ok, t()} | {:error, {:columns, list(), list()} | {:origin, term(), term()} | {:unsorted, non_neg_integer()}}
The table with other's rows after its own (spec/01 §1.1.1, §1.1.3). The two have the
same names in the same order, each of the same type; another table is
{:error, {:columns, expected, given}}, each a list of {name, type}.
A table built from rows takes other's chunks as they are, copying nothing. A source's
table keeps its alignment: the rows that fall in its last chunk's slot are written into
it, a chunk of the same id with new contents, and later rows start new chunks. other
is of the same origin, else {:error, {:origin, expected, given}}, and its rows lie at
or after the table's last row in the window field's order, else
{:error, {:unsorted, i}}, i the row of other's out of order.
@spec apply(t(), map()) :: {t(), list(), non_neg_integer()}
One increment's changes applied to the table (spec/01 §1.1.4): {table, faults, unheld}. change is %{insert, retract, replace, schema, settled, thinned}: the rows
inserted, the rows retracted and the range replacements (%{from, to, identities, rows}) of one increment, already judged for lateness, overlaps and conflicts
(spec/01 §4.2, §4.4); the source's schema (its identity and window field); the
complete_until in force; and whether the holding is thinned.
Row retractions, then range retractions, then insertions, each where its slot is: an
open chunk is written again with its new contents, a settled chunk's change lands in its
delta. A retraction that is not held, held only at another window value, or that can't
tell rows apart is {:retract, reason, row} in faults, or counted in unheld against
a thinned holding. Rows to insert that don't pack under the table's columns are
{:table, faults}, and nothing is applied. A settled chunk whose delta reaches a
quarter of its base is compacted.
@spec chunk_rows() :: pos_integer()
The rows of a canonical chunk, 8,192: new/1, from/2 and compact/1 build chunks of
this many rows from the first, the last shorter (spec/01 §1.1.1). Not a setting.
The chunks the table reads, in order, each with the run of its rows the table reads:
{chunk, first, count} (spec/01 §1.1.1). Every chunk the table references is in it, and
nothing else.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. Every value is == to the one given, integer or float, a time
in its own zone at its own precision, but that a category's atom reads back as the
string that spells it.
Each column's type, by name.
@spec column_types(source() | t()) :: {:ok, %{required(name()) => type()}} | {:error, [{list(), term()}, ...]}
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all color terms :color (a literal or a typed
color, Macaw.Data.Schema.color?/1), all atoms or booleans :category. A string is
never a color, so a column of CSS text is :text. A mixed or empty column is left out.
A suggestion for a tool, never something a reader relies on; a table's own declared
types are column_type/2's, and a table answers those. {:ok, types}, or source/1's
{:error, faults} for a source of no shape rows/1 reads (macaw_data#161).
iex> rows = [%{t: ~U[2024-01-01 00:00:00Z], v: 1, k: :a, s: "x", c: {:color, "good"}, m: 1},
...> %{t: ~U[2024-01-02 00:00:00Z], v: 2.5, k: :b, s: "y", c: nil, m: "two"}]
iex> Macaw.Data.Table.column_types(rows)
{:ok, %{t: :time, v: :number, k: :category, s: :text, c: :color}}
iex> Macaw.Data.Table.column_types([%{t: 1}, 2])
{:error, [{[], {:type, :rows}}]}
As column_types/1, the types alone, raising Macaw.Data.Error with the source's faults.
The table in canonical chunks, each whole, packed from its own values and hashed
(spec/01 §1.1.1): chunk_rows/0 rows each from the first, the last shorter, for a table
built from rows or of steps' outputs; a source's table on its origin's slots
(spec/01 §1.1.3). Two equal tables of one origin compact to the same struct; a table
already canonical is returned as it is.
The table's content digest (spec/01 §1.1.2): 64 lowercase hex characters, a SHA-256 over its canonical form, each value a tag, a word and bytes of its own, whatever form holds it. Over the values, never the buffers: a chunking, a column's form and a dictionary's order all keep it, and the origin is not part of it.
Whether two tables have the same contents (spec/01 §1.1.2): the same names in order, each
of the same type, and each column's values in order, compared strictly, as the digest
tells them apart (1 and 1.0 differ, and 0.0 and -0.0). How the rows are chunked
never matters. Exactly when digest/1 of the two is equal.
A table of any source rows/1 reads.
Columns are named by every key of every row map, in term order, a row without the key
missing there. Each column's type is types:'s (a map from name to type) when given,
else the one column_types/1 infers, else :term. A time column's values become
DateTimes here, at the edge: a Date at midnight UTC, a NaiveDateTime as UTC, an
ISO 8601 string parsed. A source whose rows are tuples has
no names; named/2 gives it some first. {:ok, table}, source/1's faults for a source
of no shape it reads, {[:types], {:type, :map}} for types: that is no map, an option
it does not take {[key], :unknown_key}, or new/1's faults.
As from/2, raising Macaw.Data.Error naming every fault.
@spec from_chunks( [{name(), type()}], [Macaw.Data.Table.Chunk.t() | view()], keyword() ) :: {:ok, t()} | {:error, {:columns, list(), list()}}
A table of chunks already built (spec/01 §1.1.3): a step's outputs (Chunk.derive/3,
Chunk.gather/5, Chunk.combine/3), in order, under header, a list of
{name, type}. Each is a chunk, read whole, or a view {chunk, first, count} reading a
run of it, as chunks/1 lists them, so a per-row step's output keeps the chunks it
passes through (spec/04 §1.4). A chunk or a run of no rows is left out. A chunk of other
columns is {:error, {:columns, header, names}}, its column names. Nothing is hashed.
sorted: names a column the rows are known to be in ascending order of, nil last
(sorted/1): a per-row step's output keeps its input's.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
iex> Macaw.Data.Table.key(%{cpu: 1}, "cpu")
:cpu
iex> Macaw.Data.Table.key(%{"cpu" => 1}, :cpu)
"cpu"
iex> Macaw.Data.Table.key(%{cpu: 1}, "mem")
nil
@spec named(source(), [name()]) :: {:ok, source()} | {:error, {:width, non_neg_integer(), non_neg_integer()} | {:rank, non_neg_integer()}}
A tensor's columns, or tuple rows' elements, under names, in order.
Column i of a rank-2 tensor is Enum.at(names, i), and a rank-1 tensor is one column.
A tensor whose width is not length(names) is {:error, {:width, length(names), width}},
one of another rank {:error, {:rank, rank}}. Tuple rows are named row by row, element
i under Enum.at(names, i); a tuple of another size is {:error, {:width, n, size}}.
Any other source is {:ok, source}, unchanged.
iex> Macaw.Data.Table.named([{1, 4.0}, {2, 5.0}], ["x", "y"])
{:ok, [%{"x" => 1, "y" => 4.0}, %{"x" => 2, "y" => 5.0}]}
iex> Macaw.Data.Table.named([{1, 4.0, 7}], ["x", "y"])
{:error, {:width, 2, 3}}
The column names, in order.
@spec new( [{name(), type(), [term()]}], keyword() ) :: {:ok, t()} | {:error, [{list(), term()}, ...]}
A table of columns, each {name, type, values}, in order.
{:ok, table}, or every fault: {[name], {:enum, types}} for an unknown type,
{[name, i], {:type, type}} for a value its type does not hold, {[name, i], :not_data}
for a value that is not data (a function, a PID, a reference, a port, an improper list:
it has no digest), {[name], {:length, n}} for a column whose length is not the first
column's n, {[name], :duplicate} for a name given twice, {[], {:type, :columns}}
for an improper list of columns and {[:opts], {:type, :keyword}} for improper options.
A number column holds
numbers; a time column DateTimes, NaiveDateTimes or Dates; a color column color
terms (Macaw.Data.Schema.color?/1); a point column {longitude, latitude} pairs of
numbers; the others any value. nil is missing in every type.
Its chunks are canonical (spec/01 §1.1.1), their ids their content digests. With
origin: it is a source's table (spec/01 §1.1.3), its chunks laid on the origin's slots
and named {source, resolution, slot}; an origin's own faults are {[:origin, key], reason}, and rows out of the window field's order {[:origin, :window], {:unsorted, i}}.
As new/2, raising Macaw.Data.Error naming every fault.
A source's table's origin (spec/01 §1.1.3), or nil for a table built from rows.
@spec row(t(), non_neg_integer()) :: map() | nil
Row i (from 0) as a map, or nil past the end.
The rows of a table or of any source: {:ok, rows}, or source/1's {:error, faults}
for a source of no shape it reads (spec/01 §1.1, spec/00 §5). A table's rows are one map
per row keyed by every column name, nil where missing, and a table always answers
them. A list of rows is itself, uncopied, its tuple rows unchanged; a column map, a
tensor or a Table.Reader source is read one row at a time.
iex> Macaw.Data.Table.rows(%{t: [1, 2], v: [3, 4]})
{:ok, [%{t: 1, v: 3}, %{t: 2, v: 4}]}
iex> Macaw.Data.Table.rows([%{t: 1}, 2])
{:error, [{[], {:type, :rows}}]}
As rows/1, the rows alone, raising Macaw.Data.Error with the source's faults.
@spec settled?(t(), Macaw.Data.Table.Chunk.t(), term()) :: boolean()
Whether chunk of the table is settled under complete_until (spec/01 §1.1.4): its
span ends at or before it. Never for a chunk of rows with no window value, and never in
a table that is not a source's in event time.
@spec size(t()) :: non_neg_integer()
The number of rows.
@spec slice(t(), non_neg_integer(), non_neg_integer()) :: t()
The rows start to start + count - 1, as a table of the same columns. It keeps the
chunks the run spans, cut at both ends, and copies nothing (spec/01 §1.1.1).
The column the table's rows are known to be in ascending order of, nil last, or nil
for none (spec/01 §1.1.3): a source's table in event time is in its window field's order,
and a per-row step's output keeps its input's order. A step binary-searches only this
column (spec/04 §4.3).
Whether source is of a shape from/2 and rows/1 read (spec/01 §1.1, spec/00 §5):
:ok, or every fault, {path, reason}. rows/1 and from/2 answer these faults for a
source this refuses.
a list of row maps, of tuple rows, or a keyword list of columns; else
{[], {:type, :rows}};a map of columns, each a list (or a rank-1 tensor) of the first column's length:
{[name], {:length, n}}for another length,{[name], {:type, :column}}for a column that is no list;a tensor of rank 1 or 2:
{[], {:rank, rank}}otherwise;a table, a delivery of such rows, or a
Table.Readersource; any other term{[], {:type, :source}}.iex> Macaw.Data.Table.source(%{t: [1, 2], v: [3, 4]}) :ok iex> Macaw.Data.Table.source([%{t: 1}, 2]) {:error, [{[], {:type, :rows}}]}
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read. {:ok, names}, or
source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161);
a table's are its :time columns.
iex> Macaw.Data.Table.time_columns(%{t: [~D[2024-01-01], nil], v: [1, 2]})
{:ok, [:t]}
iex> Macaw.Data.Table.time_columns([%{t: ~D[2024-01-01]}, 2])
{:error, [{[], {:type, :rows}}]}
As time_columns/1, the names alone, raising Macaw.Data.Error with the source's faults.
@spec trim(t(), non_neg_integer()) :: t()
The table without its oldest n rows: every chunk wholly before the cut dropped, the
first that remains read from the cut (spec/01 §1.1.1). A dropped chunk is referenced by
nothing in the table, so it is freed once nothing else references it.
@spec types() :: [type(), ...]
The column types, in the order spec/01 §1.1 lists them.
Whether a source's rows have no names: a tensor, or tuple rows (a list whose first
row is a tuple and which is not a keyword list of columns). named/2 gives them some.
iex> Macaw.Data.Table.unnamed?([{1, 4.0}])
true
iex> Macaw.Data.Table.unnamed?([x: [1, 2]])
false
The table with every cold chunk it holds fetched once (spec/01 §1.1.4): its base in memory, and its size and content digest, pending under a predicate delete, computed as read. Every read of a table whose size is pending warms it first.