Columnar typed data: the currency of deliveries, derived sources and components (spec/01 §1.1).
A table is named columns of declared types, every column the same length:
| Type | Holds | Stored as (Arrow's form) |
|---|---|---|
:number | integers and floats | int64 when every value is an integer that fits; float64 when every one is a float; both, float64 and a bitmap of the integers (macaw.integral); both with an integer beyond ±2^53, a dense union of int64 and float64; an integer beyond int64, a list |
:time | DateTimes, NaiveDateTimes or Dates, one kind a column | all UTC, timestamp[us, UTC]; other zones, UTC microseconds and a dictionary of whole zones (macaw.zoned); naive, timestamp[us]; dates, date32; the precision kept; kinds mixed, a list |
:text | any value | strings, utf8: int32 offsets and one data buffer (Arrow binary when one is not valid UTF-8); anything else, a list |
:category | any value, an atom held as its string | strings and atoms, one dictionary of strings, any booleans beside it in side bitmaps; booleans alone, a bitmap; mixed types, a list |
:color | color terms (Macaw.Data.Schema.color?/1) | a list |
:point, :term | any value; a point is {longitude, latitude} | a list |
Packing never changes a value: every column reads back values == to those given,
but that a category's atom reads back as its string, in every form. A missing value is nil, in
every type. A packed column keeps a validity bitmap beside its buffers, least-significant
bit first as Arrow numbers bits, so a gap stays a gap: absence is never zero (spec/01 §3).
The packed columns live off the process heap. A binary over 64 bytes is reference-counted, so a garbage collection never copies it: a table of a million numbers costs its process a constant number of words, whatever its length.
Row lists are accepted at the edge. from/2 reads every shape data arrives in (a
list of row maps, a map or keyword list of columns, an Nx.Tensor, anything implementing
Table.Reader) into a table, and rows/1 is the row view for code that wants rows. The
row helpers (get/2, key/2, accessor/1, temporal?/1, time_columns/1,
column_types/1) read any source, as visualize's Visualize.Data.Table does; this
module's row readers are its port.
The table package and Nx are optional: without them lists and column maps still work,
and a struct source or a tensor raises UndefinedFunctionError where it is read.
Summary
Types
A column name, as the source spells it.
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
A table.
A column's declared type.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. Every value is == to the one given, integer or float, a time
in its own zone at its own precision, but that a category's atom reads back as the
string that spells it.
Each column's type, by name.
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all color terms :color (a literal or a typed
color, Macaw.Data.Schema.color?/1), all atoms or booleans :category. A string is
never a color, so a column of CSS text is :text. A mixed or empty column is left out.
A suggestion for a tool, never something a reader relies on; a table's own declared
types are column_type/2's.
A table of any source rows/1 reads.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
A tensor's columns, or tuple rows' elements, under names, in order.
The column names, in order.
A table of columns, each {name, type, values}, in order.
As new/1, raising ArgumentError naming every fault.
Row i (from 0) as a map, or nil past the end.
The rows, one map per row keyed by every column name, nil where missing.
The number of rows.
The rows start to start + count - 1, as a table of the same columns: a packed
column's binary is sliced, not copied.
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read.
The column types, in the order spec/01 §1.1 lists them.
Whether a source's rows have no names: a tensor, or tuple rows (a list whose first
row is a tuple and which is not a keyword list of columns). named/2 gives them some.
Types
A column name, as the source spells it.
A row: a map keyed by column name, or a tuple in column order.
Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of
equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing
Table.Reader.
@type t() :: %Macaw.Data.Table{ columns: %{required(name()) => {type(), tuple()}}, names: [name()], size: non_neg_integer() }
A table.
@type type() :: :number | :time | :category | :text | :color | :point | :term
A column's declared type.
Functions
The accessor fn row -> get(row, name) end; a function is returned unchanged.
The values of column name, in order, nil where a value is missing; nil for a name
the table does not have. Every value is == to the one given, integer or float, a time
in its own zone at its own precision, but that a category's atom reads back as the
string that spells it.
Each column's type, by name.
@spec column_types(source() | t()) :: %{ required(name()) => :time | :number | :category | :text | :color }
Every named column's type, read whole: all non-nil values temporal is :time, all
numbers :number, all strings :text, all color terms :color (a literal or a typed
color, Macaw.Data.Schema.color?/1), all atoms or booleans :category. A string is
never a color, so a column of CSS text is :text. A mixed or empty column is left out.
A suggestion for a tool, never something a reader relies on; a table's own declared
types are column_type/2's.
iex> rows = [%{t: ~U[2024-01-01 00:00:00Z], v: 1, k: :a, s: "x", c: {:color, "good"}, m: 1},
...> %{t: ~U[2024-01-02 00:00:00Z], v: 2.5, k: :b, s: "y", c: nil, m: "two"}]
iex> Macaw.Data.Table.column_types(rows)
%{t: :time, v: :number, k: :category, s: :text, c: :color}A table of any source rows/1 reads.
Columns are named by every key of every row map, in term order, a row without the key
missing there. Each column's type is types:'s (a map from name to type) when given,
else the one column_types/1 infers, else :term. A time column's values become
DateTimes here, at the edge: a Date at midnight UTC, a NaiveDateTime as UTC, an
ISO 8601 string parsed. A source whose rows are tuples has
no names; named/2 gives it some first. {:ok, table} or new/1's faults.
The value of column name in row, by either spelling of the name: the key itself, else
an atom name as its string and a string name as the atom key spelled the same. nil
when neither is a key. No atom is ever made.
The key of row that spells name: name itself when it is a key, else the key of the
other spelling, nil when neither is. The atom key a string finds is remembered for the
calling process, so reading one field from every row scans one row's keys. No atom is
ever made or looked up by name: the remembered key is one a row held.
iex> Macaw.Data.Table.key(%{cpu: 1}, "cpu")
:cpu
iex> Macaw.Data.Table.key(%{"cpu" => 1}, :cpu)
"cpu"
iex> Macaw.Data.Table.key(%{cpu: 1}, "mem")
nil@spec named(source(), [name()]) :: {:ok, source()} | {:error, {:width, non_neg_integer(), non_neg_integer()} | {:rank, non_neg_integer()}}
A tensor's columns, or tuple rows' elements, under names, in order.
Column i of a rank-2 tensor is Enum.at(names, i), and a rank-1 tensor is one column.
A tensor whose width is not length(names) is {:error, {:width, length(names), width}},
one of another rank {:error, {:rank, rank}}. Tuple rows are named row by row, element
i under Enum.at(names, i); a tuple of another size is {:error, {:width, n, size}}.
Any other source is {:ok, source}, unchanged.
iex> Macaw.Data.Table.named([{1, 4.0}, {2, 5.0}], ["x", "y"])
{:ok, [%{"x" => 1, "y" => 4.0}, %{"x" => 2, "y" => 5.0}]}
iex> Macaw.Data.Table.named([{1, 4.0, 7}], ["x", "y"])
{:error, {:width, 2, 3}}The column names, in order.
A table of columns, each {name, type, values}, in order.
{:ok, table}, or every fault: {[name], {:enum, types}} for an unknown type,
{[name, i], {:type, type}} for a value its type does not hold, {[name], {:length, n}}
for a column whose length is not the first column's n, {[name], :duplicate} for a
name given twice. A number column holds numbers; a time column DateTimes,
NaiveDateTimes or Dates; a color column color terms (Macaw.Data.Schema.color?/1);
a point column {longitude, latitude} pairs of numbers; the others any value. nil is
missing in every type.
As new/1, raising ArgumentError naming every fault.
@spec row(t(), non_neg_integer()) :: map() | nil
Row i (from 0) as a map, or nil past the end.
The rows, one map per row keyed by every column name, nil where missing.
@spec size(t()) :: non_neg_integer()
The number of rows.
@spec slice(t(), non_neg_integer(), non_neg_integer()) :: t()
The rows start to start + count - 1, as a table of the same columns: a packed
column's binary is sliced, not copied.
Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string
in ISO 8601 form that one of those parses.
The names of the columns whose values are all temporal: at least one value not nil,
and every value nil or temporal?/1. The whole column is read.
@spec types() :: [type(), ...]
The column types, in the order spec/01 §1.1 lists them.
Whether a source's rows have no names: a tensor, or tuple rows (a list whose first
row is a tuple and which is not a keyword list of columns). named/2 gives them some.
iex> Macaw.Data.Table.unnamed?([{1, 4.0}])
true
iex> Macaw.Data.Table.unnamed?([x: [1, 2]])
false