Macaw.Data.Table (macaw_data v0.18.0)

View Source

Columnar typed data: the currency of deliveries, derived sources and components (spec/01 §1.1).

A table is named columns of declared types, every column the same length:

TypeHoldsStored as (Arrow's form)
:numberintegers and floatsint64 when every value is an integer that fits; float64 when every one is a float; both, float64 and a bitmap of the integers (macaw.integral); both with an integer beyond ±2^53, a dense union of int64 and float64; an integer beyond int64, a list
:timeDateTimes, NaiveDateTimes or Dates, one kind a columnall UTC, timestamp[us, UTC]; other zones, UTC microseconds and a dictionary of whole zones (macaw.zoned); naive, timestamp[us]; dates, date32; the precision kept; kinds mixed, a list
:textany valuestrings, utf8: int32 offsets and one data buffer (Arrow binary when one is not valid UTF-8); anything else, a list
:categoryany value, an atom held as its stringstrings and atoms, one dictionary of strings, any booleans beside it in side bitmaps; booleans alone, a bitmap; mixed types, a list
:colorcolor terms (Macaw.Data.Schema.color?/1)a list
:point, :termany value; a point is {longitude, latitude}a list

Packing never changes a value: every column reads back values == to those given, but that a category's atom reads back as its string, in every form. A missing value is nil, in every type. A packed column keeps a validity bitmap beside its buffers, least-significant bit first as Arrow numbers bits, so a gap stays a gap: absence is never zero (spec/01 §3).

A table is an ordered list of chunks (spec/01 §1.1.1): each a run of rows of every column, each column packed by itself, so a form and a dictionary are a chunk's. new/1 and from/2 build canonical chunks of chunk_rows/0 rows; append/2, trim/2 and slice/3 keep whole chunks and where the table's rows start and end in them, copying nothing; compact/1 rewrites any table into canonical chunks.

A table's identity is its contents (spec/01 §1.1.2): equal?/2 and digest/1 read the values, never the layout, so the same rows chunked two ways are one value. t() is opaque, and == on the struct is not a table's equality. The kernel's digest (Macaw.Layers.Document.digest/1) of a term holding a table follows equal?/2 too: a table's canonical term (Macaw.Layers.Digest) is its contents'.

The packed columns live off the process heap. A binary over 64 bytes is reference-counted, so a garbage collection never copies it: a table costs its process a constant number of words a chunk, never a row.

Row lists are accepted at the edge. from/2 reads every shape data arrives in (a list of row maps, a map or keyword list of columns, an Nx.Tensor, anything implementing Table.Reader) into a table, and rows/1 is the row view for code that wants rows. The row helpers (get/2, key/2, accessor/1, temporal?/1, time_columns/1, column_types/1) read any source, as visualize's Visualize.Data.Table does; this module's row readers are its port.

The table package and Nx are optional: without them lists and column maps still work, and a struct source or a tensor raises UndefinedFunctionError where it is read.

Summary

Types

A column name, as the source spells it.

A source's table's origin (spec/01 §1.1.3): the source, the resolution its relation is held at, its window field and its chunk span (each nil for row-count alignment). Given to new/2, it may also say where the rows start among the source's (offset:, 0).

A row: a map keyed by column name, or a tuple in column order.

Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing Table.Reader.

t()

A table: equal to another by its contents (equal?/2), never by ==.

A column's declared type.

A chunk the table reads, with the run of its rows it reads: the first, and how many.

Functions

The accessor fn row -> get(row, name) end; a function is returned unchanged.

The table with other's rows after its own (spec/01 §1.1.1, §1.1.3). The two have the same names in the same order, each of the same type; another table is {:error, {:columns, expected, given}}, each a list of {name, type}.

One increment's changes applied to the table (spec/01 §1.1.4): {table, faults, unheld}. change is %{insert, retract, replace, schema, settled, thinned}: the rows inserted, the rows retracted and the range replacements (%{from, to, identities, rows}) of one increment, already judged for lateness, overlaps and conflicts (spec/01 §4.2, §4.4); the source's schema (its identity and window field); the complete_until in force; and whether the holding is thinned.

The rows of a canonical chunk, 8,192: new/1, from/2 and compact/1 build chunks of this many rows from the first, the last shorter (spec/01 §1.1.1). Not a setting.

The chunks the table reads, in order, each with the run of its rows the table reads: {chunk, first, count} (spec/01 §1.1.1). Every chunk the table references is in it, and nothing else.

The values of column name, in order, nil where a value is missing; nil for a name the table does not have. Every value is == to the one given, integer or float, a time in its own zone at its own precision, but that a category's atom reads back as the string that spells it.

Each column's type, by name.

Every named column's type, read whole: all non-nil values temporal is :time, all numbers :number, all strings :text, all color terms :color (a literal or a typed color, Macaw.Data.Schema.color?/1), all {x, y} pairs of numbers :point (the tuple, any range, as a :point column packs; a list [x, y] is not one, macaw_data#298), all atoms or booleans :category. A string is never a color, so a column of CSS text is :text. A mixed or empty column is left out. A suggestion for a tool, never something a reader relies on; a table's own declared types are column_type/2's, and a table answers those. {:ok, types}, or source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161).

As column_types/1, the types alone, raising Macaw.Data.Error with the source's faults.

The table in canonical chunks, each whole, packed from its own values and hashed (spec/01 §1.1.1): chunk_rows/0 rows each from the first, the last shorter, for a table built from rows or of steps' outputs; a source's table on its origin's slots (spec/01 §1.1.3). Two equal tables of one origin compact to the same struct; a table already canonical is returned as it is.

The table's content digest (spec/01 §1.1.2): 64 lowercase hex characters, a SHA-256 over its canonical form, each value a tag, a word and bytes of its own, whatever form holds it. Over the values, never the buffers: a chunking, a column's form and a dictionary's order all keep it, and the origin is not part of it.

Whether two tables have the same contents (spec/01 §1.1.2): the same names in order, each of the same type, and each column's values in order, compared strictly, as the digest tells them apart (1 and 1.0 differ, and 0.0 and -0.0). How the rows are chunked never matters. Exactly when digest/1 of the two is equal.

A table of any source rows/1 reads.

As from/2, raising Macaw.Data.Error naming every fault.

A table of chunks already built (spec/01 §1.1.3): a step's outputs (Chunk.derive/3, Chunk.gather/5, Chunk.combine/3), in order, under header, a list of {name, type}. Each is a chunk, read whole, or a view {chunk, first, count} reading a run of it, as chunks/1 lists them, so a per-row step's output keeps the chunks it passes through (spec/04 §1.4). A chunk or a run of no rows is left out. A chunk of other columns is {:error, {:columns, header, names}}, its column names. Nothing is hashed.

The value of column name in row, by either spelling of the name: the key itself, else an atom name as its string and a string name as the atom key spelled the same. nil when neither is a key. No atom is ever made. A present key is read with one match, and the other spelling is looked for only on a miss (macaw_data#294): it is the read a frame loop makes for every channel of every item.

The key of row that spells name: name itself when it is a key, else the key of the other spelling, nil when neither is. The atom key a string finds is remembered for the calling process, so reading one field from every row scans one row's keys. No atom is ever made or looked up by name: the remembered key is one a row held.

A tensor's columns, or tuple rows' elements, under names, in order.

The column names, in order.

A table of columns, each {name, type, values}, in order.

As new/2, raising Macaw.Data.Error naming every fault.

A source's table's origin (spec/01 §1.1.3), or nil for a table built from rows.

Row i (from 0) as a map, or nil past the end.

The rows of a table or of any source: {:ok, rows}, or source/1's {:error, faults} for a source of no shape it reads (spec/01 §1.1, spec/00 §5). A table's rows are one map per row keyed by every column name, nil where missing, and a table always answers them. A list of rows is itself, uncopied, its tuple rows unchanged; a column map, a tensor or a Table.Reader source is read one row at a time.

As rows/1, the rows alone, raising Macaw.Data.Error with the source's faults.

The rows at indexes (from 0): {:ok, rows}, in the order given, repeats kept, each row the one row/2 answers, nil for an index past the end (spec/01 §1.1). The rows are gathered once, so a row costs the same wherever it lies in its chunk, even in a :term column, which row/2 reads from its chunk's start (macaw_data#205).

As rows_at/2, the rows alone, raising Macaw.Data.Error with the faults.

Whether chunk of the table is settled under complete_until (spec/01 §1.1.4): its span ends at or before it. Never for a chunk of rows with no window value, and never in a table that is not a source's in event time.

The number of rows.

The rows start to start + count - 1, as a table of the same columns. It keeps the chunks the run spans, cut at both ends, and copies nothing (spec/01 §1.1.1).

The column the table's rows are known to be in ascending order of, nil last, or nil for none (spec/01 §1.1.3): a source's table in event time is in its window field's order, and a per-row step's output keeps its input's order. A step binary-searches only this column (spec/04 §4.3).

Whether source is of a shape from/2 and rows/1 read (spec/01 §1.1, spec/00 §5): :ok, or every fault, {path, reason}. rows/1 and from/2 answer these faults for a source this refuses.

Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string in ISO 8601 form that one of those parses.

The names of the columns whose values are all temporal: at least one value not nil, and every value nil or temporal?/1. The whole column is read. {:ok, names}, or source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161); a table's are its :time columns.

As time_columns/1, the names alone, raising Macaw.Data.Error with the source's faults.

The table without its oldest n rows: every chunk wholly before the cut dropped, the first that remains read from the cut (spec/01 §1.1.1). A dropped chunk is referenced by nothing in the table, so it is freed once nothing else references it.

The column types, in the order spec/01 §1.1 lists them.

Whether a source's rows have no names: a tensor, or tuple rows (a list whose first row is a tuple and which is not a keyword list of columns). named/2 gives them some.

The table with every cold chunk it holds fetched once (spec/01 §1.1.4): its base in memory, and its size and content digest, pending under a predicate delete, computed as read. Every read of a table whose size is pending warms it first.

Types

name()

@type name() :: atom() | String.t()

A column name, as the source spells it.

origin()

@type origin() :: %{
  source: term(),
  resolution: term(),
  window: name() | nil,
  span: Duration.t() | number() | nil
}

A source's table's origin (spec/01 §1.1.3): the source, the resolution its relation is held at, its window field and its chunk span (each nil for row-count alignment). Given to new/2, it may also say where the rows start among the source's (offset:, 0).

row()

@type row() :: map() | tuple()

A row: a map keyed by column name, or a tuple in column order.

source()

@type source() ::
  [row()] | %{required(name()) => [term()]} | [{name(), [term()]}] | struct()

Anything from/2 and rows/1 accept: a list of rows, a keyword list or map of equal-length columns, an Nx.Tensor of rank 1 or 2, or a struct implementing Table.Reader.

t()

@opaque t()

A table: equal to another by its contents (equal?/2), never by ==.

type()

@type type() :: :number | :time | :category | :text | :color | :point | :term

A column's declared type.

view()

A chunk the table reads, with the run of its rows it reads: the first, and how many.

Functions

accessor(fun)

@spec accessor(name() | (row() -> term())) :: (row() -> term())

The accessor fn row -> get(row, name) end; a function is returned unchanged.

append(table, other)

@spec append(t(), t()) ::
  {:ok, t()}
  | {:error,
     {:columns, list(), list()}
     | {:origin, term(), term()}
     | {:unsorted, non_neg_integer()}}

The table with other's rows after its own (spec/01 §1.1.1, §1.1.3). The two have the same names in the same order, each of the same type; another table is {:error, {:columns, expected, given}}, each a list of {name, type}.

A table built from rows takes other's chunks as they are, copying nothing; it stays sorted by its column (sorted/1) only when other's rows are in that column's order at or after its last, other's own rows and the table's last read, else it is sorted by none (macaw_data#261). A source's table keeps its alignment: the rows that fall in its last chunk's slot are written into it, a chunk of the same id with new contents, and later rows start new chunks. other is of the same origin, else {:error, {:origin, expected, given}}, and its rows lie at or after the table's last row in the window field's order, else {:error, {:unsorted, i}}, i the row of other's out of order.

apply(table, change)

@spec apply(t(), map()) :: {t(), list(), non_neg_integer()}

One increment's changes applied to the table (spec/01 §1.1.4): {table, faults, unheld}. change is %{insert, retract, replace, schema, settled, thinned}: the rows inserted, the rows retracted and the range replacements (%{from, to, identities, rows}) of one increment, already judged for lateness, overlaps and conflicts (spec/01 §4.2, §4.4); the source's schema (its identity and window field); the complete_until in force; and whether the holding is thinned.

Row retractions, then range retractions, then insertions, each where its slot is: an open chunk is written again with its new contents, a settled chunk's change lands in its delta. A retraction that is not held, held only at another window value, or that can't tell rows apart is {:retract, reason, row} in faults, or counted in unheld against a thinned holding. Rows to insert that don't pack under the table's columns are {:table, faults}, and nothing is applied. A settled chunk whose delta reaches a quarter of its base is compacted.

A table that is not a source's keeps the column it is sorted by only while its rows are in that column's order (macaw_data#261): when nothing is inserted, when schema's window field is that column, or when the rows inserted after the held ones are in its order from the last held row; else it is sorted by none.

chunk_rows()

@spec chunk_rows() :: pos_integer()

The rows of a canonical chunk, 8,192: new/1, from/2 and compact/1 build chunks of this many rows from the first, the last shorter (spec/01 §1.1.1). Not a setting.

chunks(table)

@spec chunks(t()) :: [view()]

The chunks the table reads, in order, each with the run of its rows the table reads: {chunk, first, count} (spec/01 §1.1.1). Every chunk the table references is in it, and nothing else.

column(table, name)

@spec column(t(), name()) :: [term()] | nil

The values of column name, in order, nil where a value is missing; nil for a name the table does not have. Every value is == to the one given, integer or float, a time in its own zone at its own precision, but that a category's atom reads back as the string that spells it.

column_type(table, name)

@spec column_type(t(), name()) :: type() | nil

Each column's type, by name.

column_types(table)

@spec column_types(source() | t()) ::
  {:ok, %{required(name()) => type()}} | {:error, [{list(), term()}, ...]}

Every named column's type, read whole: all non-nil values temporal is :time, all numbers :number, all strings :text, all color terms :color (a literal or a typed color, Macaw.Data.Schema.color?/1), all {x, y} pairs of numbers :point (the tuple, any range, as a :point column packs; a list [x, y] is not one, macaw_data#298), all atoms or booleans :category. A string is never a color, so a column of CSS text is :text. A mixed or empty column is left out. A suggestion for a tool, never something a reader relies on; a table's own declared types are column_type/2's, and a table answers those. {:ok, types}, or source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161).

iex> rows = [%{t: ~U[2024-01-01 00:00:00Z], v: 1, k: :a, s: "x", c: {:color, "good"}, m: 1},
...>         %{t: ~U[2024-01-02 00:00:00Z], v: 2.5, k: :b, s: "y", c: nil, m: "two"}]
iex> Macaw.Data.Table.column_types(rows)
{:ok, %{t: :time, v: :number, k: :category, s: :text, c: :color}}
iex> Macaw.Data.Table.column_types([%{pos: {12.5, 41.9}}, %{pos: nil}, %{pos: {-0.1, 51.5}}])
{:ok, %{pos: :point}}
iex> Macaw.Data.Table.column_types([%{pos: {12.5, 41.9}}, %{pos: [1, 2]}])
{:ok, %{}}
iex> Macaw.Data.Table.column_types([%{t: 1}, 2])
{:error, [{[], {:type, :rows}}]}

column_types!(source)

@spec column_types!(source() | t()) :: %{required(name()) => type()}

As column_types/1, the types alone, raising Macaw.Data.Error with the source's faults.

compact(table)

@spec compact(t()) :: t()

The table in canonical chunks, each whole, packed from its own values and hashed (spec/01 §1.1.1): chunk_rows/0 rows each from the first, the last shorter, for a table built from rows or of steps' outputs; a source's table on its origin's slots (spec/01 §1.1.3). Two equal tables of one origin compact to the same struct; a table already canonical is returned as it is.

digest(table)

@spec digest(t()) :: String.t()

The table's content digest (spec/01 §1.1.2): 64 lowercase hex characters, a SHA-256 over its canonical form, each value a tag, a word and bytes of its own, whatever form holds it. Over the values, never the buffers: a chunking, a column's form and a dictionary's order all keep it, and the origin is not part of it.

equal?(a, b)

@spec equal?(t(), t()) :: boolean()

Whether two tables have the same contents (spec/01 §1.1.2): the same names in order, each of the same type, and each column's values in order, compared strictly, as the digest tells them apart (1 and 1.0 differ, and 0.0 and -0.0). How the rows are chunked never matters. Exactly when digest/1 of the two is equal.

from(source, opts \\ [])

@spec from(
  source() | t(),
  keyword()
) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

A table of any source rows/1 reads.

Columns are named by every key of every row map, in term order, a row without the key missing there. Each column's type is types:'s (a map from name to type) when given, else the one column_types/1 infers, else :term. A time column's values become DateTimes here, at the edge: a Date at midnight UTC, a NaiveDateTime as UTC, an ISO 8601 string parsed. A source whose rows are tuples has no names; named/2 gives it some first. {:ok, table}, source/1's faults for a source of no shape it reads, {[:types], {:type, :map}} for types: that is no map, an option it does not take {[key], :unknown_key}, or new/1's faults.

from!(source, opts \\ [])

@spec from!(
  source() | t(),
  keyword()
) :: t()

As from/2, raising Macaw.Data.Error naming every fault.

from_chunks(header, chunks, opts \\ [])

@spec from_chunks(
  [{name(), type()}],
  [Macaw.Data.Table.Chunk.t() | view()],
  keyword()
) ::
  {:ok, t()} | {:error, {:columns, list(), list()}}

A table of chunks already built (spec/01 §1.1.3): a step's outputs (Chunk.derive/3, Chunk.gather/5, Chunk.combine/3), in order, under header, a list of {name, type}. Each is a chunk, read whole, or a view {chunk, first, count} reading a run of it, as chunks/1 lists them, so a per-row step's output keeps the chunks it passes through (spec/04 §1.4). A chunk or a run of no rows is left out. A chunk of other columns is {:error, {:columns, header, names}}, its column names. Nothing is hashed.

sorted: names a column the rows are known to be in ascending order of, nil last (sorted/1): a per-row step's output keeps its input's.

get(row, name)

@spec get(map(), name()) :: term()

The value of column name in row, by either spelling of the name: the key itself, else an atom name as its string and a string name as the atom key spelled the same. nil when neither is a key. No atom is ever made. A present key is read with one match, and the other spelling is looked for only on a miss (macaw_data#294): it is the read a frame loop makes for every channel of every item.

key(row, name)

@spec key(map(), name()) :: name() | nil

The key of row that spells name: name itself when it is a key, else the key of the other spelling, nil when neither is. The atom key a string finds is remembered for the calling process, so reading one field from every row scans one row's keys. No atom is ever made or looked up by name: the remembered key is one a row held.

iex> Macaw.Data.Table.key(%{cpu: 1}, "cpu")
:cpu
iex> Macaw.Data.Table.key(%{"cpu" => 1}, :cpu)
"cpu"
iex> Macaw.Data.Table.key(%{cpu: 1}, "mem")
nil

named(source, names)

@spec named(source(), [name()]) ::
  {:ok, source()}
  | {:error,
     {:width, non_neg_integer(), non_neg_integer()} | {:rank, non_neg_integer()}}

A tensor's columns, or tuple rows' elements, under names, in order.

Column i of a rank-2 tensor is Enum.at(names, i), and a rank-1 tensor is one column. A tensor whose width is not length(names) is {:error, {:width, length(names), width}}, one of another rank {:error, {:rank, rank}}. Tuple rows are named row by row, element i under Enum.at(names, i); a tuple of another size is {:error, {:width, n, size}}. Any other source is {:ok, source}, unchanged.

iex> Macaw.Data.Table.named([{1, 4.0}, {2, 5.0}], ["x", "y"])
{:ok, [%{"x" => 1, "y" => 4.0}, %{"x" => 2, "y" => 5.0}]}
iex> Macaw.Data.Table.named([{1, 4.0, 7}], ["x", "y"])
{:error, {:width, 2, 3}}

names(table)

@spec names(t()) :: [name()]

The column names, in order.

new(columns, opts \\ [])

@spec new(
  [{name(), type(), [term()]}],
  keyword()
) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

A table of columns, each {name, type, values}, in order.

{:ok, table}, or every fault: {[name], {:enum, types}} for an unknown type, {[name, i], {:type, type}} for a value its type does not hold, {[name, i], :not_data} for a value that is not data (a function, a PID, a reference, a port, an improper list: it has no digest), {[name], {:length, n}} for a column whose length is not the first column's n, {[name], :duplicate} for a name given twice, {[], {:type, :columns}} for an improper list of columns and {[:opts], {:type, :keyword}} for improper options. A number column holds numbers; a time column DateTimes, NaiveDateTimes or Dates; a color column color terms (Macaw.Data.Schema.color?/1); a point column {longitude, latitude} pairs of numbers; the others any value. nil is missing in every type.

Its chunks are canonical (spec/01 §1.1.1), their ids their content digests. With origin: it is a source's table (spec/01 §1.1.3), its chunks laid on the origin's slots and named {source, resolution, slot}; an origin's own faults are {[:origin, key], reason} (a source or resolution that is not data {[:origin, key], :not_data}), and rows out of the window field's order {[:origin, :window], {:unsorted, i}}.

new!(columns, opts \\ [])

@spec new!(
  [{name(), type(), [term()]}],
  keyword()
) :: t()

As new/2, raising Macaw.Data.Error naming every fault.

origin(table)

@spec origin(t()) :: origin() | nil

A source's table's origin (spec/01 §1.1.3), or nil for a table built from rows.

row(table, i)

@spec row(t(), non_neg_integer()) :: map() | nil

Row i (from 0) as a map, or nil past the end.

rows(table)

@spec rows(source() | t()) :: {:ok, [row()]} | {:error, [{list(), term()}, ...]}

The rows of a table or of any source: {:ok, rows}, or source/1's {:error, faults} for a source of no shape it reads (spec/01 §1.1, spec/00 §5). A table's rows are one map per row keyed by every column name, nil where missing, and a table always answers them. A list of rows is itself, uncopied, its tuple rows unchanged; a column map, a tensor or a Table.Reader source is read one row at a time.

iex> Macaw.Data.Table.rows(%{t: [1, 2], v: [3, 4]})
{:ok, [%{t: 1, v: 3}, %{t: 2, v: 4}]}
iex> Macaw.Data.Table.rows([%{t: 1}, 2])
{:error, [{[], {:type, :rows}}]}

rows!(source)

@spec rows!(source() | t()) :: [row()]

As rows/1, the rows alone, raising Macaw.Data.Error with the source's faults.

rows_at(table, indexes)

@spec rows_at(t(), [non_neg_integer()]) ::
  {:ok, [map() | nil]} | {:error, [{list(), term()}, ...]}

The rows at indexes (from 0): {:ok, rows}, in the order given, repeats kept, each row the one row/2 answers, nil for an index past the end (spec/01 §1.1). The rows are gathered once, so a row costs the same wherever it lies in its chunk, even in a :term column, which row/2 reads from its chunk's start (macaw_data#205).

An index that is no non-negative integer is {[k], {:type, :non_neg_integer}}, k its position, every fault at once; indexes that are no proper list are {[], {:type, :indexes}} (spec/00 §5). rows_at!/2 answers the rows alone or raises.

iex> table = Macaw.Data.Table.new!([{"t", :number, [10, 20, 30]}])
iex> Macaw.Data.Table.rows_at(table, [2, 0, 7])
{:ok, [%{"t" => 30}, %{"t" => 10}, nil]}
iex> Macaw.Data.Table.rows_at(table, [0, -1, :a])
{:error, [{[1], {:type, :non_neg_integer}}, {[2], {:type, :non_neg_integer}}]}

rows_at!(table, indexes)

@spec rows_at!(t(), [non_neg_integer()]) :: [map() | nil]

As rows_at/2, the rows alone, raising Macaw.Data.Error with the faults.

settled?(table, chunk, complete_until)

@spec settled?(t(), Macaw.Data.Table.Chunk.t(), term()) :: boolean()

Whether chunk of the table is settled under complete_until (spec/01 §1.1.4): its span ends at or before it. Never for a chunk of rows with no window value, and never in a table that is not a source's in event time.

size(table)

@spec size(t()) :: non_neg_integer()

The number of rows.

slice(table, start, count)

@spec slice(t(), non_neg_integer(), non_neg_integer()) :: t()

The rows start to start + count - 1, as a table of the same columns. It keeps the chunks the run spans, cut at both ends, and copies nothing (spec/01 §1.1.1).

sorted(table)

@spec sorted(t()) :: name() | nil

The column the table's rows are known to be in ascending order of, nil last, or nil for none (spec/01 §1.1.3): a source's table in event time is in its window field's order, and a per-row step's output keeps its input's order. A step binary-searches only this column (spec/04 §4.3).

source(source)

@spec source(term()) :: :ok | {:error, [{list(), term()}, ...]}

Whether source is of a shape from/2 and rows/1 read (spec/01 §1.1, spec/00 §5): :ok, or every fault, {path, reason}. rows/1 and from/2 answer these faults for a source this refuses.

  • a list of row maps, of tuple rows, or a keyword list of columns; else {[], {:type, :rows}};

  • a map of columns, each a list (or a rank-1 tensor) of the first column's length: {[name], {:length, n}} for another length, {[name], {:type, :column}} for a column that is no list;

  • a tensor of rank 1 or 2: {[], {:rank, rank}} otherwise;

  • a table, a delivery of such rows, or a Table.Reader source; any other term {[], {:type, :source}}.

    iex> Macaw.Data.Table.source(%{t: [1, 2], v: [3, 4]}) :ok iex> Macaw.Data.Table.source([%{t: 1}, 2]) {:error, [{[], {:type, :rows}}]}

temporal?(value)

@spec temporal?(term()) :: boolean()

Whether a value is a point in time: a DateTime, Date or NaiveDateTime, or a string in ISO 8601 form that one of those parses.

time_columns(table)

@spec time_columns(source() | t()) ::
  {:ok, [name()]} | {:error, [{list(), term()}, ...]}

The names of the columns whose values are all temporal: at least one value not nil, and every value nil or temporal?/1. The whole column is read. {:ok, names}, or source/1's {:error, faults} for a source of no shape rows/1 reads (macaw_data#161); a table's are its :time columns.

iex> Macaw.Data.Table.time_columns(%{t: [~D[2024-01-01], nil], v: [1, 2]})
{:ok, [:t]}
iex> Macaw.Data.Table.time_columns([%{t: ~D[2024-01-01]}, 2])
{:error, [{[], {:type, :rows}}]}

time_columns!(source)

@spec time_columns!(source() | t()) :: [name()]

As time_columns/1, the names alone, raising Macaw.Data.Error with the source's faults.

trim(table, n)

@spec trim(t(), non_neg_integer()) :: t()

The table without its oldest n rows: every chunk wholly before the cut dropped, the first that remains read from the cut (spec/01 §1.1.1). A dropped chunk is referenced by nothing in the table, so it is freed once nothing else references it.

types()

@spec types() :: [type(), ...]

The column types, in the order spec/01 §1.1 lists them.

unnamed?(source)

@spec unnamed?(term()) :: boolean()

Whether a source's rows have no names: a tensor, or tuple rows (a list whose first row is a tuple and which is not a keyword list of columns). named/2 gives them some.

iex> Macaw.Data.Table.unnamed?([{1, 4.0}])
true
iex> Macaw.Data.Table.unnamed?([x: [1, 2]])
false

warm(table)

@spec warm(t()) :: t()

The table with every cold chunk it holds fetched once (spec/01 §1.1.4): its base in memory, and its size and content digest, pending under a predicate delete, computed as read. Every read of a table whose size is pending warms it first.