Macaw.Data.Delivery (macaw_data v0.13.0)
View SourceOne answer of a source: its rows, as the source gave them, and what it actually served (spec/01 §3).
A delivery is a handful of terms beside its rows, never per row:
rows: the rows as the source gave them (macaw#185): a list of row maps or tuples, a map or keyword list of columns, anNx.Tensor, aTable.Readersource, or aMacaw.Data.Table. Nothing is packed or rewritten at construction:rows/1reads them as they are, a row list uncopied, andtable/2packs them into a table for a reader that wants columns, normalizing there and only there. A source that has a table delivers it, and it passes as it is. A gap staysnilin either form: absence is never zero.names: the column names of tuple rows or a tensor, in order, for whoever must read them by name (named_rows/1,table/2, finishing);nilotherwise.types: column types fortable/2to pack with,%{}to infer them.schema: the version of the schema the rows were produced under (Macaw.Data.Schema.version/1), never the schema itself, whichMacaw.Data.Schema.Cacheholds by version;nilwhen the source has none.served: what this answer served, which may differ from what was asked:window: a range[from, to], a box over a:pointwindow field%{west: w, south: s, east: e, north: n}, ornil;resolution: the spacing of the rows as served (seconds, or a name such as:minute), ornil;capped: whether the source stopped short of the window it was asked for;stale: whether the source answered from a store it could not refresh, so the answer may be out of date (its upstream failed, and it served its last good answer); a fact of this one answer, ascappedis, not a capability;reduced_by: who brought the rows within budget,:sourceor:runtime, ornilwhen nobody had to;ordered: who put the rows in the request'sorder(spec/01 §2.4)::source, which served its first rows in it;:runtime, which ordered every row the source served and kept the first;:sample, which ordered rows the source had already cut (itsreduced_by: :source, orcapped), so they are in the order but need not be the first in it; ornil, no order asked;params: the parameters it was answered for, a map;note: a line for a person ("coarsened from raw"), ornil;position: for a snapshot taken for a push subscription, the sequence number of the last increment it reflects (0for none);nilfor any other answer (spec/01 §4.1);complete_until: a value of the window field, a time or a number, below which the source expects no further change, ornilfor no claim (spec/01 §4.4).
origin: where it came from,%{source: name, request: digest, as_of: time | nil}, ornil;retractedandreplace: the changes a pushed increment carries beside its inserted rows (spec/01 §4.2), rows withdrawn and range replacements%{from, to, identities, rows}; both[]by default, and a snapshot carries neither.
The shape of visualize's Visualize.Data.Delivery, whose facts are flat and whose
reduced_by is :feed or :visualize, is spec/01 §3's here.
Examples
iex> alias Macaw.Data.Delivery
iex> rows = [%{t: 1, v: 3}, %{t: 2, v: nil}]
iex> {:ok, d} = Delivery.new(rows, served: [window: [0, 10]], schema: "ab12")
iex> Delivery.rows(d) === {:ok, rows}
true
iex> {d.served.window, d.served.capped, d.served.reduced_by}
{[0, 10], false, nil}
iex> d |> Delivery.table!() |> Macaw.Data.Table.column(:v)
[3, nil]
Summary
Types
A fault in a delivery's arguments: where it is, and why (format_error/1).
Where an answer came from.
A range replacement (spec/01 §4.2): over [from, to), limited to identities, exactly rows.
What one answer served (spec/01 §3).
A window as served: a range, a box over positions, or none.
Functions
Whether the delivery has no rows.
A delivery's fault in words: where it is, its path read as served.position or
replace[2], and what is wrong there.
The rows as maps, for a reader that reads them by name (finishing, push, a reducer):
{:ok, rows}, rows/1 with tuple rows and a tensor's named by names
(Macaw.Data.Table.named/2). Tuple rows without names are {:error, [{[:names], :required}]}: they cannot be read by name, and are never dropped for it. Rows that do
not fit their names are {[:names], reason}, Macaw.Data.Table.named/2's reason
({:width, names, width}, {:rank, rank}).
As named_rows/1, raising Macaw.Data.Error in format_error/1's words.
A delivery of rows, any source Macaw.Data.Table.rows/1 reads, kept as given, with
served: (a keyword list or map of the served facts), schema: (a version), origin:,
names: (tuple rows' or a tensor's column names, in order) and types: (for
table/2): {:ok, delivery}, or every fault, {path, reason} (spec/01 §3)
As new/2, raising Macaw.Data.Error in format_error/1's words.
The rows a delivery carries, as Macaw.Data.Table.rows/1 reads its source:
{:ok, rows}, a row list itself, uncopied, its tuple rows unchanged; a table, a column
map or a tensor one row at a time. Nothing is rewritten. new/2 checks the rows' shape,
so a delivery it built always answers them; rows of a delivery built another way that
Macaw.Data.Table.source/1 refuses are its faults under [:rows].
As rows/1, the rows alone, raising Macaw.Data.Error in format_error/1's words.
The rows packed into a Macaw.Data.Table (spec/01 §1.1), for a reader that wants
columns: {:ok, table} or {:error, faults}. A table passes as it is. The delivery's
types: and names: apply, and opts may give others (types: merged over them); an
option it does not take is {[key], :unknown_key}.
Packing normalizes (an integer beside floats reads back a float, a naive time a UTC
DateTime); the delivery's rows are never changed by it.
As table/2, raising Macaw.Data.Error naming the faults.
Types
@type fault() :: {[atom() | non_neg_integer()], term()}
A fault in a delivery's arguments: where it is, and why (format_error/1).
@type origin() :: %{source: term(), request: String.t(), as_of: DateTime.t() | nil} | nil
Where an answer came from.
A range replacement (spec/01 §4.2): over [from, to), limited to identities, exactly rows.
@type served() :: %{ window: window(), resolution: term(), capped: boolean(), stale: boolean(), reduced_by: :source | :runtime | nil, ordered: :source | :runtime | :sample | nil, params: map(), note: String.t() | nil, position: non_neg_integer() | nil, complete_until: term() }
What one answer served (spec/01 §3).
@type t() :: %Macaw.Data.Delivery{ names: [Macaw.Data.Table.name()] | nil, origin: origin(), replace: [replacement()], retracted: [map()], rows: Macaw.Data.Table.source() | Macaw.Data.Table.t(), schema: String.t() | nil, served: served(), types: %{required(Macaw.Data.Table.name()) => Macaw.Data.Table.type()} }
@type window() :: [term()] | %{west: number(), south: number(), east: number(), north: number()} | nil
A window as served: a range, a box over positions, or none.
Functions
Whether the delivery has no rows.
A delivery's fault in words: where it is, its path read as served.position or
replace[2], and what is wrong there.
iex> Macaw.Data.Delivery.format_error({[:served, :position], {:type, :non_neg_integer}})
"served.position: not a non-negative integer"
The rows as maps, for a reader that reads them by name (finishing, push, a reducer):
{:ok, rows}, rows/1 with tuple rows and a tensor's named by names
(Macaw.Data.Table.named/2). Tuple rows without names are {:error, [{[:names], :required}]}: they cannot be read by name, and are never dropped for it. Rows that do
not fit their names are {[:names], reason}, Macaw.Data.Table.named/2's reason
({:width, names, width}, {:rank, rank}).
As named_rows/1, raising Macaw.Data.Error in format_error/1's words.
@spec new( Macaw.Data.Table.source() | Macaw.Data.Table.t(), keyword() ) :: {:ok, t()} | {:error, [fault(), ...]}
A delivery of rows, any source Macaw.Data.Table.rows/1 reads, kept as given, with
served: (a keyword list or map of the served facts), schema: (a version), origin:,
names: (tuple rows' or a tensor's column names, in order) and types: (for
table/2): {:ok, delivery}, or every fault, {path, reason} (spec/01 §3):
{[:rows], :delivery}: a delivery of a delivery;{[:rows], {:type, :rows}}: rows of no shape a source has;{[key], :unknown_key}: an option that is not a delivery's, and{[:opts], {:type, :keyword}}options that are not a keyword list;{[:served, fact], :unknown_key}: a fact that is not a served fact, and{[:served, fact], reason}a value it does not take:{:type, :window},{:type, :boolean},{:enum, values},{:type, :map},{:type, :string},{:type, :non_neg_integer}or{:type, :instant};{[:served], {:type, :keyword}}for served facts that are neither a keyword list nor a map;{[:names], {:type, :names}},{[:types], {:type, :map}},{[:schema], {:type, :version}}and{[:origin], {:type, :origin}};{[:retracted], {:type, :list}}and{[:retracted, i], {:type, :row}};{[:replace], {:type, :list}}and{[:replace, i], {:type, :replacement}}.
format_error/1 gives a fault's words. The rows themselves are not read: a source's
rows are its own until a reader reads them. So a list of rows is not walked here, and an
improper one is Macaw.Data.Table.rows/1's fault when it is read.
@spec new!( Macaw.Data.Table.source() | Macaw.Data.Table.t(), keyword() ) :: t()
As new/2, raising Macaw.Data.Error in format_error/1's words.
@spec rows(t()) :: {:ok, [Macaw.Data.Table.row()]} | {:error, [fault(), ...]}
The rows a delivery carries, as Macaw.Data.Table.rows/1 reads its source:
{:ok, rows}, a row list itself, uncopied, its tuple rows unchanged; a table, a column
map or a tensor one row at a time. Nothing is rewritten. new/2 checks the rows' shape,
so a delivery it built always answers them; rows of a delivery built another way that
Macaw.Data.Table.source/1 refuses are its faults under [:rows].
@spec rows!(t()) :: [Macaw.Data.Table.row()]
As rows/1, the rows alone, raising Macaw.Data.Error in format_error/1's words.
@spec table( t(), keyword() ) :: {:ok, Macaw.Data.Table.t()} | {:error, term()}
The rows packed into a Macaw.Data.Table (spec/01 §1.1), for a reader that wants
columns: {:ok, table} or {:error, faults}. A table passes as it is. The delivery's
types: and names: apply, and opts may give others (types: merged over them); an
option it does not take is {[key], :unknown_key}.
Packing normalizes (an integer beside floats reads back a float, a naive time a UTC
DateTime); the delivery's rows are never changed by it.
@spec table!( t(), keyword() ) :: Macaw.Data.Table.t()
As table/2, raising Macaw.Data.Error naming the faults.