Macaw.Data.Delivery (macaw_data v0.13.0)

View Source

One answer of a source: its rows, as the source gave them, and what it actually served (spec/01 §3).

A delivery is a handful of terms beside its rows, never per row:

  • rows: the rows as the source gave them (macaw#185): a list of row maps or tuples, a map or keyword list of columns, an Nx.Tensor, a Table.Reader source, or a Macaw.Data.Table. Nothing is packed or rewritten at construction: rows/1 reads them as they are, a row list uncopied, and table/2 packs them into a table for a reader that wants columns, normalizing there and only there. A source that has a table delivers it, and it passes as it is. A gap stays nil in either form: absence is never zero.
  • names: the column names of tuple rows or a tensor, in order, for whoever must read them by name (named_rows/1, table/2, finishing); nil otherwise.
  • types: column types for table/2 to pack with, %{} to infer them.
  • schema: the version of the schema the rows were produced under (Macaw.Data.Schema.version/1), never the schema itself, which Macaw.Data.Schema.Cache holds by version; nil when the source has none.
  • served: what this answer served, which may differ from what was asked:
    • window: a range [from, to], a box over a :point window field %{west: w, south: s, east: e, north: n}, or nil;
    • resolution: the spacing of the rows as served (seconds, or a name such as :minute), or nil;
    • capped: whether the source stopped short of the window it was asked for;
    • stale: whether the source answered from a store it could not refresh, so the answer may be out of date (its upstream failed, and it served its last good answer); a fact of this one answer, as capped is, not a capability;
    • reduced_by: who brought the rows within budget, :source or :runtime, or nil when nobody had to;
    • ordered: who put the rows in the request's order (spec/01 §2.4): :source, which served its first rows in it; :runtime, which ordered every row the source served and kept the first; :sample, which ordered rows the source had already cut (its reduced_by: :source, or capped), so they are in the order but need not be the first in it; or nil, no order asked;
    • params: the parameters it was answered for, a map;
    • note: a line for a person ("coarsened from raw"), or nil;
    • position: for a snapshot taken for a push subscription, the sequence number of the last increment it reflects (0 for none); nil for any other answer (spec/01 §4.1);
    • complete_until: a value of the window field, a time or a number, below which the source expects no further change, or nil for no claim (spec/01 §4.4).
  • origin: where it came from, %{source: name, request: digest, as_of: time | nil}, or nil;

  • retracted and replace: the changes a pushed increment carries beside its inserted rows (spec/01 §4.2), rows withdrawn and range replacements %{from, to, identities, rows}; both [] by default, and a snapshot carries neither.

The shape of visualize's Visualize.Data.Delivery, whose facts are flat and whose reduced_by is :feed or :visualize, is spec/01 §3's here.

Examples

iex> alias Macaw.Data.Delivery
iex> rows = [%{t: 1, v: 3}, %{t: 2, v: nil}]
iex> {:ok, d} = Delivery.new(rows, served: [window: [0, 10]], schema: "ab12")
iex> Delivery.rows(d) === {:ok, rows}
true
iex> {d.served.window, d.served.capped, d.served.reduced_by}
{[0, 10], false, nil}
iex> d |> Delivery.table!() |> Macaw.Data.Table.column(:v)
[3, nil]

Summary

Types

A fault in a delivery's arguments: where it is, and why (format_error/1).

Where an answer came from.

A range replacement (spec/01 §4.2): over [from, to), limited to identities, exactly rows.

What one answer served (spec/01 §3).

t()

A window as served: a range, a box over positions, or none.

Functions

Whether the delivery has no rows.

A delivery's fault in words: where it is, its path read as served.position or replace[2], and what is wrong there.

The rows as maps, for a reader that reads them by name (finishing, push, a reducer): {:ok, rows}, rows/1 with tuple rows and a tensor's named by names (Macaw.Data.Table.named/2). Tuple rows without names are {:error, [{[:names], :required}]}: they cannot be read by name, and are never dropped for it. Rows that do not fit their names are {[:names], reason}, Macaw.Data.Table.named/2's reason ({:width, names, width}, {:rank, rank}).

A delivery of rows, any source Macaw.Data.Table.rows/1 reads, kept as given, with served: (a keyword list or map of the served facts), schema: (a version), origin:, names: (tuple rows' or a tensor's column names, in order) and types: (for table/2): {:ok, delivery}, or every fault, {path, reason} (spec/01 §3)

The rows a delivery carries, as Macaw.Data.Table.rows/1 reads its source: {:ok, rows}, a row list itself, uncopied, its tuple rows unchanged; a table, a column map or a tensor one row at a time. Nothing is rewritten. new/2 checks the rows' shape, so a delivery it built always answers them; rows of a delivery built another way that Macaw.Data.Table.source/1 refuses are its faults under [:rows].

As rows/1, the rows alone, raising Macaw.Data.Error in format_error/1's words.

The rows packed into a Macaw.Data.Table (spec/01 §1.1), for a reader that wants columns: {:ok, table} or {:error, faults}. A table passes as it is. The delivery's types: and names: apply, and opts may give others (types: merged over them); an option it does not take is {[key], :unknown_key}. Packing normalizes (an integer beside floats reads back a float, a naive time a UTC DateTime); the delivery's rows are never changed by it.

As table/2, raising Macaw.Data.Error naming the faults.

Types

fault()

@type fault() :: {[atom() | non_neg_integer()], term()}

A fault in a delivery's arguments: where it is, and why (format_error/1).

origin()

@type origin() ::
  %{source: term(), request: String.t(), as_of: DateTime.t() | nil} | nil

Where an answer came from.

replacement()

@type replacement() :: %{
  from: term(),
  to: term(),
  identities: [term()] | nil,
  rows: [map()]
}

A range replacement (spec/01 §4.2): over [from, to), limited to identities, exactly rows.

served()

@type served() :: %{
  window: window(),
  resolution: term(),
  capped: boolean(),
  stale: boolean(),
  reduced_by: :source | :runtime | nil,
  ordered: :source | :runtime | :sample | nil,
  params: map(),
  note: String.t() | nil,
  position: non_neg_integer() | nil,
  complete_until: term()
}

What one answer served (spec/01 §3).

t()

@type t() :: %Macaw.Data.Delivery{
  names: [Macaw.Data.Table.name()] | nil,
  origin: origin(),
  replace: [replacement()],
  retracted: [map()],
  rows: Macaw.Data.Table.source() | Macaw.Data.Table.t(),
  schema: String.t() | nil,
  served: served(),
  types: %{required(Macaw.Data.Table.name()) => Macaw.Data.Table.type()}
}

window()

@type window() ::
  [term()]
  | %{west: number(), south: number(), east: number(), north: number()}
  | nil

A window as served: a range, a box over positions, or none.

Functions

empty?(delivery)

@spec empty?(t()) :: boolean()

Whether the delivery has no rows.

format_error(arg)

@spec format_error(fault()) :: String.t()

A delivery's fault in words: where it is, its path read as served.position or replace[2], and what is wrong there.

iex> Macaw.Data.Delivery.format_error({[:served, :position], {:type, :non_neg_integer}})
"served.position: not a non-negative integer"

named_rows(delivery)

@spec named_rows(t()) :: {:ok, [map()]} | {:error, [fault(), ...]}

The rows as maps, for a reader that reads them by name (finishing, push, a reducer): {:ok, rows}, rows/1 with tuple rows and a tensor's named by names (Macaw.Data.Table.named/2). Tuple rows without names are {:error, [{[:names], :required}]}: they cannot be read by name, and are never dropped for it. Rows that do not fit their names are {[:names], reason}, Macaw.Data.Table.named/2's reason ({:width, names, width}, {:rank, rank}).

named_rows!(delivery)

@spec named_rows!(t()) :: [map()]

As named_rows/1, raising Macaw.Data.Error in format_error/1's words.

new(rows, opts \\ [])

@spec new(
  Macaw.Data.Table.source() | Macaw.Data.Table.t(),
  keyword()
) :: {:ok, t()} | {:error, [fault(), ...]}

A delivery of rows, any source Macaw.Data.Table.rows/1 reads, kept as given, with served: (a keyword list or map of the served facts), schema: (a version), origin:, names: (tuple rows' or a tensor's column names, in order) and types: (for table/2): {:ok, delivery}, or every fault, {path, reason} (spec/01 §3):

  • {[:rows], :delivery}: a delivery of a delivery;
  • {[:rows], {:type, :rows}}: rows of no shape a source has;
  • {[key], :unknown_key}: an option that is not a delivery's, and {[:opts], {:type, :keyword}} options that are not a keyword list;
  • {[:served, fact], :unknown_key}: a fact that is not a served fact, and {[:served, fact], reason} a value it does not take: {:type, :window}, {:type, :boolean}, {:enum, values}, {:type, :map}, {:type, :string}, {:type, :non_neg_integer} or {:type, :instant}; {[:served], {:type, :keyword}} for served facts that are neither a keyword list nor a map;
  • {[:names], {:type, :names}}, {[:types], {:type, :map}}, {[:schema], {:type, :version}} and {[:origin], {:type, :origin}};
  • {[:retracted], {:type, :list}} and {[:retracted, i], {:type, :row}}; {[:replace], {:type, :list}} and {[:replace, i], {:type, :replacement}}.

format_error/1 gives a fault's words. The rows themselves are not read: a source's rows are its own until a reader reads them. So a list of rows is not walked here, and an improper one is Macaw.Data.Table.rows/1's fault when it is read.

new!(rows, opts \\ [])

@spec new!(
  Macaw.Data.Table.source() | Macaw.Data.Table.t(),
  keyword()
) :: t()

As new/2, raising Macaw.Data.Error in format_error/1's words.

rows(delivery)

@spec rows(t()) :: {:ok, [Macaw.Data.Table.row()]} | {:error, [fault(), ...]}

The rows a delivery carries, as Macaw.Data.Table.rows/1 reads its source: {:ok, rows}, a row list itself, uncopied, its tuple rows unchanged; a table, a column map or a tensor one row at a time. Nothing is rewritten. new/2 checks the rows' shape, so a delivery it built always answers them; rows of a delivery built another way that Macaw.Data.Table.source/1 refuses are its faults under [:rows].

rows!(delivery)

@spec rows!(t()) :: [Macaw.Data.Table.row()]

As rows/1, the rows alone, raising Macaw.Data.Error in format_error/1's words.

table(delivery, opts \\ [])

@spec table(
  t(),
  keyword()
) :: {:ok, Macaw.Data.Table.t()} | {:error, term()}

The rows packed into a Macaw.Data.Table (spec/01 §1.1), for a reader that wants columns: {:ok, table} or {:error, faults}. A table passes as it is. The delivery's types: and names: apply, and opts may give others (types: merged over them); an option it does not take is {[key], :unknown_key}. Packing normalizes (an integer beside floats reads back a float, a naive time a UTC DateTime); the delivery's rows are never changed by it.

table!(delivery, opts \\ [])

@spec table!(
  t(),
  keyword()
) :: Macaw.Data.Table.t()

As table/2, raising Macaw.Data.Error naming the faults.