Macaw.Data.Derived (macaw_data v0.12.0)
View SourceA derived source: a source computed from other sources by a function, rows in and rows
out by default, or read as columns under input: :columns (spec/02 §2.2). Ported from visualize's Visualize.VirtualSource (visualize#643, D-194).
It is data: id, its inputs (from), its output's schema, and its function as
source text (elixir, checked by Macaw.Data.Derived.Sandbox) or a host function's
name (function, defined with defsource, spec/02 §7), so a document holds it and its
forms write it. With per: the function runs once per entity of one input and its rows
are joined back by that input's declared identity. Every name is a string.
iex> {:ok, health} =
...> Macaw.Data.Derived.new(
...> id: :health,
...> from: [hosts: %{last: 20}],
...> per: :identity,
...> schema: %{
...> host: %{type: :category, identity: true},
...> state: %{type: :category, values: [:ok, :warning, :error]}
...> },
...> elixir: "fn rows -> %{state: :ok} end"
...> )
iex> {health.id, health.from}
{"health", [{"hosts", %{last: 20}}]}
iex> Macaw.Data.Derived.conform(health, [%{host: "a", state: :ok}])
{:ok, [%{"host" => "a", "state" => :ok}]}
iex> Macaw.Data.Derived.conform(health, [%{host: "a", state: :unknown}])
{:error, {0, "state", :not_in_values}}
iex> Macaw.Data.Derived.resolve([["health"], ["ops", "health"]], [:ops], :health)
{:ok, ["ops", "health"]}
Summary
Types
A fault: the path from the source to the value, and why (spec/02 §1.1).
An input as its owner read it (spec/02 §1.3, §2.2): its name, its rows in its window, its schema, and the delivery it came in.
What an input names (spec/02 §2.1): a short id, a source, or a path.
How much of an input the function needs (spec/02 §2.2).
Functions
The first argument a function is called with (spec/02 §2.2): the inputs as one map from
key to rows. A stored function reads a key, a row's field and a row's category value as
it reads its own literals, a word's atom, else the string; a host's function reads each
key as its string and each row as its source gave it. Under input: :columns each key
holds the input's table, as the engine hands it (spec/02 §8.2).
An answer held to the source's schema (spec/02 §1.2): {:ok, rows}, the rows keyed by
their fields' strings, or {:error, {index, field, reason}} for the first row that does
not hold, never the value.
Every cycle among definitions whose inputs are resolved (spec/02 §3): reads maps each
path to the paths it reads. Each strongly connected component that holds a cycle, a
source that reads itself included, is its members' paths, sorted; the cycles are sorted
too. Sources downstream of a cycle are in none.
A path shown dotted (spec/02 §3).
One fault as a line a person reads, the root named derived source (spec/02 §1.1).
The derived source a JSON document holds: as from_map/2, its bytes and nesting held to
the limits before it is decoded.
The derived source a map form holds: through macaw_layers' gates
(Macaw.Layers.Form.gates/3: the depth and item limits, the version), read by its tag
table and keys, making no atom, its schema by Macaw.Data.Schema.from_map/1, then
judged by new/1. limits: tightens the limits. A derived source holds no URL: one the
document holds (a $url at a window's bound) is {path, {:url, reason}}, under the
kernel's closed policy (Macaw.Layers.Url.Policy.new([])).
The name of the input a per source's entities come from: the only input under
:identity, the one named under {:identity, input}; nil without per (spec/02 §1.3).
The key an input is read under (spec/02 §2.1): a short id, a source's name, or a path's last name, its string.
A derived source of its value, a keyword list or a map of its fields as terms, checked
alone (spec/02 §1.1): {:ok, derived}, or {:error, errors} with every fault by path.
Its keys are atoms, and a string key is none of them: a map form is refused key by key
as :unknown_key, since it is read by from_map/2, never guessed at here.
The derived source, or Macaw.Data.Error naming every fault.
The type a window field orders by under last: n (spec/02 §2.2): {:ok, :time} or
{:ok, :number}, keyed by Macaw.Data.Source.Order.key/2; :error for a field of
another type, or none, whose rows keep their order.
A per source's batch over read inputs (spec/02 §1.3): call is the function applied to
one entity's argument (argument/2's map, or the entity's rows alone when the source
has one input), called once per entity of the identity input in its order of first
appearance, a row with no identity dropped. Each answer, a row or a list of rows, is
given its entity's identity and the answers joined: {:ok, rows}, or
{:error, :nonconforming, {index, field, reason}} for an answer that is not rows or a
row naming another identity. The caller runs the whole batch as one bounded call.
What a reference names (spec/02 §3), given every definition's path (a list, or a map
keyed by path) and the scope it is read from: a short id from the inside out, a full
path exactly. {:ok, path} or :error; {:source, name} is never resolved.
The source's JSON form: to_map/1, stamped "$version" and encoded.
The source's map form (spec/02 §1.5), JSON-ready, written by macaw_layers' JSON adapter
(Macaw.Layers.Form) under the kernel's tags and the source's own, its schema by
Macaw.Data.Schema.to_map/1: {:ok, map}, or every value without a form, by path (a
window's time not in Etc/UTC is {:type, :term}). from_map(map) == {:ok, derived}.
:ok, or every fault of the source given what it reads (spec/02 §1.1): inputs:, a map
from an input's key (or name) to its %Macaw.Data.Schema{} where it is known, and
functions:, the host functions' registry (Macaw.Data.Derived.Functions.registry/1),
or nil to leave the name unchecked, and operations:, the registry a stored
function's Macaw.Data.Ops calls are held to (Macaw.Data.Operations.registry/1), or
nil to hold their names only to being literals (spec/04 §6.1). An option it does not
take is {[key], :unknown_key} (spec/00 §5).
An input in its window (spec/02 §2.2), a table or a list of rows: nil every row and
%{last: n} each identity's newest n rows by the window field, else the newest n,
each in the input's form; a request the rows its window, box, region and latest select,
finished by Macaw.Data.Source.Finish, also in the input's form: a table finished on
its columns stays a table (macaw_data#105).
Types
A fault: the path from the source to the value, and why (spec/02 §1.1).
@type input() :: %{ :name => name(), :rows => [map()] | Macaw.Data.Table.t(), :schema => Macaw.Data.Schema.t(), optional(:delivery) => Macaw.Data.Delivery.t() }
An input as its owner read it (spec/02 §1.3, §2.2): its name, its rows in its window, its schema, and the delivery it came in.
What an input names (spec/02 §2.1): a short id, a source, or a path.
@type window() :: nil | %{last: pos_integer()} | %{ optional(:window) => list(), optional(:box) => map(), optional(:region) => map(), optional(:latest) => boolean() }
How much of an input the function needs (spec/02 §2.2).
Functions
The first argument a function is called with (spec/02 §2.2): the inputs as one map from
key to rows. A stored function reads a key, a row's field and a row's category value as
it reads its own literals, a word's atom, else the string; a host's function reads each
key as its string and each row as its source gave it. Under input: :columns each key
holds the input's table, as the engine hands it (spec/02 §8.2).
@spec conform(t(), term()) :: {:ok, [map()] | Macaw.Data.Table.t()} | {:error, {non_neg_integer() | nil, String.t() | nil, term()}}
An answer held to the source's schema (spec/02 §1.2): {:ok, rows}, the rows keyed by
their fields' strings, or {:error, {index, field, reason}} for the first row that does
not hold, never the value.
A source that reads its inputs as columns (input: :columns) may also answer a
Macaw.Data.Table or a column map, %{field => [values]}, held to the schema column by
column and answered as the output table, packed under the schema and never made into
rows: {:ok, table}. A fault of a whole column (:undescribed, :duplicate,
:not_a_column, {:length, n}, a table column's {:type, type}) has index nil;
any other is the lowest row index at fault, an identity gap before a value fault at one
index, fields in name order.
iex> source = Macaw.Data.Derived.new!(id: :c, from: [in: nil], input: :columns,
...> schema: %{n: %{type: :number}, s: %{type: :category, values: [:ok]}}, function: :f)
iex> {:ok, table} = Macaw.Data.Derived.conform(source, %{n: [1, 2], s: [:ok, nil]})
iex> Macaw.Data.Table.rows!(table)
[%{"n" => 1, "s" => "ok"}, %{"n" => 2, "s" => nil}]
iex> Macaw.Data.Derived.conform(source, %{n: [1, "two"]})
{:error, {1, "n", {:type, :number}}}
iex> Macaw.Data.Derived.conform(source, %{n: [1], s: []})
{:error, {nil, "s", {:length, 1}}}
Every cycle among definitions whose inputs are resolved (spec/02 §3): reads maps each
path to the paths it reads. Each strongly connected component that holds a cycle, a
source that reads itself included, is its members' paths, sorted; the cycles are sorted
too. Sources downstream of a cycle are in none.
iex> Macaw.Data.Derived.cycles(%{["a"] => [["b"]], ["b"] => [["a"]], ["c"] => [["a"]], ["d"] => [["d"]]})
[[["a"], ["b"]], [["d"]]]
A path shown dotted (spec/02 §3).
iex> Macaw.Data.Derived.dotted(["ops", "health"])
"ops.health"
One fault as a line a person reads, the root named derived source (spec/02 §1.1).
iex> Macaw.Data.Derived.format_error({[:per], {:per, :ambiguous}})
"per: is :identity over several inputs: name the input, {:identity, input}"
The derived source a JSON document holds: as from_map/2, its bytes and nesting held to
the limits before it is decoded.
The derived source a map form holds: through macaw_layers' gates
(Macaw.Layers.Form.gates/3: the depth and item limits, the version), read by its tag
table and keys, making no atom, its schema by Macaw.Data.Schema.from_map/1, then
judged by new/1. limits: tightens the limits. A derived source holds no URL: one the
document holds (a $url at a window's bound) is {path, {:url, reason}}, under the
kernel's closed policy (Macaw.Layers.Url.Policy.new([])).
The name of the input a per source's entities come from: the only input under
:identity, the one named under {:identity, input}; nil without per (spec/02 §1.3).
The key an input is read under (spec/02 §2.1): a short id, a source's name, or a path's last name, its string.
iex> Macaw.Data.Derived.key(["ops", "health"])
"health"
iex> Macaw.Data.Derived.key({:source, "vm.memory.total"})
"vm.memory.total"
A derived source of its value, a keyword list or a map of its fields as terms, checked
alone (spec/02 §1.1): {:ok, derived}, or {:error, errors} with every fault by path.
Its keys are atoms, and a string key is none of them: a map form is refused key by key
as :unknown_key, since it is read by from_map/2, never guessed at here.
The derived source, or Macaw.Data.Error naming every fault.
@spec order_key(Macaw.Data.Schema.t(), name() | atom()) :: {:ok, :time | :number} | :error
The type a window field orders by under last: n (spec/02 §2.2): {:ok, :time} or
{:ok, :number}, keyed by Macaw.Data.Source.Order.key/2; :error for a field of
another type, or none, whose rows keep their order.
@spec per(t(), [input()], (term() -> term())) :: {:ok, [map()]} | {:error, :nonconforming, {term(), term(), term()}}
A per source's batch over read inputs (spec/02 §1.3): call is the function applied to
one entity's argument (argument/2's map, or the entity's rows alone when the source
has one input), called once per entity of the identity input in its order of first
appearance, a row with no identity dropped. Each answer, a row or a list of rows, is
given its entity's identity and the answers joined: {:ok, rows}, or
{:error, :nonconforming, {index, field, reason}} for an answer that is not rows or a
row naming another identity. The caller runs the whole batch as one bounded call.
What a reference names (spec/02 §3), given every definition's path (a list, or a map
keyed by path) and the scope it is read from: a short id from the inside out, a full
path exactly. {:ok, path} or :error; {:source, name} is never resolved.
iex> paths = [["health"], ["ops", "health"], ["ops", "box", "load"]]
iex> Macaw.Data.Derived.resolve(paths, ["ops", "box"], "health")
{:ok, ["ops", "health"]}
iex> Macaw.Data.Derived.resolve(paths, ["other"], :health)
{:ok, ["health"]}
iex> Macaw.Data.Derived.resolve(paths, [], ["ops", "box", "load"])
{:ok, ["ops", "box", "load"]}
iex> Macaw.Data.Derived.resolve(paths, ["ops"], {:source, "health"})
:error
The source's JSON form: to_map/1, stamped "$version" and encoded.
The source's map form (spec/02 §1.5), JSON-ready, written by macaw_layers' JSON adapter
(Macaw.Layers.Form) under the kernel's tags and the source's own, its schema by
Macaw.Data.Schema.to_map/1: {:ok, map}, or every value without a form, by path (a
window's time not in Etc/UTC is {:type, :term}). from_map(map) == {:ok, derived}.
:ok, or every fault of the source given what it reads (spec/02 §1.1): inputs:, a map
from an input's key (or name) to its %Macaw.Data.Schema{} where it is known, and
functions:, the host functions' registry (Macaw.Data.Derived.Functions.registry/1),
or nil to leave the name unchecked, and operations:, the registry a stored
function's Macaw.Data.Ops calls are held to (Macaw.Data.Operations.registry/1), or
nil to hold their names only to being literals (spec/04 §6.1). An option it does not
take is {[key], :unknown_key} (spec/00 §5).
@spec window(Macaw.Data.Table.t() | [map()], window(), Macaw.Data.Schema.t()) :: Macaw.Data.Table.t() | [map()]
An input in its window (spec/02 §2.2), a table or a list of rows: nil every row and
%{last: n} each identity's newest n rows by the window field, else the newest n,
each in the input's form; a request the rows its window, box, region and latest select,
finished by Macaw.Data.Source.Finish, also in the input's form: a table finished on
its columns stays a table (macaw_data#105).
The window field orders by its type's key (order_key/2): a :time by its instant
(Macaw.Data.Source.Order.instant/1, every kind of time on one scale), a :number by
its value, stably; when a row has no key, the rows are left as they came. A table is
windowed on its columns, by last: n and by a request alike, and no row is made: the
kept rows are views of its chunks or gathered from them.
iex> schema = Macaw.Data.Schema.new!(%{t: %{type: :number, window: true}, v: %{type: :number}})
iex> rows = for t <- 1..5, do: %{t: t, v: t * 10}
iex> Macaw.Data.Derived.window(rows, %{last: 2}, schema)
[%{t: 4, v: 40}, %{t: 5, v: 50}]
iex> Macaw.Data.Derived.window(rows, %{window: [2, 3]}, schema)
[%{t: 2, v: 20}, %{t: 3, v: 30}]
iex> table = Macaw.Data.Table.from(rows) |> elem(1)
iex> Macaw.Data.Derived.window(table, %{last: 2}, schema) |> Macaw.Data.Table.column(:v)
[40, 50]
iex> Macaw.Data.Derived.window(table, %{window: [2, 3]}, schema) |> Macaw.Data.Table.column(:v)
[20, 30]