Macaw.Data.Derived (macaw_data v0.18.0)

View Source

A derived source: a source computed from other sources by a function, rows in and rows out by default, or read as columns under input: :columns (spec/02 §2.2). Ported from visualize's Visualize.VirtualSource (visualize#643, D-194).

It is data: id, its inputs (from), its output's schema, and its function as source text (elixir, checked by Macaw.Data.Derived.Sandbox) or a host function's name (function, defined with defsource, spec/02 §7), so a document holds it and its forms write it. With per: the function runs once per entity of one input and its rows are joined back by that input's declared identity. Every name is a string.

iex> {:ok, health} =
...>   Macaw.Data.Derived.new(
...>     id: :health,
...>     from: [hosts: %{last: 20}],
...>     per: :identity,
...>     schema: %{
...>       host: %{type: :category, identity: true},
...>       state: %{type: :category, values: [:ok, :warning, :error]}
...>     },
...>     elixir: "fn rows -> %{state: :ok} end"
...>   )
iex> {health.id, health.from}
{"health", [{"hosts", %{last: 20}}]}
iex> Macaw.Data.Derived.conform(health, [%{host: "a", state: :ok}])
{:ok, [%{"host" => "a", "state" => :ok}]}
iex> Macaw.Data.Derived.conform(health, [%{host: "a", state: :unknown}])
{:error, {0, "state", :not_in_values}}
iex> Macaw.Data.Derived.resolve([["health"], ["ops", "health"]], [:ops], :health)
{:ok, ["ops", "health"]}

Summary

Types

A fault: the path from the source to the value, and why (spec/02 §1.1).

An input as its owner read it (spec/02 §1.3, §2.2): its name, its rows in its window, its schema, and the delivery it came in.

What an input names (spec/02 §2.1): a short id, a source, or a path.

t()

How much of an input the function needs (spec/02 §2.2).

Functions

The first argument a function is called with (spec/02 §2.2): the inputs as one map from key to rows. A stored function reads a key, a row's field and a row's category value as it reads its own literals, a word's atom, else the string; a host's function reads each key as its string and each row as its source gave it. Under input: :columns each key holds the input's table, as the engine hands it (spec/02 §8.2).

An answer held to the source's schema (spec/02 §1.2): {:ok, rows}, the rows keyed by their fields' strings, or {:error, {index, field, reason}} for the first row that does not hold, never the value.

Every cycle among definitions whose inputs are resolved (spec/02 §3): reads maps each path to the paths it reads. Each strongly connected component that holds a cycle, a source that reads itself included, is its members' paths, sorted; the cycles are sorted too. Sources downstream of a cycle are in none.

A path shown dotted (spec/02 §3).

One fault as a line a person reads, the root named derived source (spec/02 §1.1).

The derived source a JSON document holds: as from_map/2, its bytes and nesting held to the limits before it is decoded.

The derived source a map form holds: through macaw_layers' gates (Macaw.Layers.Form.gates/3: the depth and item limits, the version), read by its tag table and keys, making no atom, its schema by Macaw.Data.Schema.from_map/1, then judged by new/1. limits: tightens the limits. A derived source holds no URL: one the document holds (a $url at a window's bound) is {path, {:url, reason}}, under the kernel's closed policy (Macaw.Layers.Url.Policy.new!([])).

As from_map/2, raising Macaw.Data.Error naming every fault.

The name of the input a per source's entities come from: the only input under :identity, the one named under {:identity, input}; nil without per (spec/02 §1.3).

The key an input is read under (spec/02 §2.1): a short id, a source's name, or a path's last name, its string.

A derived source of its value, a keyword list or a map of its fields as terms, checked alone (spec/02 §1.1): {:ok, derived}, or {:error, errors} with every fault by path. Its keys are atoms, and a string key is none of them: a map form is refused key by key as :unknown_key, since it is read by from_map/2, never guessed at here.

The derived source, or Macaw.Data.Error naming every fault.

The type a window field orders by under last: n (spec/02 §2.2): {:ok, :time} or {:ok, :number}, keyed by Macaw.Data.Source.Order.key/2; :error for a field of another type, or none, whose rows keep their order.

A per source's batch over read inputs (spec/02 §1.3): call is the function applied to one entity's argument (argument/2's map, or the entity's rows alone when the source has one input), called once per entity of the identity input in its order of first appearance, a row with no identity dropped. Each answer, a row or a list of rows, is given its entity's identity and the answers joined: {:ok, rows}, or {:error, :nonconforming, {index, field, reason}} for an answer that is not rows or a row naming another identity. The caller runs the whole batch as one bounded call.

What a reference names (spec/02 §3), given every definition's path (a list, or a map keyed by path) and the scope it is read from: a short id from the inside out, a full path exactly. {:ok, path} or :error; {:source, name} is never resolved.

The source's JSON form: to_map/1, stamped "$version" and encoded.

The source's map form (spec/02 §1.5), JSON-ready, written by macaw_layers' JSON adapter (Macaw.Layers.Form) under the kernel's tags and the source's own, its schema by Macaw.Data.Schema.to_map/1: {:ok, map}, or every value without a form, by path (a window's time not in Etc/UTC is {:type, :term}). from_map(map) == {:ok, derived}.

:ok, or every fault of the source given what it reads (spec/02 §1.1): inputs:, a map from an input's key (or name) to its %Macaw.Data.Schema{} where it is known, and functions:, the host functions' registry (Macaw.Data.Derived.Functions.registry/1), or nil to leave the name unchecked, and operations:, the registry a stored function's Macaw.Data.Ops calls are held to (Macaw.Data.Operations.registry/1), or nil to hold their names only to being literals (spec/04 §6.1). An option it does not take is {[key], :unknown_key} (spec/00 §5).

An input in its window (spec/02 §2.2), a table or a list of rows: nil every row and %{last: n} each identity's newest n rows by the window field, else the newest n, each in the input's form; a request the rows its window, box, region and latest select, finished by Macaw.Data.Source.Finish, also in the input's form: a table finished on its columns stays a table (macaw_data#105).

Types

error()

@type error() :: {list(), term()}

A fault: the path from the source to the value, and why (spec/02 §1.1).

input()

@type input() :: %{
  :name => name(),
  :rows => [map()] | Macaw.Data.Table.t(),
  :schema => Macaw.Data.Schema.t(),
  optional(:delivery) => Macaw.Data.Delivery.t()
}

An input as its owner read it (spec/02 §1.3, §2.2): its name, its rows in its window, its schema, and the delivery it came in.

name()

@type name() :: String.t() | {:source, String.t()} | [String.t(), ...]

What an input names (spec/02 §2.1): a short id, a source, or a path.

t()

@type t() :: %Macaw.Data.Derived{
  elixir: String.t() | nil,
  from: [{name(), window()}, ...],
  function: String.t() | nil,
  id: String.t(),
  input: :rows | :columns,
  per: nil | :identity | {:identity, name()},
  schema: Macaw.Data.Schema.t()
}

window()

@type window() ::
  nil
  | %{last: pos_integer()}
  | %{
      optional(:window) => list(),
      optional(:box) => map(),
      optional(:region) => map(),
      optional(:latest) => boolean()
    }

How much of an input the function needs (spec/02 §2.2).

Functions

argument(source, inputs)

@spec argument(t(), [input()]) :: map()

The first argument a function is called with (spec/02 §2.2): the inputs as one map from key to rows. A stored function reads a key, a row's field and a row's category value as it reads its own literals, a word's atom, else the string; a host's function reads each key as its string and each row as its source gave it. Under input: :columns each key holds the input's table, as the engine hands it (spec/02 §8.2).

conform(derived, rows)

@spec conform(t(), term()) ::
  {:ok, [map()] | Macaw.Data.Table.t()}
  | {:error, {non_neg_integer() | nil, String.t() | nil, term()}}

An answer held to the source's schema (spec/02 §1.2): {:ok, rows}, the rows keyed by their fields' strings, or {:error, {index, field, reason}} for the first row that does not hold, never the value.

A source that reads its inputs as columns (input: :columns) may also answer a Macaw.Data.Table or a column map, %{field => [values]}, held to the schema column by column and answered as the output table, packed under the schema and never made into rows: {:ok, table}. A fault of a whole column (:undescribed, :duplicate, :not_a_column, {:length, n}, a table column's {:type, type}) has index nil; any other is the lowest row index at fault, an identity gap before a value fault at one index, fields in name order.

iex> source = Macaw.Data.Derived.new!(id: :c, from: [in: nil], input: :columns,
...>   schema: %{n: %{type: :number}, s: %{type: :category, values: [:ok]}}, function: :f)
iex> {:ok, table} = Macaw.Data.Derived.conform(source, %{n: [1, 2], s: [:ok, nil]})
iex> Macaw.Data.Table.rows!(table)
[%{"n" => 1, "s" => "ok"}, %{"n" => 2, "s" => nil}]
iex> Macaw.Data.Derived.conform(source, %{n: [1, "two"]})
{:error, {1, "n", {:type, :number}}}
iex> Macaw.Data.Derived.conform(source, %{n: [1], s: []})
{:error, {nil, "s", {:length, 1}}}

cycles(reads)

@spec cycles(%{required(list()) => [list()]}) :: [[list(), ...]]

Every cycle among definitions whose inputs are resolved (spec/02 §3): reads maps each path to the paths it reads. Each strongly connected component that holds a cycle, a source that reads itself included, is its members' paths, sorted; the cycles are sorted too. Sources downstream of a cycle are in none.

iex> Macaw.Data.Derived.cycles(%{["a"] => [["b"]], ["b"] => [["a"]], ["c"] => [["a"]], ["d"] => [["d"]]})
[[["a"], ["b"]], [["d"]]]

dotted(path)

@spec dotted([String.t() | atom()]) :: String.t()

A path shown dotted (spec/02 §3).

iex> Macaw.Data.Derived.dotted(["ops", "health"])
"ops.health"

format_error(arg)

@spec format_error(error()) :: String.t()

One fault as a line a person reads, the root named derived source (spec/02 §1.1).

iex> Macaw.Data.Derived.format_error({[:per], {:per, :ambiguous}})
"per: is :identity over several inputs: name the input, {:identity, input}"

from_json(json, opts \\ [])

@spec from_json(
  binary(),
  keyword()
) :: {:ok, t()} | {:error, [error(), ...]}

The derived source a JSON document holds: as from_map/2, its bytes and nesting held to the limits before it is decoded.

from_map(map, opts \\ [])

@spec from_map(
  map(),
  keyword()
) :: {:ok, t()} | {:error, [error(), ...]}

The derived source a map form holds: through macaw_layers' gates (Macaw.Layers.Form.gates/3: the depth and item limits, the version), read by its tag table and keys, making no atom, its schema by Macaw.Data.Schema.from_map/1, then judged by new/1. limits: tightens the limits. A derived source holds no URL: one the document holds (a $url at a window's bound) is {path, {:url, reason}}, under the kernel's closed policy (Macaw.Layers.Url.Policy.new!([])).

from_map!(map, opts \\ [])

@spec from_map!(
  map(),
  keyword()
) :: t()

As from_map/2, raising Macaw.Data.Error naming every fault.

identity_input(derived)

@spec identity_input(t()) :: name() | nil

The name of the input a per source's entities come from: the only input under :identity, the one named under {:identity, input}; nil without per (spec/02 §1.3).

key(path)

@spec key(name() | atom()) :: String.t() | nil

The key an input is read under (spec/02 §2.1): a short id, a source's name, or a path's last name, its string.

iex> Macaw.Data.Derived.key(["ops", "health"])
"health"
iex> Macaw.Data.Derived.key({:source, "vm.memory.total"})
"vm.memory.total"

new(source)

@spec new(map() | keyword()) :: {:ok, t()} | {:error, [error(), ...]}

A derived source of its value, a keyword list or a map of its fields as terms, checked alone (spec/02 §1.1): {:ok, derived}, or {:error, errors} with every fault by path. Its keys are atoms, and a string key is none of them: a map form is refused key by key as :unknown_key, since it is read by from_map/2, never guessed at here.

new!(source)

@spec new!(map() | keyword()) :: t()

The derived source, or Macaw.Data.Error naming every fault.

order_key(schema, field)

@spec order_key(Macaw.Data.Schema.t(), name() | atom()) ::
  {:ok, :time | :number} | :error

The type a window field orders by under last: n (spec/02 §2.2): {:ok, :time} or {:ok, :number}, keyed by Macaw.Data.Source.Order.key/2; :error for a field of another type, or none, whose rows keep their order.

per(source, inputs, call)

@spec per(t(), [input()], (term() -> term())) ::
  {:ok, [map()]} | {:error, :nonconforming, {term(), term(), term()}}

A per source's batch over read inputs (spec/02 §1.3): call is the function applied to one entity's argument (argument/2's map, or the entity's rows alone when the source has one input), called once per entity of the identity input in its order of first appearance, a row with no identity dropped. Each answer, a row or a list of rows, is given its entity's identity and the answers joined: {:ok, rows}, or {:error, :nonconforming, {index, field, reason}} for an answer that is not rows or a row naming another identity. The caller runs the whole batch as one bounded call.

resolve(paths, scope, ref)

@spec resolve([list()] | map(), list(), term()) :: {:ok, [String.t()]} | :error

What a reference names (spec/02 §3), given every definition's path (a list, or a map keyed by path) and the scope it is read from: a short id from the inside out, a full path exactly. {:ok, path} or :error; {:source, name} is never resolved.

iex> paths = [["health"], ["ops", "health"], ["ops", "box", "load"]]
iex> Macaw.Data.Derived.resolve(paths, ["ops", "box"], "health")
{:ok, ["ops", "health"]}
iex> Macaw.Data.Derived.resolve(paths, ["other"], :health)
{:ok, ["health"]}
iex> Macaw.Data.Derived.resolve(paths, [], ["ops", "box", "load"])
{:ok, ["ops", "box", "load"]}
iex> Macaw.Data.Derived.resolve(paths, ["ops"], {:source, "health"})
:error

to_json(source)

@spec to_json(t()) :: {:ok, String.t()} | {:error, [error(), ...]}

The source's JSON form: to_map/1, stamped "$version" and encoded.

to_map(source)

@spec to_map(t()) :: {:ok, map()} | {:error, [error(), ...]}

The source's map form (spec/02 §1.5), JSON-ready, written by macaw_layers' JSON adapter (Macaw.Layers.Form) under the kernel's tags and the source's own, its schema by Macaw.Data.Schema.to_map/1: {:ok, map}, or every value without a form, by path (a window's time not in Etc/UTC is {:type, :term}). from_map(map) == {:ok, derived}.

validate(source, opts \\ [])

@spec validate(
  t(),
  keyword()
) :: :ok | {:error, [error(), ...]}

:ok, or every fault of the source given what it reads (spec/02 §1.1): inputs:, a map from an input's key (or name) to its %Macaw.Data.Schema{} where it is known, and functions:, the host functions' registry (Macaw.Data.Derived.Functions.registry/1), or nil to leave the name unchecked, and operations:, the registry a stored function's Macaw.Data.Ops calls are held to (Macaw.Data.Operations.registry/1), or nil to hold their names only to being literals (spec/04 §6.1). An option it does not take is {[key], :unknown_key} (spec/00 §5).

window(subject, window, schema)

@spec window(Macaw.Data.Table.t() | [map()], window(), Macaw.Data.Schema.t()) ::
  Macaw.Data.Table.t() | [map()]

An input in its window (spec/02 §2.2), a table or a list of rows: nil every row and %{last: n} each identity's newest n rows by the window field, else the newest n, each in the input's form; a request the rows its window, box, region and latest select, finished by Macaw.Data.Source.Finish, also in the input's form: a table finished on its columns stays a table (macaw_data#105).

The window field orders by its type's key (order_key/2): a :time by its instant (Macaw.Data.Source.Order.instant/1, every kind of time on one scale), a :number by its value, stably; when a row has no key, the rows are left as they came. A table is windowed on its columns, by last: n and by a request alike, and no row is made: the kept rows are views of its chunks or gathered from them.

iex> schema = Macaw.Data.Schema.new!(%{t: %{type: :number, window: true}, v: %{type: :number}})
iex> rows = for t <- 1..5, do: %{t: t, v: t * 10}
iex> Macaw.Data.Derived.window(rows, %{last: 2}, schema)
[%{t: 4, v: 40}, %{t: 5, v: 50}]
iex> Macaw.Data.Derived.window(rows, %{window: [2, 3]}, schema)
[%{t: 2, v: 20}, %{t: 3, v: 30}]
iex> table = Macaw.Data.Table.from(rows) |> elem(1)
iex> Macaw.Data.Derived.window(table, %{last: 2}, schema) |> Macaw.Data.Table.column(:v)
[40, 50]
iex> Macaw.Data.Derived.window(table, %{window: [2, 3]}, schema) |> Macaw.Data.Table.column(:v)
[20, 30]