Macaw.Data.Source.Finish (macaw_data v0.16.0)

View Source

Finishing (spec/01 §2.4): Macaw does what a source did not.

Every capability is optional. Where a source's description does not list one the request asks, finish/3 does it to the delivery and reports it in served:

  1. :window: rows trimmed to the window field's range, the served window intersected;
  2. :box: rows trimmed to the box over the :point window field;
  3. :region: rows trimmed to each ranged field's range;
  4. :latest: each identity's latest row by the window field (spec/01 §1.4);
  5. :order: the rows sorted by the request's order (Macaw.Data.Source.Order), ordered: :runtime, or ordered: :sample where the source had already cut them;
  6. :budget: the rows reduced to the budget by a registered reducer (reduce/5, spec/04 §7), or, for an ordered delivery, kept to its first budget rows; reduced_by: :runtime.

The budget never widens the window: finishing only removes rows. A delivery that needs none of it comes back as it is.

A table stays a table, finished on its columns (macaw_data#105): every step reads the columns it decides by and keeps rows of the table's chunks, as views or gathered by their slots, and never reads the table's rows. An order under a budget that keeps its head gathers that head alone. Each step keeps exactly the rows finishing the table's rows would keep. Any other delivery is finished over its rows, read by name.

The pure rules of running sources (spec/01 §6) are here too: ceiling/2 clamps a budget to the host's ceiling, and pooled/1 is the one budget a pooled fetch asks for, the largest among its readers, each reader then reduced to its own by reduce/5.

Ported from visualize's Visualize.Feed.finish/3 and reduce/4.

Summary

Functions

A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as page:, may lower the host's, never raise it. host: is required: without it, {:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5). ceiling!/2 answers the pair or raises.

As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.

The delivery finished for the request, each step where the description's supports lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). reduce: names the budget's reducer by spelling, a registered transform that declares reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and operations: is the registry it is found in (macaw_data's own by default).

As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.

The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil when any reader wants every row.

The delivery reduced to budget rows by the reducer method names in registry (spec/04 §7): "lttb" by default, "m4", or any registered transform declaring reduces/0, by spelling; macaw_data's own registry by default. It records reduced_by: :runtime. Each series (the rows sharing every :category field's value) is reduced field by field against the window field, or the row's place without one, by the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units reduces/0 counts, never below the least its option takes, and the union of the kept rows returned whole, in their order, never more than the budget. A nil budget, a delivery within it, and a schema with no :number field to reduce by come back as they are.

Functions

ceiling(budget, opts)

@spec ceiling(
  pos_integer() | nil,
  keyword()
) :: {:ok, {pos_integer(), boolean()}} | {:error, [{list(), term()}, ...]}

A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as page:, may lower the host's, never raise it. host: is required: without it, {:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5). ceiling!/2 answers the pair or raises.

iex> Macaw.Data.Source.Finish.ceiling(50_000, host: 10_000)
{:ok, {10_000, true}}
iex> Macaw.Data.Source.Finish.ceiling!(nil, host: 10_000, page: 2_000)
{2_000, false}
iex> Macaw.Data.Source.Finish.ceiling(5, page: 2_000)
{:error, [{[:host], :required}]}
iex> Macaw.Data.Source.Finish.ceiling(50_000, host: :nope, page: 0)
{:error, [{[:host], {:type, :pos_integer}}, {[:page], {:type, :pos_integer}}]}
iex> Macaw.Data.Source.Finish.ceiling(0, host: 10_000)
{:error, [{[:budget], {:type, :pos_integer}}]}

ceiling!(budget, opts)

@spec ceiling!(
  pos_integer() | nil,
  keyword()
) :: {pos_integer(), boolean()}

As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.

finish(delivery, description, request, opts \\ [])

The delivery finished for the request, each step where the description's supports lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). reduce: names the budget's reducer by spelling, a registered transform that declares reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and operations: is the registry it is found in (macaw_data's own by default).

The request is the caller's, so each fault is answered before any step runs: Macaw.Data.Window.check/2's faults for a request the schema cannot answer, {[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer (spec/04 §7), and {[key], :unknown_key} for an option finish/4 does not take. A step that must read tuple rows without names answers Delivery.named_rows/1's fault, and a reading the reducer cannot place {[:rows, field], reason} (reduce/5). finish!/4 answers the delivery or raises.

finish!(delivery, description, request, opts \\ [])

As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.

pooled(budgets)

@spec pooled([pos_integer() | nil, ...]) :: pos_integer() | nil

The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil when any reader wants every row.

iex> Macaw.Data.Source.Finish.pooled([100, 4_000, 250])
4000

reduce(delivery, schema, budget, method \\ "lttb", registry \\ Operations.registry())

@spec reduce(
  Macaw.Data.Delivery.t(),
  Macaw.Data.Schema.t(),
  pos_integer() | nil,
  String.t() | atom(),
  Macaw.Data.Operations.Registry.t()
) :: {:ok, Macaw.Data.Delivery.t()} | {:error, [{list(), term()}, ...]}

The delivery reduced to budget rows by the reducer method names in registry (spec/04 §7): "lttb" by default, "m4", or any registered transform declaring reduces/0, by spelling; macaw_data's own registry by default. It records reduced_by: :runtime. Each series (the rows sharing every :category field's value) is reduced field by field against the window field, or the row's place without one, by the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units reduces/0 counts, never below the least its option takes, and the union of the kept rows returned whole, in their order, never more than the budget. A nil budget, a delivery within it, and a schema with no :number field to reduce by come back as they are.

An ordered delivery keeps its head (spec/01 §2.4): one whose served.ordered is set holds its rows in a reader's order, and a reader that orders rows wants the first of them, not a shape, so its first budget rows are kept, reduced_by: :runtime, with any reducer. So a pooled fetch's ordered rows reduce to each reader's own head.

{:ok, delivery}, or {:error, faults} (spec/00 §5): {[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer, Delivery.named_rows/1's fault for tuple rows without names, and {[:rows, field], reason} for a field's reading the reducer cannot place (a number field holding a string: {:reads, :number}). reduce!/5 answers the delivery or raises.

iex> alias Macaw.Data.{Delivery, Schema, Source.Finish}
iex> schema = Schema.new!(%{t: %{type: :number, window: true}, v: %{type: :number}})
iex> {:ok, reduced} = Finish.reduce(Delivery.new!(for t <- 1..100, do: %{t: t, v: rem(t * 7, 5)}), schema, 10)
iex> {length(Delivery.rows!(reduced)), reduced.served.reduced_by}
{10, :runtime}

reduce!(delivery, schema, budget, method \\ "lttb", registry \\ Operations.registry())

As reduce/5, the delivery alone, raising Macaw.Data.Error naming the faults.