Macaw.Data.Source.Finish (macaw_data v0.16.0)
View SourceFinishing (spec/01 §2.4): Macaw does what a source did not.
Every capability is optional. Where a source's description does not list one the
request asks, finish/3 does it to the delivery and reports it in served:
:window: rows trimmed to the window field's range, the served window intersected;:box: rows trimmed to the box over the:pointwindow field;:region: rows trimmed to each ranged field's range;:latest: each identity's latest row by the window field (spec/01 §1.4);:order: the rows sorted by the request's order (Macaw.Data.Source.Order),ordered: :runtime, orordered: :samplewhere the source had already cut them;:budget: the rows reduced to the budget by a registered reducer (reduce/5, spec/04 §7), or, for an ordered delivery, kept to its firstbudgetrows;reduced_by: :runtime.
The budget never widens the window: finishing only removes rows. A delivery that needs none of it comes back as it is.
A table stays a table, finished on its columns (macaw_data#105): every step reads the columns it decides by and keeps rows of the table's chunks, as views or gathered by their slots, and never reads the table's rows. An order under a budget that keeps its head gathers that head alone. Each step keeps exactly the rows finishing the table's rows would keep. Any other delivery is finished over its rows, read by name.
The pure rules of running sources (spec/01 §6) are here too: ceiling/2 clamps a
budget to the host's ceiling, and pooled/1 is the one budget a pooled fetch asks for,
the largest among its readers, each reader then reduced to its own by reduce/5.
Ported from visualize's Visualize.Feed.finish/3 and reduce/4.
Summary
Functions
A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as
page:, may lower the host's, never raise it. host: is required: without it,
{:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer
is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive
integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and
nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5).
ceiling!/2 answers the pair or raises.
As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.
The delivery finished for the request, each step where the description's supports
lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). reduce:
names the budget's reducer by spelling, a registered transform that declares
reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and
operations: is the registry it is found in (macaw_data's own by default).
As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.
The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil
when any reader wants every row.
The delivery reduced to budget rows by the reducer method names in registry
(spec/04 §7): "lttb" by default, "m4", or any registered transform declaring
reduces/0, by spelling; macaw_data's own registry by default. It records
reduced_by: :runtime. Each series (the rows sharing every :category field's value)
is reduced field by field against the window field, or the row's place without one, by
the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units
reduces/0 counts, never below the least its option takes, and
the union of the kept rows returned whole, in their order, never more than the budget.
A nil budget, a delivery within it, and a schema with no :number field to reduce by
come back as they are.
As reduce/5, the delivery alone, raising Macaw.Data.Error naming the faults.
Functions
@spec ceiling( pos_integer() | nil, keyword() ) :: {:ok, {pos_integer(), boolean()}} | {:error, [{list(), term()}, ...]}
A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as
page:, may lower the host's, never raise it. host: is required: without it,
{:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer
is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive
integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and
nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5).
ceiling!/2 answers the pair or raises.
iex> Macaw.Data.Source.Finish.ceiling(50_000, host: 10_000)
{:ok, {10_000, true}}
iex> Macaw.Data.Source.Finish.ceiling!(nil, host: 10_000, page: 2_000)
{2_000, false}
iex> Macaw.Data.Source.Finish.ceiling(5, page: 2_000)
{:error, [{[:host], :required}]}
iex> Macaw.Data.Source.Finish.ceiling(50_000, host: :nope, page: 0)
{:error, [{[:host], {:type, :pos_integer}}, {[:page], {:type, :pos_integer}}]}
iex> Macaw.Data.Source.Finish.ceiling(0, host: 10_000)
{:error, [{[:budget], {:type, :pos_integer}}]}
@spec ceiling!( pos_integer() | nil, keyword() ) :: {pos_integer(), boolean()}
As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.
@spec finish( Macaw.Data.Delivery.t(), Macaw.Data.Source.Description.t(), Macaw.Data.Source.Request.t(), keyword() ) :: {:ok, Macaw.Data.Delivery.t()} | {:error, [{list(), term()}, ...]}
The delivery finished for the request, each step where the description's supports
lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). reduce:
names the budget's reducer by spelling, a registered transform that declares
reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and
operations: is the registry it is found in (macaw_data's own by default).
The request is the caller's, so each fault is answered before any step runs:
Macaw.Data.Window.check/2's faults for a request the schema cannot answer,
{[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer
(spec/04 §7), and {[key], :unknown_key} for an option finish/4 does not take. A
step that must read tuple rows without names answers Delivery.named_rows/1's fault,
and a reading the reducer cannot place {[:rows, field], reason} (reduce/5).
finish!/4 answers the delivery or raises.
@spec finish!( Macaw.Data.Delivery.t(), Macaw.Data.Source.Description.t(), Macaw.Data.Source.Request.t(), keyword() ) :: Macaw.Data.Delivery.t()
As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.
@spec pooled([pos_integer() | nil, ...]) :: pos_integer() | nil
The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil
when any reader wants every row.
iex> Macaw.Data.Source.Finish.pooled([100, 4_000, 250])
4000
@spec reduce( Macaw.Data.Delivery.t(), Macaw.Data.Schema.t(), pos_integer() | nil, String.t() | atom(), Macaw.Data.Operations.Registry.t() ) :: {:ok, Macaw.Data.Delivery.t()} | {:error, [{list(), term()}, ...]}
The delivery reduced to budget rows by the reducer method names in registry
(spec/04 §7): "lttb" by default, "m4", or any registered transform declaring
reduces/0, by spelling; macaw_data's own registry by default. It records
reduced_by: :runtime. Each series (the rows sharing every :category field's value)
is reduced field by field against the window field, or the row's place without one, by
the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units
reduces/0 counts, never below the least its option takes, and
the union of the kept rows returned whole, in their order, never more than the budget.
A nil budget, a delivery within it, and a schema with no :number field to reduce by
come back as they are.
An ordered delivery keeps its head (spec/01 §2.4): one whose served.ordered is
set holds its rows in a reader's order, and a reader that orders rows wants the first
of them, not a shape, so its first budget rows are kept, reduced_by: :runtime, with
any reducer. So a pooled fetch's ordered rows reduce to each reader's own head.
{:ok, delivery}, or {:error, faults} (spec/00 §5): {[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer, Delivery.named_rows/1's fault for
tuple rows without names, and {[:rows, field], reason} for a field's reading the
reducer cannot place (a number field holding a string: {:reads, :number}).
reduce!/5 answers the delivery or raises.
iex> alias Macaw.Data.{Delivery, Schema, Source.Finish}
iex> schema = Schema.new!(%{t: %{type: :number, window: true}, v: %{type: :number}})
iex> {:ok, reduced} = Finish.reduce(Delivery.new!(for t <- 1..100, do: %{t: t, v: rem(t * 7, 5)}), schema, 10)
iex> {length(Delivery.rows!(reduced)), reduced.served.reduced_by}
{10, :runtime}
@spec reduce!( Macaw.Data.Delivery.t(), Macaw.Data.Schema.t(), pos_integer() | nil, String.t() | atom(), Macaw.Data.Operations.Registry.t() ) :: Macaw.Data.Delivery.t()
As reduce/5, the delivery alone, raising Macaw.Data.Error naming the faults.