Macaw.Data.Source.Finish (macaw_data v0.18.0)

View Source

Finishing (spec/01 §2.4): Macaw does what a source did not.

Every capability is optional. Where a source's description does not list one the request asks, finish/3 does it to the delivery and reports it in served:

  1. :window: rows trimmed to the window field's range, the served window intersected;
  2. :box: rows trimmed to the box over the :point window field;
  3. :region: rows trimmed to each ranged field's range;
  4. :latest: each identity's latest row by the window field (spec/01 §1.4);
  5. :order: the rows sorted by the request's order (Macaw.Data.Source.Order), ordered: :runtime, or ordered: :sample where the source had already cut them;
  6. :budget: the rows reduced to the budget by a registered reducer (reduce/5, spec/04 §7), the request's own at its count where it names one, or, for an ordered delivery, kept to its first budget rows; reduced_by: :runtime.

The budget never widens the window: finishing only removes rows. A delivery that needs none of it comes back as it is.

A table stays a table, finished on its columns (macaw_data#105): every step reads the columns it decides by and keeps rows of the table's chunks, as views or gathered by their slots, and never reads the table's rows. An order under a budget that keeps its head gathers that head alone. Each step keeps exactly the rows finishing the table's rows would keep. Any other delivery is finished over its rows, read by name.

finish/5 and reduce/5 given a Macaw.Data.Operations.Cache reduce a table chunk by chunk from the partials it holds (macaw_data#247), with the same answer.

The pure rules of running sources (spec/01 §6) are here too: ceiling/2 clamps a budget to the host's ceiling, and pooled/1 is the one budget a pooled fetch asks for, the largest among its readers, each reader then reduced to its own by reduce/5.

Ported from visualize's Visualize.Feed.finish/3 and reduce/4.

Summary

Functions

A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as page:, may lower the host's, never raise it. host: is required: without it, {:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5). ceiling!/2 answers the pair or raises.

As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.

The delivery finished for the request, each step where the description's supports lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). The request's reduce names the reader's reducer (spec/01 §2.3, macaw_data#246); else reduce: names the budget's reducer by spelling, a registered transform that declares reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and operations: is the registry it is found in (macaw_data's own by default).

As finish/4, the budget reduced through the caller's Macaw.Data.Operations.Cache (spec/01 §2.4, macaw_data#247): {:ok, delivery, cache}, the delivery finish/4 answers and the cache holding this call's parts, or finish/4's faults. A table reduced by m4 or lttb is reduced chunk by chunk from the partials the cache holds (reduce/5), so a warm call costs the chunks that changed.

As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.

The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil when any reader wants every row. A reader is its budget or its request (macaw_data#246): a request asking reduce: :none, or with no budget, wants every row, so readers with different reducers share one fetch, each then reduced its own way by reduce/5.

The delivery reduced to budget rows by the reducer method names in registry (spec/04 §7): "lttb" by default, "m4", or any registered transform declaring reduces/0, by spelling; macaw_data's own registry by default. It records reduced_by: :runtime. Each series (the rows sharing every :category field's value) is reduced field by field against the window field, or the row's place without one, by the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units reduces/0 counts, never below the least its option takes, and the union of the kept rows returned whole, in their order, never more than the budget. A nil budget, a delivery within it, and a schema with no :number field to reduce by come back as they are. It records reduced_with: {name, c}.

Functions

ceiling(budget, opts)

@spec ceiling(
  pos_integer() | nil,
  keyword()
) :: {:ok, {pos_integer(), boolean()}} | {:error, [{list(), term()}, ...]}

A budget clamped to the host's ceiling (spec/01 §6, macaw_compose 05 §1.2): {:ok, {budget, clamped?}}. A nil budget reads at the ceiling; a page's ceiling, given as page:, may lower the host's, never raise it. host: is required: without it, {:error, [{[:host], :required}]}. A budget that is neither nil nor a positive integer is {[:budget], {:type, :pos_integer}}, and a host: or page: that is no positive integer is {[key], {:type, :pos_integer}}, every fault at once, the budget's first, and nothing is clamped; an option it does not take is {[key], :unknown_key} (spec/00 §5). ceiling!/2 answers the pair or raises.

iex> Macaw.Data.Source.Finish.ceiling(50_000, host: 10_000)
{:ok, {10_000, true}}
iex> Macaw.Data.Source.Finish.ceiling!(nil, host: 10_000, page: 2_000)
{2_000, false}
iex> Macaw.Data.Source.Finish.ceiling(5, page: 2_000)
{:error, [{[:host], :required}]}
iex> Macaw.Data.Source.Finish.ceiling(50_000, host: :nope, page: 0)
{:error, [{[:host], {:type, :pos_integer}}, {[:page], {:type, :pos_integer}}]}
iex> Macaw.Data.Source.Finish.ceiling(0, host: 10_000)
{:error, [{[:budget], {:type, :pos_integer}}]}

ceiling!(budget, opts)

@spec ceiling!(
  pos_integer() | nil,
  keyword()
) :: {pos_integer(), boolean()}

As ceiling/2, the pair alone, raising Macaw.Data.Error naming the faults.

finish(delivery, description, request, opts \\ [])

The delivery finished for the request, each step where the description's supports lacks it: {:ok, delivery}, or {:error, faults} (spec/01 §2.4, spec/00 §5). The request's reduce names the reader's reducer (spec/01 §2.3, macaw_data#246); else reduce: names the budget's reducer by spelling, a registered transform that declares reduces/0 ("lttb", the default, or "m4" in macaw_data's own package), and operations: is the registry it is found in (macaw_data's own by default).

The request is the caller's, so each fault is answered before any step runs: Macaw.Data.Window.check/2's faults for a request the schema cannot answer, {[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer (spec/04 §7), the request's own or reduce:'s, {[:reduce], {:least, least}} for a request's count below its reducer's least, and {[key], :unknown_key} for an option finish/4 does not take. A step that must read tuple rows without names answers Delivery.named_rows/1's fault, and a reading the reducer cannot place {[:rows, field], reason} (reduce/5). finish!/4 answers the delivery or raises.

finish(delivery, description, request, opts, cache)

As finish/4, the budget reduced through the caller's Macaw.Data.Operations.Cache (spec/01 §2.4, macaw_data#247): {:ok, delivery, cache}, the delivery finish/4 answers and the cache holding this call's parts, or finish/4's faults. A table reduced by m4 or lttb is reduced chunk by chunk from the partials the cache holds (reduce/5), so a warm call costs the chunks that changed.

finish!(delivery, description, request, opts \\ [])

As finish/4, the delivery alone, raising Macaw.Data.Error naming the faults.

pooled(readers)

@spec pooled([pos_integer() | nil | Macaw.Data.Source.Request.t(), ...]) ::
  pos_integer() | nil

The budget a pooled fetch asks for: the largest among its readers' (spec/01 §6), nil when any reader wants every row. A reader is its budget or its request (macaw_data#246): a request asking reduce: :none, or with no budget, wants every row, so readers with different reducers share one fetch, each then reduced its own way by reduce/5.

iex> Macaw.Data.Source.Finish.pooled([100, 4_000, 250])
4000

reduce(delivery, schema, budget, method \\ "lttb", registry \\ Operations.registry())

The delivery reduced to budget rows by the reducer method names in registry (spec/04 §7): "lttb" by default, "m4", or any registered transform declaring reduces/0, by spelling; macaw_data's own registry by default. It records reduced_by: :runtime. Each series (the rows sharing every :category field's value) is reduced field by field against the window field, or the row's place without one, by the reducer's node %{fields: [x, y], count => c}, c the budget's share in the units reduces/0 counts, never below the least its option takes, and the union of the kept rows returned whole, in their order, never more than the budget. A nil budget, a delivery within it, and a schema with no :number field to reduce by come back as they are. It records reduced_with: {name, c}.

A reader's request in the budget's place (spec/01 §6.1, macaw_data#246) reduces by that request's reducer (reduce:, spec/01 §2.3), else by method, to its budget: a reducer it names at its own count reduces each series and field to that count, the budget, when it has one, still capping the union; :none keeps every row up to the budget, which is then only a ceiling, an even stride keeping the first and the last. So each reader of a pooled fetch is reduced its own way.

Through the caller's cache (spec/01 §2.4, macaw_data#247): given a Macaw.Data.Operations.Cache in method's place, the budget or request is reduced as without one and the answer is {:ok, delivery, cache} (reduce!/5: {delivery, cache}). A table reduced by m4 or lttb is reduced chunk by chunk from the ladder partials the cache holds by chunk id and version (Finish.Chunked), the same rows, names and served facts at the cost of the chunks that changed; any other reduces whole and the cache comes back as it was. Prune it with the held table (Cache.prune/2).

An ordered delivery keeps its head (spec/01 §2.4): one whose served.ordered is set holds its rows in a reader's order, and a reader that orders rows wants the first of them, not a shape, so its first budget rows are kept, reduced_by: :runtime, with any reducer. So a pooled fetch's ordered rows reduce to each reader's own head.

{:ok, delivery}, or {:error, faults} (spec/00 §5): {[:reduce], {:unknown, :reducer, name}} for a name that is no registered reducer, {[:reduce], {:least, least}} for a request's count below the least its reducer takes, Delivery.named_rows/1's fault for tuple rows without names, and {[:rows, field], reason} for a field's reading the reducer cannot place (a number field holding a string: {:reads, :number}). reduce!/5 answers the delivery or raises.

iex> alias Macaw.Data.{Delivery, Schema, Source.Finish}
iex> schema = Schema.new!(%{t: %{type: :number, window: true}, v: %{type: :number}})
iex> {:ok, reduced} = Finish.reduce(Delivery.new!(for t <- 1..100, do: %{t: t, v: rem(t * 7, 5)}), schema, 10)
iex> {length(Delivery.rows!(reduced)), reduced.served.reduced_by}
{10, :runtime}

reduce!(delivery, schema, budget, method \\ "lttb", registry \\ Operations.registry())

As reduce/5, the delivery alone, raising Macaw.Data.Error naming the faults.