Macaw.Data.Source.Push (macaw_data v0.18.0)
View SourceOne push subscription, as a value (spec/01 §4): the source's changelog, applied to the relation it holds (spec/01 §2.5).
A source that supports :push sends its subscriber three messages, each tagged with the
reference subscribe/2 answered:
{:macaw_source, ref, {:schema, schema}} # first, and again when the schema changes
{:macaw_source, ref, {:snapshot, delivery}} # the relation, as of delivery.served.position
{:macaw_source, ref, {:increment, seq, delivery}} # the change numbered seqschema_message/2, snapshot_message/2 and increment_message/3 build them. The
subscriber holds a Push in its state: receive/2 takes each message (a schema is
cached, a snapshot or an increment buffered), snapshot/2 buffers a snapshot the
holder fetched itself, and drain/2 applies everything buffered since the last pulse
at once, so a burst of pushes between two pulses costs one drain. held/1 is the
relation as it stands; refetch/1 says when the holder must fetch a snapshot.
take_errors/1 hands the holder the errors recorded since it last took them, and
pending?/1 says whether a drain has anything to apply, and trim/2 drops the chunks
wholly before a sliding window's start, so a holder needs no field of the struct.
- The handshake sends the schema first. A delivery names its schema's version; one
naming a version not yet cached falls back to
describe(drain/2's function, theMacaw.Data.Schema.Cache.resolve/4describe function), asked once and cached. - Increments are numbered from 1 per subscription and apply in the order of their
numbers (spec/01 §4.1). A repeated or older one is dropped whole and counted in
ignored; one past the next is held back, the gap recorded inerrorsand answered byrefetch/1. - A snapshot hands off by position: it replaces what is held and takes its
served.position; increments at or before it are dropped, later ones apply. A snapshot older than the position held is dropped whole. - An increment's changes apply in order (spec/01 §4.2): its
retractedrows, each matched by identity, held window value and the fields it names, one copy per unit of weight; then itsreplaceranges, everything held in each retracted; then its insertions. A change that can't be matched, or contradicts a replacement, is a fault inerrors, never a guess; against a capped or reduced holding an unheld retraction is counted inignoredinstead. - Only at the held resolution (spec/01 §4.3): an increment whose
served.resolutiondiffers from what is held is not merged, andrefetch/1answers{:resolution, held, given}. - An increment's rows are insertions, merged by the window field: they stay
ordered by it, a late row placed among the held, and the served window grows to
cover them. Rows are never deduplicated: an identical row is a second row. A schema
with no window field keeps the newest
limitrows, marking the deliverycappedwhen it drops any. staleandnoteare the last answer's (spec/01 §4): a snapshot'sserved.staleandserved.notebecome the holding's, and an applied increment's replace them, as the source answers as of that change: one stating no note clears the last. What was done to the snapshot's rows (capped,reduced_by,reduced_with) and itsparamsstay the snapshot's, and the holding states noordered: it keeps its rows in the window field's order.- What is held is a chunked table (spec/01 §2.6):
held/1's rows are aMacaw.Data.Tableof the source's origin, its columns the schema's fields, packed asDelivery.table/2packs; every change reaches it throughMacaw.Data.Table.apply/2, a settled range's in its delta. Rows that don't pack, or can't be read by name (tuple rows withoutnames:, macaw_data#166), are{:table, faults}inerrors, and nothing of that delivery is applied.
It is not a process: the process that subscribes receives the messages in its mailbox
and passes them here. Ported from the logic of visualize's Visualize.Feed.Monitor,
whose message tag was :visualize_feed.
Summary
Functions
The schema cache the subscription resolves versions through (macaw_data#202): a holder
that describes the source again hands it to new/3's cache:, so no version is asked
for twice.
Everything buffered applied, in the order received: one drain per pulse, whatever the
push rate. Increments apply by their numbers, a snapshot hands off by its position
(spec/01 §4.1). A delivery naming a version the cache lacks is resolved through
describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason
kept in errors. A delivery naming no version is read under the held schema's. A
delivery Delivery.new/2 accepts never raises here: rows that can't be read by name or
don't pack are {:table, faults} in errors, nothing of that delivery applied.
How many snapshots and increments the last drain/2 applied (macaw_data#202): each taken
and merged with no fault recorded for it. Not counted are a repeat and a stale snapshot,
which ignored counts; a message whose taking recorded a fault in the errors, an
increment with any faulted change among them; and increments still waiting behind a gap.
An increment applied once its gap fills counts in the drain that applies it. 0 before
any drain, and each drain counts afresh.
The relation as it stands, or nil before anything is held: a delivery, what was
served beside its rows, which are a chunked Macaw.Data.Table of the source's origin
(spec/01 §2.6), its columns the schema's fields.
The message a source sends with the increment numbered seq (from 1).
A subscription to the source named name under ref, holding nothing yet, at
position 0. Options: cache: a Schema.Cache to start from, limit: the most rows
held for a schema with no window field (100_000), lateness: the source's
description's hard bound (spec/01 §4.4), nil by default, accepting a change of any
age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default,
derived from the resolution held.
As new/3, the subscription alone, raising Macaw.Data.Error naming the faults.
Whether drain/2 has anything buffered to apply (macaw_data#123): a message received
or a snapshot handed in since the last drain, or increments held back waiting on a gap
(spec/01 §4.1), which apply once the numbers before them, or a snapshot, arrive.
The number of the last message taken (macaw_data#202): a snapshot's position, or an
increment's number, consumed whether or not its changes applied (spec/01 §4.1); 0
before any. A holder that fetches a snapshot itself hands it this position.
A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or
an increment buffered; :ignore for a message that is not this subscription's.
nil, or why the holder must fetch a snapshot for this subscription and hand it to
snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or
{:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A
snapshot taken clears it.
The message a source sends with its schema, first and on every change.
A snapshot the holder fetched for this subscription (its request naming the
subscription, spec/01 §2.3), buffered as a received one is; its position is its
served.position.
The message a source sends with a snapshot for the subscription, its position in
served.position (spec/01 §4.1).
The errors recorded since they were last taken, in the order recorded, and the push
holding none (macaw_data#123): a gap, an unresolvable schema, a fault of a change, or
rows that can't be read by name or don't pack ({:table, faults}). A holder takes them
after each drain, so a subscription that lives long holds only what it has not yet read.
The holding without the chunks wholly before from in the window field's order, for a
holder whose window slides (macaw_data#123, spec/01 §4): {:ok, push}, or {:error, [{[:from], {:type, type}}]} for a from not of the window field's type (a time of any
kind for a :time field, compared on the instant scale; a number for a :number one).
As trim/2, the push alone, raising Macaw.Data.Error naming the fault.
Types
@type message() :: {:snapshot, Macaw.Data.Delivery.t()} | {:increment, pos_integer(), Macaw.Data.Delivery.t()}
A buffered message: a snapshot, or an increment and its number.
@type t() :: %Macaw.Data.Source.Push{ cache: Macaw.Data.Schema.Cache.t(), chunk_span: Duration.t() | number() | nil, drained: non_neg_integer(), errors: [term()], held: Macaw.Data.Delivery.t() | nil, ignored: %{required(atom()) => pos_integer()}, lateness: Duration.t() | number() | nil, limit: pos_integer(), name: term(), pending: [message()], position: non_neg_integer(), ref: reference(), refetch: term(), trimmed: term(), waiting: %{required(pos_integer()) => Macaw.Data.Delivery.t()} }
Functions
@spec cache(t()) :: Macaw.Data.Schema.Cache.t()
The schema cache the subscription resolves versions through (macaw_data#202): a holder
that describes the source again hands it to new/3's cache:, so no version is asked
for twice.
iex> alias Macaw.Data.Source.Push
iex> Push.cache(Push.new!(:ticks, make_ref())) == %Macaw.Data.Schema.Cache{}
true
@spec drain(t(), Macaw.Data.Schema.Cache.describe()) :: t()
Everything buffered applied, in the order received: one drain per pulse, whatever the
push rate. Increments apply by their numbers, a snapshot hands off by its position
(spec/01 §4.1). A delivery naming a version the cache lacks is resolved through
describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason
kept in errors. A delivery naming no version is read under the held schema's. A
delivery Delivery.new/2 accepts never raises here: rows that can't be read by name or
don't pack are {:table, faults} in errors, nothing of that delivery applied.
@spec drained(t()) :: non_neg_integer()
How many snapshots and increments the last drain/2 applied (macaw_data#202): each taken
and merged with no fault recorded for it. Not counted are a repeat and a stale snapshot,
which ignored counts; a message whose taking recorded a fault in the errors, an
increment with any faulted change among them; and increments still waiting behind a gap.
An increment applied once its gap fills counts in the drain that applies it. 0 before
any drain, and each drain counts afresh.
iex> alias Macaw.Data.Source.Push
iex> Push.drained(Push.new!(:ticks, make_ref()))
0
@spec held(t()) :: Macaw.Data.Delivery.t() | nil
The relation as it stands, or nil before anything is held: a delivery, what was
served beside its rows, which are a chunked Macaw.Data.Table of the source's origin
(spec/01 §2.6), its columns the schema's fields.
@spec increment_message(reference(), pos_integer(), Macaw.Data.Delivery.t()) :: tuple()
The message a source sends with the increment numbered seq (from 1).
A subscription to the source named name under ref, holding nothing yet, at
position 0. Options: cache: a Schema.Cache to start from, limit: the most rows
held for a schema with no window field (100_000), lateness: the source's
description's hard bound (spec/01 §4.4), nil by default, accepting a change of any
age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default,
derived from the resolution held.
A value it can't use is its fault by path, every one at once, never a failure at a later
drain (macaw_data#177): {[:cache], {:type, :cache}}; {[:limit], {:type, :integer}};
{[:lateness], {:type, :lateness}} for anything but nil, a number or a well-formed
Duration; {[:chunk_span], {:type, :chunk_span}} for anything but nil, a positive
number or a positive Duration of fixed length. There is no schema here to judge a
value against: a number against a time field, or the reverse, is the delivery's fault, as
ever. A limit of 0 or below holds nothing.
iex> alias Macaw.Data.Source.Push
iex> Push.new(:ticks, make_ref(), limit: "x", chunk_span: 0)
{:error, [{[:limit], {:type, :integer}}, {[:chunk_span], {:type, :chunk_span}}]}
As new/3, the subscription alone, raising Macaw.Data.Error naming the faults.
Whether drain/2 has anything buffered to apply (macaw_data#123): a message received
or a snapshot handed in since the last drain, or increments held back waiting on a gap
(spec/01 §4.1), which apply once the numbers before them, or a snapshot, arrive.
iex> alias Macaw.Data.Source.Push
iex> Push.pending?(Push.new!(:ticks, make_ref()))
false
@spec position(t()) :: non_neg_integer()
The number of the last message taken (macaw_data#202): a snapshot's position, or an
increment's number, consumed whether or not its changes applied (spec/01 §4.1); 0
before any. A holder that fetches a snapshot itself hands it this position.
iex> alias Macaw.Data.Source.Push
iex> Push.position(Push.new!(:ticks, make_ref()))
0
A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or
an increment buffered; :ignore for a message that is not this subscription's.
nil, or why the holder must fetch a snapshot for this subscription and hand it to
snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or
{:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A
snapshot taken clears it.
@spec schema_message(reference(), Macaw.Data.Schema.t()) :: tuple()
The message a source sends with its schema, first and on every change.
@spec snapshot(t(), Macaw.Data.Delivery.t()) :: t()
A snapshot the holder fetched for this subscription (its request naming the
subscription, spec/01 §2.3), buffered as a received one is; its position is its
served.position.
@spec snapshot_message(reference(), Macaw.Data.Delivery.t()) :: tuple()
The message a source sends with a snapshot for the subscription, its position in
served.position (spec/01 §4.1).
The errors recorded since they were last taken, in the order recorded, and the push
holding none (macaw_data#123): a gap, an unresolvable schema, a fault of a change, or
rows that can't be read by name or don't pack ({:table, faults}). A holder takes them
after each drain, so a subscription that lives long holds only what it has not yet read.
iex> alias Macaw.Data.Source.Push
iex> {[], push} = Push.take_errors(Push.new!(:ticks, make_ref()))
iex> push.errors
[]
The holding without the chunks wholly before from in the window field's order, for a
holder whose window slides (macaw_data#123, spec/01 §4): {:ok, push}, or {:error, [{[:from], {:type, type}}]} for a from not of the window field's type (a time of any
kind for a :time field, compared on the instant scale; a number for a :number one).
- Only whole chunks are dropped: each held chunk whose every window value is before
from, throughMacaw.Data.Table.trim/2over their rows, so nothing is copied or cut and every chunk kept is the same chunk, by id and version. A chunk straddlingfromis kept whole; finishing trims its rows. - The served window is narrowed, never widened: it starts at
from, or at the held start where that is later; its upper end andcomplete_untilare unchanged. A box has no start to narrow, so it stays as it is. - The trimmed start is kept (spec/01 §4.2): after it, a retraction naming a row
before it that is not held, or a replace range wholly before it, is counted in
ignoredas:trimmed, not recorded as a fault. A late insertion before it is still placed; the next trim drops it. A snapshot taken replaces the holding and clears it.
It is a no-op {:ok, push} when nothing is held, when the schema has no window field,
for a :point window field, which has no order, and for a nil from, an open start.
iex> alias Macaw.Data.Source.Push
iex> push = Push.new!(:ticks, make_ref())
iex> Push.trim(push, 10) == {:ok, push}
true
As trim/2, the push alone, raising Macaw.Data.Error naming the fault.