Macaw.Data.Source.Push (macaw_data v0.9.0)
View SourceOne push subscription, as a value (spec/01 §4): the source's changelog, applied to the relation it holds (spec/01 §2.5).
A source that supports :push sends its subscriber three messages, each tagged with the
reference subscribe/2 answered:
{:macaw_source, ref, {:schema, schema}} # first, and again when the schema changes
{:macaw_source, ref, {:snapshot, delivery}} # the relation, as of delivery.served.position
{:macaw_source, ref, {:increment, seq, delivery}} # the change numbered seqschema_message/2, snapshot_message/2 and increment_message/3 build them. The
subscriber holds a Push in its state: receive/2 takes each message (a schema is
cached, a snapshot or an increment buffered), snapshot/2 buffers a snapshot the
holder fetched itself, and drain/2 applies everything buffered since the last pulse
at once, so a burst of pushes between two pulses costs one drain. held/1 is the
relation as it stands; refetch/1 says when the holder must fetch a snapshot.
- The handshake sends the schema first. A delivery names its schema's version; one
naming a version not yet cached falls back to
describe(drain/2's function, theMacaw.Data.Schema.Cache.resolve/4describe function), asked once and cached. - Increments are numbered from 1 per subscription and apply in the order of their
numbers (spec/01 §4.1). A repeated or older one is dropped whole and counted in
ignored; one past the next is held back, the gap recorded inerrorsand answered byrefetch/1. - A snapshot hands off by position: it replaces what is held and takes its
served.position; increments at or before it are dropped, later ones apply. A snapshot older than the position held is dropped whole. - An increment's changes apply in order (spec/01 §4.2): its
retractedrows, each matched by identity, held window value and the fields it names, one copy per unit of weight; then itsreplaceranges, everything held in each retracted; then its insertions. A change that can't be matched, or contradicts a replacement, is a fault inerrors, never a guess; against a capped or reduced holding an unheld retraction is counted inignoredinstead. - Only at the held resolution (spec/01 §4.3): an increment whose
served.resolutiondiffers from what is held is not merged, andrefetch/1answers{:resolution, held, given}. - An increment's rows are insertions, merged by the window field: they stay
ordered by it, a late row placed among the held, and the served window grows to
cover them. Rows are never deduplicated: an identical row is a second row. A schema
with no window field keeps the newest
limitrows, marking the deliverycappedwhen it drops any. - What is held is a chunked table (spec/01 §2.6):
held/1's rows are aMacaw.Data.Tableof the source's origin, its columns the schema's fields, packed asDelivery.table/2packs; every change reaches it throughMacaw.Data.Table.apply/2, a settled range's in its delta. Rows that don't pack are{:table, faults}inerrors.
It is not a process: the process that subscribes receives the messages in its mailbox
and passes them here. Ported from the logic of visualize's Visualize.Feed.Monitor,
whose message tag was :visualize_feed.
Summary
Functions
Everything buffered applied, in the order received: one drain per pulse, whatever the
push rate. Increments apply by their numbers, a snapshot hands off by its position
(spec/01 §4.1). A delivery naming a version the cache lacks is resolved through
describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason
kept in errors. A delivery naming no version is read under the held schema's.
The relation as it stands, or nil before anything is held: a delivery, what was
served beside its rows, which are a chunked Macaw.Data.Table of the source's origin
(spec/01 §2.6), its columns the schema's fields.
The message a source sends with the increment numbered seq (from 1).
A subscription to the source named name under ref, holding nothing yet, at
position 0. Options: cache: a Schema.Cache to start from, limit: the most rows
held for a schema with no window field (100_000), lateness: the source's
description's hard bound (spec/01 §4.4), nil by default, accepting a change of any
age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default,
derived from the resolution held.
A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or
an increment buffered; :ignore for a message that is not this subscription's.
nil, or why the holder must fetch a snapshot for this subscription and hand it to
snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or
{:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A
snapshot taken clears it.
The message a source sends with its schema, first and on every change.
A snapshot the holder fetched for this subscription (its request naming the
subscription, spec/01 §2.3), buffered as a received one is; its position is its
served.position.
The message a source sends with a snapshot for the subscription, its position in
served.position (spec/01 §4.1).
Types
@type message() :: {:snapshot, Macaw.Data.Delivery.t()} | {:increment, pos_integer(), Macaw.Data.Delivery.t()}
A buffered message: a snapshot, or an increment and its number.
@type t() :: %Macaw.Data.Source.Push{ cache: Macaw.Data.Schema.Cache.t(), chunk_span: Duration.t() | number() | nil, errors: [term()], held: Macaw.Data.Delivery.t() | nil, ignored: %{required(atom()) => pos_integer()}, lateness: Duration.t() | number() | nil, limit: pos_integer(), name: term(), pending: [message()], position: non_neg_integer(), ref: reference(), refetch: term(), waiting: %{required(pos_integer()) => Macaw.Data.Delivery.t()} }
Functions
@spec drain(t(), Macaw.Data.Schema.Cache.describe()) :: t()
Everything buffered applied, in the order received: one drain per pulse, whatever the
push rate. Increments apply by their numbers, a snapshot hands off by its position
(spec/01 §4.1). A delivery naming a version the cache lacks is resolved through
describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason
kept in errors. A delivery naming no version is read under the held schema's.
@spec held(t()) :: Macaw.Data.Delivery.t() | nil
The relation as it stands, or nil before anything is held: a delivery, what was
served beside its rows, which are a chunked Macaw.Data.Table of the source's origin
(spec/01 §2.6), its columns the schema's fields.
@spec increment_message(reference(), pos_integer(), Macaw.Data.Delivery.t()) :: tuple()
The message a source sends with the increment numbered seq (from 1).
A subscription to the source named name under ref, holding nothing yet, at
position 0. Options: cache: a Schema.Cache to start from, limit: the most rows
held for a schema with no window field (100_000), lateness: the source's
description's hard bound (spec/01 §4.4), nil by default, accepting a change of any
age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default,
derived from the resolution held.
A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or
an increment buffered; :ignore for a message that is not this subscription's.
nil, or why the holder must fetch a snapshot for this subscription and hand it to
snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or
{:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A
snapshot taken clears it.
@spec schema_message(reference(), Macaw.Data.Schema.t()) :: tuple()
The message a source sends with its schema, first and on every change.
@spec snapshot(t(), Macaw.Data.Delivery.t()) :: t()
A snapshot the holder fetched for this subscription (its request naming the
subscription, spec/01 §2.3), buffered as a received one is; its position is its
served.position.
@spec snapshot_message(reference(), Macaw.Data.Delivery.t()) :: tuple()
The message a source sends with a snapshot for the subscription, its position in
served.position (spec/01 §4.1).