Macaw.Data.Source.Push (macaw_data v0.9.0)

View Source

One push subscription, as a value (spec/01 §4): the source's changelog, applied to the relation it holds (spec/01 §2.5).

A source that supports :push sends its subscriber three messages, each tagged with the reference subscribe/2 answered:

{:macaw_source, ref, {:schema, schema}}            # first, and again when the schema changes
{:macaw_source, ref, {:snapshot, delivery}}        # the relation, as of delivery.served.position
{:macaw_source, ref, {:increment, seq, delivery}}  # the change numbered seq

schema_message/2, snapshot_message/2 and increment_message/3 build them. The subscriber holds a Push in its state: receive/2 takes each message (a schema is cached, a snapshot or an increment buffered), snapshot/2 buffers a snapshot the holder fetched itself, and drain/2 applies everything buffered since the last pulse at once, so a burst of pushes between two pulses costs one drain. held/1 is the relation as it stands; refetch/1 says when the holder must fetch a snapshot.

  • The handshake sends the schema first. A delivery names its schema's version; one naming a version not yet cached falls back to describe (drain/2's function, the Macaw.Data.Schema.Cache.resolve/4 describe function), asked once and cached.
  • Increments are numbered from 1 per subscription and apply in the order of their numbers (spec/01 §4.1). A repeated or older one is dropped whole and counted in ignored; one past the next is held back, the gap recorded in errors and answered by refetch/1.
  • A snapshot hands off by position: it replaces what is held and takes its served.position; increments at or before it are dropped, later ones apply. A snapshot older than the position held is dropped whole.
  • An increment's changes apply in order (spec/01 §4.2): its retracted rows, each matched by identity, held window value and the fields it names, one copy per unit of weight; then its replace ranges, everything held in each retracted; then its insertions. A change that can't be matched, or contradicts a replacement, is a fault in errors, never a guess; against a capped or reduced holding an unheld retraction is counted in ignored instead.
  • Only at the held resolution (spec/01 §4.3): an increment whose served.resolution differs from what is held is not merged, and refetch/1 answers {:resolution, held, given}.
  • An increment's rows are insertions, merged by the window field: they stay ordered by it, a late row placed among the held, and the served window grows to cover them. Rows are never deduplicated: an identical row is a second row. A schema with no window field keeps the newest limit rows, marking the delivery capped when it drops any.
  • What is held is a chunked table (spec/01 §2.6): held/1's rows are a Macaw.Data.Table of the source's origin, its columns the schema's fields, packed as Delivery.table/2 packs; every change reaches it through Macaw.Data.Table.apply/2, a settled range's in its delta. Rows that don't pack are {:table, faults} in errors.

It is not a process: the process that subscribes receives the messages in its mailbox and passes them here. Ported from the logic of visualize's Visualize.Feed.Monitor, whose message tag was :visualize_feed.

Summary

Types

A buffered message: a snapshot, or an increment and its number.

t()

Functions

Everything buffered applied, in the order received: one drain per pulse, whatever the push rate. Increments apply by their numbers, a snapshot hands off by its position (spec/01 §4.1). A delivery naming a version the cache lacks is resolved through describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason kept in errors. A delivery naming no version is read under the held schema's.

The relation as it stands, or nil before anything is held: a delivery, what was served beside its rows, which are a chunked Macaw.Data.Table of the source's origin (spec/01 §2.6), its columns the schema's fields.

The message a source sends with the increment numbered seq (from 1).

A subscription to the source named name under ref, holding nothing yet, at position 0. Options: cache: a Schema.Cache to start from, limit: the most rows held for a schema with no window field (100_000), lateness: the source's description's hard bound (spec/01 §4.4), nil by default, accepting a change of any age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default, derived from the resolution held.

A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or an increment buffered; :ignore for a message that is not this subscription's.

nil, or why the holder must fetch a snapshot for this subscription and hand it to snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or {:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A snapshot taken clears it.

The message a source sends with its schema, first and on every change.

A snapshot the holder fetched for this subscription (its request naming the subscription, spec/01 §2.3), buffered as a received one is; its position is its served.position.

The message a source sends with a snapshot for the subscription, its position in served.position (spec/01 §4.1).

Types

message()

@type message() ::
  {:snapshot, Macaw.Data.Delivery.t()}
  | {:increment, pos_integer(), Macaw.Data.Delivery.t()}

A buffered message: a snapshot, or an increment and its number.

t()

@type t() :: %Macaw.Data.Source.Push{
  cache: Macaw.Data.Schema.Cache.t(),
  chunk_span: Duration.t() | number() | nil,
  errors: [term()],
  held: Macaw.Data.Delivery.t() | nil,
  ignored: %{required(atom()) => pos_integer()},
  lateness: Duration.t() | number() | nil,
  limit: pos_integer(),
  name: term(),
  pending: [message()],
  position: non_neg_integer(),
  ref: reference(),
  refetch: term(),
  waiting: %{required(pos_integer()) => Macaw.Data.Delivery.t()}
}

Functions

drain(push, describe)

@spec drain(t(), Macaw.Data.Schema.Cache.describe()) :: t()

Everything buffered applied, in the order received: one drain per pulse, whatever the push rate. Increments apply by their numbers, a snapshot hands off by its position (spec/01 §4.1). A delivery naming a version the cache lacks is resolved through describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason kept in errors. A delivery naming no version is read under the held schema's.

held(push)

@spec held(t()) :: Macaw.Data.Delivery.t() | nil

The relation as it stands, or nil before anything is held: a delivery, what was served beside its rows, which are a chunked Macaw.Data.Table of the source's origin (spec/01 §2.6), its columns the schema's fields.

increment_message(ref, seq, delivery)

@spec increment_message(reference(), pos_integer(), Macaw.Data.Delivery.t()) ::
  tuple()

The message a source sends with the increment numbered seq (from 1).

new(name, ref, opts \\ [])

@spec new(term(), reference(), keyword()) :: t()

A subscription to the source named name under ref, holding nothing yet, at position 0. Options: cache: a Schema.Cache to start from, limit: the most rows held for a schema with no window field (100_000), lateness: the source's description's hard bound (spec/01 §4.4), nil by default, accepting a change of any age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default, derived from the resolution held.

receive(push, arg2)

@spec receive(t(), term()) :: {:ok, t()} | :ignore

A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or an increment buffered; :ignore for a message that is not this subscription's.

refetch(push)

@spec refetch(t()) :: term()

nil, or why the holder must fetch a snapshot for this subscription and hand it to snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or {:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A snapshot taken clears it.

schema_message(ref, schema)

@spec schema_message(reference(), Macaw.Data.Schema.t()) :: tuple()

The message a source sends with its schema, first and on every change.

snapshot(push, delivery)

@spec snapshot(t(), Macaw.Data.Delivery.t()) :: t()

A snapshot the holder fetched for this subscription (its request naming the subscription, spec/01 §2.3), buffered as a received one is; its position is its served.position.

snapshot_message(ref, delivery)

@spec snapshot_message(reference(), Macaw.Data.Delivery.t()) :: tuple()

The message a source sends with a snapshot for the subscription, its position in served.position (spec/01 §4.1).