Macaw.Data.Source.Push (macaw_data v0.16.0)

View Source

One push subscription, as a value (spec/01 §4): the source's changelog, applied to the relation it holds (spec/01 §2.5).

A source that supports :push sends its subscriber three messages, each tagged with the reference subscribe/2 answered:

{:macaw_source, ref, {:schema, schema}}            # first, and again when the schema changes
{:macaw_source, ref, {:snapshot, delivery}}        # the relation, as of delivery.served.position
{:macaw_source, ref, {:increment, seq, delivery}}  # the change numbered seq

schema_message/2, snapshot_message/2 and increment_message/3 build them. The subscriber holds a Push in its state: receive/2 takes each message (a schema is cached, a snapshot or an increment buffered), snapshot/2 buffers a snapshot the holder fetched itself, and drain/2 applies everything buffered since the last pulse at once, so a burst of pushes between two pulses costs one drain. held/1 is the relation as it stands; refetch/1 says when the holder must fetch a snapshot. take_errors/1 hands the holder the errors recorded since it last took them, and pending?/1 says whether a drain has anything to apply, and trim/2 drops the chunks wholly before a sliding window's start, so a holder needs no field of the struct.

  • The handshake sends the schema first. A delivery names its schema's version; one naming a version not yet cached falls back to describe (drain/2's function, the Macaw.Data.Schema.Cache.resolve/4 describe function), asked once and cached.
  • Increments are numbered from 1 per subscription and apply in the order of their numbers (spec/01 §4.1). A repeated or older one is dropped whole and counted in ignored; one past the next is held back, the gap recorded in errors and answered by refetch/1.
  • A snapshot hands off by position: it replaces what is held and takes its served.position; increments at or before it are dropped, later ones apply. A snapshot older than the position held is dropped whole.
  • An increment's changes apply in order (spec/01 §4.2): its retracted rows, each matched by identity, held window value and the fields it names, one copy per unit of weight; then its replace ranges, everything held in each retracted; then its insertions. A change that can't be matched, or contradicts a replacement, is a fault in errors, never a guess; against a capped or reduced holding an unheld retraction is counted in ignored instead.
  • Only at the held resolution (spec/01 §4.3): an increment whose served.resolution differs from what is held is not merged, and refetch/1 answers {:resolution, held, given}.
  • An increment's rows are insertions, merged by the window field: they stay ordered by it, a late row placed among the held, and the served window grows to cover them. Rows are never deduplicated: an identical row is a second row. A schema with no window field keeps the newest limit rows, marking the delivery capped when it drops any.
  • stale is the last answer's (spec/01 §4): a snapshot's served.stale becomes the holding's, and an applied increment's replaces it, as the source answers as of that change.
  • What is held is a chunked table (spec/01 §2.6): held/1's rows are a Macaw.Data.Table of the source's origin, its columns the schema's fields, packed as Delivery.table/2 packs; every change reaches it through Macaw.Data.Table.apply/2, a settled range's in its delta. Rows that don't pack, or can't be read by name (tuple rows without names:, macaw_data#166), are {:table, faults} in errors, and nothing of that delivery is applied.

It is not a process: the process that subscribes receives the messages in its mailbox and passes them here. Ported from the logic of visualize's Visualize.Feed.Monitor, whose message tag was :visualize_feed.

Summary

Types

A buffered message: a snapshot, or an increment and its number.

t()

Functions

The schema cache the subscription resolves versions through (macaw_data#202): a holder that describes the source again hands it to new/3's cache:, so no version is asked for twice.

Everything buffered applied, in the order received: one drain per pulse, whatever the push rate. Increments apply by their numbers, a snapshot hands off by its position (spec/01 §4.1). A delivery naming a version the cache lacks is resolved through describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason kept in errors. A delivery naming no version is read under the held schema's. A delivery Delivery.new/2 accepts never raises here: rows that can't be read by name or don't pack are {:table, faults} in errors, nothing of that delivery applied.

How many snapshots and increments the last drain/2 applied (macaw_data#202): each taken and merged with no fault recorded for it. Not counted are a repeat and a stale snapshot, which ignored counts; a message whose taking recorded a fault in the errors, an increment with any faulted change among them; and increments still waiting behind a gap. An increment applied once its gap fills counts in the drain that applies it. 0 before any drain, and each drain counts afresh.

The relation as it stands, or nil before anything is held: a delivery, what was served beside its rows, which are a chunked Macaw.Data.Table of the source's origin (spec/01 §2.6), its columns the schema's fields.

The message a source sends with the increment numbered seq (from 1).

A subscription to the source named name under ref, holding nothing yet, at position 0. Options: cache: a Schema.Cache to start from, limit: the most rows held for a schema with no window field (100_000), lateness: the source's description's hard bound (spec/01 §4.4), nil by default, accepting a change of any age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default, derived from the resolution held.

As new/3, the subscription alone, raising Macaw.Data.Error naming the faults.

Whether drain/2 has anything buffered to apply (macaw_data#123): a message received or a snapshot handed in since the last drain, or increments held back waiting on a gap (spec/01 §4.1), which apply once the numbers before them, or a snapshot, arrive.

The number of the last message taken (macaw_data#202): a snapshot's position, or an increment's number, consumed whether or not its changes applied (spec/01 §4.1); 0 before any. A holder that fetches a snapshot itself hands it this position.

A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or an increment buffered; :ignore for a message that is not this subscription's.

nil, or why the holder must fetch a snapshot for this subscription and hand it to snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or {:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A snapshot taken clears it.

The message a source sends with its schema, first and on every change.

A snapshot the holder fetched for this subscription (its request naming the subscription, spec/01 §2.3), buffered as a received one is; its position is its served.position.

The message a source sends with a snapshot for the subscription, its position in served.position (spec/01 §4.1).

The errors recorded since they were last taken, in the order recorded, and the push holding none (macaw_data#123): a gap, an unresolvable schema, a fault of a change, or rows that can't be read by name or don't pack ({:table, faults}). A holder takes them after each drain, so a subscription that lives long holds only what it has not yet read.

The holding without the chunks wholly before from in the window field's order, for a holder whose window slides (macaw_data#123, spec/01 §4): {:ok, push}, or {:error, [{[:from], {:type, type}}]} for a from not of the window field's type (a time of any kind for a :time field, compared on the instant scale; a number for a :number one).

As trim/2, the push alone, raising Macaw.Data.Error naming the fault.

Types

message()

@type message() ::
  {:snapshot, Macaw.Data.Delivery.t()}
  | {:increment, pos_integer(), Macaw.Data.Delivery.t()}

A buffered message: a snapshot, or an increment and its number.

t()

@type t() :: %Macaw.Data.Source.Push{
  cache: Macaw.Data.Schema.Cache.t(),
  chunk_span: Duration.t() | number() | nil,
  drained: non_neg_integer(),
  errors: [term()],
  held: Macaw.Data.Delivery.t() | nil,
  ignored: %{required(atom()) => pos_integer()},
  lateness: Duration.t() | number() | nil,
  limit: pos_integer(),
  name: term(),
  pending: [message()],
  position: non_neg_integer(),
  ref: reference(),
  refetch: term(),
  trimmed: term(),
  waiting: %{required(pos_integer()) => Macaw.Data.Delivery.t()}
}

Functions

cache(push)

@spec cache(t()) :: Macaw.Data.Schema.Cache.t()

The schema cache the subscription resolves versions through (macaw_data#202): a holder that describes the source again hands it to new/3's cache:, so no version is asked for twice.

iex> alias Macaw.Data.Source.Push
iex> Push.cache(Push.new!(:ticks, make_ref())) == %Macaw.Data.Schema.Cache{}
true

drain(push, describe)

@spec drain(t(), Macaw.Data.Schema.Cache.describe()) :: t()

Everything buffered applied, in the order received: one drain per pulse, whatever the push rate. Increments apply by their numbers, a snapshot hands off by its position (spec/01 §4.1). A delivery naming a version the cache lacks is resolved through describe (Schema.Cache.resolve/4); one that cannot be is dropped and its reason kept in errors. A delivery naming no version is read under the held schema's. A delivery Delivery.new/2 accepts never raises here: rows that can't be read by name or don't pack are {:table, faults} in errors, nothing of that delivery applied.

drained(push)

@spec drained(t()) :: non_neg_integer()

How many snapshots and increments the last drain/2 applied (macaw_data#202): each taken and merged with no fault recorded for it. Not counted are a repeat and a stale snapshot, which ignored counts; a message whose taking recorded a fault in the errors, an increment with any faulted change among them; and increments still waiting behind a gap. An increment applied once its gap fills counts in the drain that applies it. 0 before any drain, and each drain counts afresh.

iex> alias Macaw.Data.Source.Push
iex> Push.drained(Push.new!(:ticks, make_ref()))
0

held(push)

@spec held(t()) :: Macaw.Data.Delivery.t() | nil

The relation as it stands, or nil before anything is held: a delivery, what was served beside its rows, which are a chunked Macaw.Data.Table of the source's origin (spec/01 §2.6), its columns the schema's fields.

increment_message(ref, seq, delivery)

@spec increment_message(reference(), pos_integer(), Macaw.Data.Delivery.t()) ::
  tuple()

The message a source sends with the increment numbered seq (from 1).

new(name, ref, opts \\ [])

@spec new(term(), reference(), keyword()) ::
  {:ok, t()} | {:error, [{list(), term()}, ...]}

A subscription to the source named name under ref, holding nothing yet, at position 0. Options: cache: a Schema.Cache to start from, limit: the most rows held for a schema with no window field (100_000), lateness: the source's description's hard bound (spec/01 §4.4), nil by default, accepting a change of any age; chunk_span: the description's chunk_span (spec/01 §1.1.3), nil by default, derived from the resolution held.

A value it can't use is its fault by path, every one at once, never a failure at a later drain (macaw_data#177): {[:cache], {:type, :cache}}; {[:limit], {:type, :integer}}; {[:lateness], {:type, :lateness}} for anything but nil, a number or a well-formed Duration; {[:chunk_span], {:type, :chunk_span}} for anything but nil, a positive number or a positive Duration of fixed length. There is no schema here to judge a value against: a number against a time field, or the reverse, is the delivery's fault, as ever. A limit of 0 or below holds nothing.

iex> alias Macaw.Data.Source.Push
iex> Push.new(:ticks, make_ref(), limit: "x", chunk_span: 0)
{:error, [{[:limit], {:type, :integer}}, {[:chunk_span], {:type, :chunk_span}}]}

new!(name, ref, opts \\ [])

@spec new!(term(), reference(), keyword()) :: t()

As new/3, the subscription alone, raising Macaw.Data.Error naming the faults.

pending?(push)

@spec pending?(t()) :: boolean()

Whether drain/2 has anything buffered to apply (macaw_data#123): a message received or a snapshot handed in since the last drain, or increments held back waiting on a gap (spec/01 §4.1), which apply once the numbers before them, or a snapshot, arrive.

iex> alias Macaw.Data.Source.Push
iex> Push.pending?(Push.new!(:ticks, make_ref()))
false

position(push)

@spec position(t()) :: non_neg_integer()

The number of the last message taken (macaw_data#202): a snapshot's position, or an increment's number, consumed whether or not its changes applied (spec/01 §4.1); 0 before any. A holder that fetches a snapshot itself hands it this position.

iex> alias Macaw.Data.Source.Push
iex> Push.position(Push.new!(:ticks, make_ref()))
0

receive(push, arg2)

@spec receive(t(), term()) :: {:ok, t()} | :ignore

A message the subscriber received: {:ok, push} with a schema cached, or a snapshot or an increment buffered; :ignore for a message that is not this subscription's.

refetch(push)

@spec refetch(t()) :: term()

nil, or why the holder must fetch a snapshot for this subscription and hand it to snapshot/2 (spec/01 §4.1): {:gap, from, to}, the numbers that never arrived, or {:resolution, held, given}, an increment at another resolution (spec/01 §4.3). A snapshot taken clears it.

schema_message(ref, schema)

@spec schema_message(reference(), Macaw.Data.Schema.t()) :: tuple()

The message a source sends with its schema, first and on every change.

snapshot(push, delivery)

@spec snapshot(t(), Macaw.Data.Delivery.t()) :: t()

A snapshot the holder fetched for this subscription (its request naming the subscription, spec/01 §2.3), buffered as a received one is; its position is its served.position.

snapshot_message(ref, delivery)

@spec snapshot_message(reference(), Macaw.Data.Delivery.t()) :: tuple()

The message a source sends with a snapshot for the subscription, its position in served.position (spec/01 §4.1).

take_errors(push)

@spec take_errors(t()) :: {[term()], t()}

The errors recorded since they were last taken, in the order recorded, and the push holding none (macaw_data#123): a gap, an unresolvable schema, a fault of a change, or rows that can't be read by name or don't pack ({:table, faults}). A holder takes them after each drain, so a subscription that lives long holds only what it has not yet read.

iex> alias Macaw.Data.Source.Push
iex> {[], push} = Push.take_errors(Push.new!(:ticks, make_ref()))
iex> push.errors
[]

trim(push, from)

@spec trim(t(), term()) ::
  {:ok, t()} | {:error, [{[:from], {:type, :time | :number}}]}

The holding without the chunks wholly before from in the window field's order, for a holder whose window slides (macaw_data#123, spec/01 §4): {:ok, push}, or {:error, [{[:from], {:type, type}}]} for a from not of the window field's type (a time of any kind for a :time field, compared on the instant scale; a number for a :number one).

  • Only whole chunks are dropped: each held chunk whose every window value is before from, through Macaw.Data.Table.trim/2 over their rows, so nothing is copied or cut and every chunk kept is the same chunk, by id and version. A chunk straddling from is kept whole; finishing trims its rows.
  • The served window is narrowed, never widened: it starts at from, or at the held start where that is later; its upper end and complete_until are unchanged. A box has no start to narrow, so it stays as it is.
  • The trimmed start is kept (spec/01 §4.2): after it, a retraction naming a row before it that is not held, or a replace range wholly before it, is counted in ignored as :trimmed, not recorded as a fault. A late insertion before it is still placed; the next trim drops it. A snapshot taken replaces the holding and clears it.

It is a no-op {:ok, push} when nothing is held, when the schema has no window field, for a :point window field, which has no order, and for a nil from, an open start.

iex> alias Macaw.Data.Source.Push
iex> push = Push.new!(:ticks, make_ref())
iex> Push.trim(push, 10) == {:ok, push}
true

trim!(push, from)

@spec trim!(t(), term()) :: t()

As trim/2, the push alone, raising Macaw.Data.Error naming the fault.