Macaw.Data.Monitor (macaw_data v0.18.0)

View Source

The loop over a set of sources, as a value (spec/01 §6.3): subscribe where a source pushes, poll where it cannot, ask each source with a Macaw.Data.Source.Request whose now is the owner's clock, finish each answer, and hand each reader one finished Macaw.Data.Delivery, pooled at the largest budget and each reduced to its own (spec/01 §6.1).

It is not a process. The process that owns the sources (a page's, a LiveView's, a board's) holds the monitor in its state, calls tick/2 each frame and handle_message/2 from its handle_info/2, and reads deliveries/1 and status/1. The process that made the monitor is the one subscribed, whichever process ticks it, so the pushes arrive in its mailbox. The monitor runs nothing of its own: describe, fetch and subscribe are the source contract's (spec/01 §2), called in the caller.

  • Sources are named, each a source (described with no context) or {source, context}. Readers are named, each reading one source with its request fields (ask): window, box, region, latest, order, params, as_of, viewer, budget and reduce. A request's now is tick/2's at:; a subscription is the monitor's.
  • Readers of one source asking one request but for their budgets and reducers are one pool: one subscription, one holding, one fetch at Macaw.Data.Source.Finish.pooled/1 of their requests, the holding finished once for the request, then each reader reduced its own way (Macaw.Data.Source.Finish.reduce/5 with its request) through the pool's Macaw.Data.Operations.Cache, so a reduction costs the chunks that changed.
  • A pool's holding is a Macaw.Data.Source.Push: a fetch is handed to it as a snapshot by its position, and every message buffers in it until the next tick drains them: one drain, one trim and one finishing per pool per tick, whatever the push rate. A gap or a change of resolution is answered by a fetch. A sliding window drops the holding's chunks before it. A pull pool is fetched every every: milliseconds.
  • Errors and staleness are states, never raises: status/1 is each reader's :loading, :live, :stale or {:down, reason}. A source that fails to describe, subscribe or fetch is asked again after every:, doubled at each consecutive failure up to backoff_max:; one live and unanswered past stale_after: is :stale.

Ported from visualize's Visualize.Feed.Monitor, its slots become readers.

Examples

iex> alias Macaw.Data.Monitor
iex> alias Macaw.Data.Source.Signal
iex> sources = %{wave: Signal.new!(:sine, [:t, :v], window: 100, tick: 500)}
iex> readers = %{chart: [source: :wave, ask: [budget: 20]], table: [source: :wave, ask: [budget: 50]]}
iex> {:ok, monitor} = Monitor.new(sources, readers)
iex> {:ok, monitor} = Monitor.tick(monitor, now: 0, at: ~U[2026-11-01 00:00:00Z])
iex> Monitor.status(monitor)
%{chart: :live, table: :live}
iex> %{chart: chart, table: table} = Monitor.deliveries(monitor)
iex> {Macaw.Data.Table.size(chart.rows), Macaw.Data.Table.size(table.rows), chart.served.reduced_by}
{20, 50, :runtime}

Summary

Types

A source's or a reader's name: any term.

A reader: the source it reads and its request fields.

A source: described with no context, or a source and its context.

t()

Functions

The monitor with readers' requests changed, %{reader => fields} as new/3 takes a reader's ask: {:ok, monitor} or the faults under [:readers, name]. A reader whose request but for budget and reducer moved leaves its pool for the pool of its new request. A pool all of whose readers move together keeps its subscription and holding, and is fetched again at the next tick unless it pushes and what it holds answers the new window: its window field's range starting within what it holds, everything else asked the same, and under latest a to no earlier. A budget or reducer that changed refinishes from the holding, and refetches where the source honors :budget and the pool's largest budget moved.

The monitor with readers' budgets changed, %{reader => budget}, each nil or a positive integer: ask/2 with only the budget moved.

The schema cache, holding every version the deliveries name, by source name.

Each reader's finished delivery, for the readers that have one.

Each reader's pool's last error that left it live, for the readers that have one.

A reason's words (spec/00 §5): this module's, else the library's.

A message the owner received: {:ok, monitor} with a pushed schema cached, or a snapshot or an increment buffered in its pool's holding for the next tick; :ignore for a message that is none of the source contract's (spec/01 §4). A push of a subscription the monitor no longer holds is dropped.

Each reader's holding, for the readers whose pool holds one: the delivery its push merge keeps (Macaw.Data.Source.Push.held/1), shared by the pool's readers. deliveries/1 is it finished.

A monitor over sources and readers by name, calling nothing: {:ok, monitor}, or every fault at once (spec/00 §5). Options: every: (a pull pool's poll interval and the first retry, milliseconds, 1000), backoff_max: (the longest retry, 30_000), stale_after: (milliseconds without an answer before a live reader is :stale, nil for never) and limit: (the most rows held for a source whose schema has no window field, Macaw.Data.Source.Push.new/3's, 100_000).

As new/3, the monitor alone, raising Macaw.Data.Error naming the faults.

The monitor with a reader added or replaced, as new/3 takes one: {:ok, monitor} or its faults under [:readers, name].

The monitor with a source added or replaced under a name, described at the next tick: {:ok, monitor}, or {[:sources, name], {:type, :source}}. A replaced source's pools are unsubscribed and start again, its readers kept.

Every pool fetched again at the next tick, push pools included, whatever it holds.

The monitor without a reader; its pool unsubscribed once it has none.

The monitor without a source and its readers, its pools unsubscribed.

Each reader's state: :loading until its first answer, :live, :stale past stale_after:, or {:down, reason}: its source's describe error, a request its schema cannot answer ({:request, faults}), a subscribe or fetch error, a snapshot the holding could not take, or a finishing's faults ({:finish, faults}).

Every subscription stopped; the monitor's end.

One turn of the loop at now: (the owner's monotonic clock in milliseconds, which polls, retries and staleness are counted on) and at: (the owner's wall clock, a DateTime, every request's now): {:ok, monitor}, or {[:now], {:type, :integer}}, {[:at], {:type, :datetime}} and {[key], :unknown_key}, every one at once, with nothing done. Every source, in name order, is described if it is not yet and its retry is due; then every pool of it is subscribed where it pushes, fetched if it holds nothing, its push asks a refetch, its request moved or its poll is due, its buffered pushes drained into its holding, its chunks before the window dropped, and finished for each reader when it changed.

As tick/2, the monitor alone, raising Macaw.Data.Error naming the faults.

Types

name()

@type name() :: term()

A source's or a reader's name: any term.

reader()

@type reader() ::
  [source: name(), ask: keyword() | map()]
  | %{source: name(), ask: keyword() | map()}

A reader: the source it reads and its request fields.

source()

@type source() :: Macaw.Data.Source.t() | {Macaw.Data.Source.t(), term()}

A source: described with no context, or a source and its context.

status()

@type status() :: :loading | :live | :stale | {:down, term()}

t()

@type t() :: %Macaw.Data.Monitor{
  backoff_max: pos_integer(),
  cache: Macaw.Data.Schema.Cache.t(),
  every: non_neg_integer(),
  limit: integer(),
  owner: pid(),
  pools: %{required(term()) => map()},
  readers: %{required(name()) => map()},
  refs: %{required(reference()) => term()},
  sources: %{required(name()) => map()},
  stale_after: pos_integer() | nil
}

Functions

ask(monitor, asks)

@spec ask(t(), term()) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

The monitor with readers' requests changed, %{reader => fields} as new/3 takes a reader's ask: {:ok, monitor} or the faults under [:readers, name]. A reader whose request but for budget and reducer moved leaves its pool for the pool of its new request. A pool all of whose readers move together keeps its subscription and holding, and is fetched again at the next tick unless it pushes and what it holds answers the new window: its window field's range starting within what it holds, everything else asked the same, and under latest a to no earlier. A budget or reducer that changed refinishes from the holding, and refetches where the source honors :budget and the pool's largest budget moved.

budget(monitor, budgets)

@spec budget(t(), term()) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

The monitor with readers' budgets changed, %{reader => budget}, each nil or a positive integer: ask/2 with only the budget moved.

cache(monitor)

@spec cache(t()) :: Macaw.Data.Schema.Cache.t()

The schema cache, holding every version the deliveries name, by source name.

deliveries(monitor)

@spec deliveries(t()) :: %{required(name()) => Macaw.Data.Delivery.t()}

Each reader's finished delivery, for the readers that have one.

errors(monitor)

@spec errors(t()) :: %{required(name()) => term()}

Each reader's pool's last error that left it live, for the readers that have one.

format_error(reason)

@spec format_error(term()) :: String.t()

A reason's words (spec/00 §5): this module's, else the library's.

handle_message(monitor, received)

@spec handle_message(t(), term()) :: {:ok, t()} | :ignore

A message the owner received: {:ok, monitor} with a pushed schema cached, or a snapshot or an increment buffered in its pool's holding for the next tick; :ignore for a message that is none of the source contract's (spec/01 §4). A push of a subscription the monitor no longer holds is dropped.

held(monitor)

@spec held(t()) :: %{required(name()) => Macaw.Data.Delivery.t()}

Each reader's holding, for the readers whose pool holds one: the delivery its push merge keeps (Macaw.Data.Source.Push.held/1), shared by the pool's readers. deliveries/1 is it finished.

new(sources, readers, opts \\ [])

@spec new(term(), term(), keyword()) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

A monitor over sources and readers by name, calling nothing: {:ok, monitor}, or every fault at once (spec/00 §5). Options: every: (a pull pool's poll interval and the first retry, milliseconds, 1000), backoff_max: (the longest retry, 30_000), stale_after: (milliseconds without an answer before a live reader is :stale, nil for never) and limit: (the most rows held for a source whose schema has no window field, Macaw.Data.Source.Push.new/3's, 100_000).

The faults: {[:sources], {:type, :map}} and {[:readers], {:type, :map}}; {[:sources, name], {:type, :source}} for an entry that is no struct nor {struct, context}; {[:readers, name], {:type, :reader}} for a reader that is no keyword list or map of source and ask; {[:readers, name, :source], {:unknown_source, source}} for a source the monitor lacks; {[:readers, name, :ask, key], :unknown_key} for a field a reader does not ask, and Macaw.Data.Source.Request.new/1's faults under [:readers, name, :ask]; and an option's {[key], {:type, type}} or {[key], :unknown_key}.

new!(sources, readers, opts \\ [])

@spec new!(term(), term(), keyword()) :: t()

As new/3, the monitor alone, raising Macaw.Data.Error naming the faults.

put_reader(monitor, name, given)

@spec put_reader(t(), name(), term()) ::
  {:ok, t()} | {:error, [{list(), term()}, ...]}

The monitor with a reader added or replaced, as new/3 takes one: {:ok, monitor} or its faults under [:readers, name].

put_source(monitor, name, given)

@spec put_source(t(), name(), term()) ::
  {:ok, t()} | {:error, [{list(), term()}, ...]}

The monitor with a source added or replaced under a name, described at the next tick: {:ok, monitor}, or {[:sources, name], {:type, :source}}. A replaced source's pools are unsubscribed and start again, its readers kept.

refetch(monitor)

@spec refetch(t()) :: t()

Every pool fetched again at the next tick, push pools included, whatever it holds.

remove_reader(monitor, name)

@spec remove_reader(t(), name()) :: t()

The monitor without a reader; its pool unsubscribed once it has none.

remove_source(monitor, name)

@spec remove_source(t(), name()) :: t()

The monitor without a source and its readers, its pools unsubscribed.

status(monitor)

@spec status(t()) :: %{required(name()) => status()}

Each reader's state: :loading until its first answer, :live, :stale past stale_after:, or {:down, reason}: its source's describe error, a request its schema cannot answer ({:request, faults}), a subscribe or fetch error, a snapshot the holding could not take, or a finishing's faults ({:finish, faults}).

stop(monitor)

@spec stop(t()) :: :ok

Every subscription stopped; the monitor's end.

tick(monitor, opts)

@spec tick(
  t(),
  keyword()
) :: {:ok, t()} | {:error, [{list(), term()}, ...]}

One turn of the loop at now: (the owner's monotonic clock in milliseconds, which polls, retries and staleness are counted on) and at: (the owner's wall clock, a DateTime, every request's now): {:ok, monitor}, or {[:now], {:type, :integer}}, {[:at], {:type, :datetime}} and {[key], :unknown_key}, every one at once, with nothing done. Every source, in name order, is described if it is not yet and its retry is due; then every pool of it is subscribed where it pushes, fetched if it holds nothing, its push asks a refetch, its request moved or its poll is due, its buffered pushes drained into its holding, its chunks before the window dropped, and finished for each reader when it changed.

tick!(monitor, opts)

@spec tick!(
  t(),
  keyword()
) :: t()

As tick/2, the monitor alone, raising Macaw.Data.Error naming the faults.