Macaw.Data.Monitor (macaw_data v0.18.0)
View SourceThe loop over a set of sources, as a value (spec/01 §6.3): subscribe where a source
pushes, poll where it cannot, ask each source with a Macaw.Data.Source.Request whose
now is the owner's clock, finish each answer, and hand each reader one finished
Macaw.Data.Delivery, pooled at the largest budget and each reduced to its own
(spec/01 §6.1).
It is not a process. The process that owns the sources (a page's, a LiveView's, a
board's) holds the monitor in its state, calls tick/2 each frame and
handle_message/2 from its handle_info/2, and reads deliveries/1 and status/1.
The process that made the monitor is the one subscribed, whichever process ticks it, so
the pushes arrive in its mailbox. The monitor runs nothing of its own: describe, fetch
and subscribe are the source contract's (spec/01 §2), called in the caller.
- Sources are named, each a source (described with no context) or
{source, context}. Readers are named, each reading one source with its request fields (ask):window,box,region,latest,order,params,as_of,viewer,budgetandreduce. A request'snowistick/2'sat:; asubscriptionis the monitor's. - Readers of one source asking one request but for their budgets and reducers are one
pool: one subscription, one holding, one fetch at
Macaw.Data.Source.Finish.pooled/1of their requests, the holding finished once for the request, then each reader reduced its own way (Macaw.Data.Source.Finish.reduce/5with its request) through the pool'sMacaw.Data.Operations.Cache, so a reduction costs the chunks that changed. - A pool's holding is a
Macaw.Data.Source.Push: a fetch is handed to it as a snapshot by its position, and every message buffers in it until the next tick drains them: one drain, one trim and one finishing per pool per tick, whatever the push rate. A gap or a change of resolution is answered by a fetch. A sliding window drops the holding's chunks before it. A pull pool is fetched everyevery:milliseconds. - Errors and staleness are states, never raises:
status/1is each reader's:loading,:live,:staleor{:down, reason}. A source that fails to describe, subscribe or fetch is asked again afterevery:, doubled at each consecutive failure up tobackoff_max:; one live and unanswered paststale_after:is:stale.
Ported from visualize's Visualize.Feed.Monitor, its slots become readers.
Examples
iex> alias Macaw.Data.Monitor
iex> alias Macaw.Data.Source.Signal
iex> sources = %{wave: Signal.new!(:sine, [:t, :v], window: 100, tick: 500)}
iex> readers = %{chart: [source: :wave, ask: [budget: 20]], table: [source: :wave, ask: [budget: 50]]}
iex> {:ok, monitor} = Monitor.new(sources, readers)
iex> {:ok, monitor} = Monitor.tick(monitor, now: 0, at: ~U[2026-11-01 00:00:00Z])
iex> Monitor.status(monitor)
%{chart: :live, table: :live}
iex> %{chart: chart, table: table} = Monitor.deliveries(monitor)
iex> {Macaw.Data.Table.size(chart.rows), Macaw.Data.Table.size(table.rows), chart.served.reduced_by}
{20, 50, :runtime}
Summary
Types
A source's or a reader's name: any term.
A reader: the source it reads and its request fields.
A source: described with no context, or a source and its context.
Functions
The monitor with readers' requests changed, %{reader => fields} as new/3 takes a
reader's ask: {:ok, monitor} or the faults under [:readers, name]. A reader whose
request but for budget and reducer moved leaves its pool for the pool of its new
request. A pool all of whose readers move together keeps its subscription and holding,
and is fetched again at the next tick unless it pushes and what it holds answers the
new window: its window field's range starting within what it holds, everything else
asked the same, and under latest a to no earlier. A budget or reducer that changed
refinishes from the holding, and refetches where the source honors :budget and the
pool's largest budget moved.
The monitor with readers' budgets changed, %{reader => budget}, each nil or a
positive integer: ask/2 with only the budget moved.
The schema cache, holding every version the deliveries name, by source name.
Each reader's finished delivery, for the readers that have one.
Each reader's pool's last error that left it live, for the readers that have one.
A reason's words (spec/00 §5): this module's, else the library's.
A message the owner received: {:ok, monitor} with a pushed schema cached, or a
snapshot or an increment buffered in its pool's holding for the next tick; :ignore
for a message that is none of the source contract's (spec/01 §4). A push of a
subscription the monitor no longer holds is dropped.
Each reader's holding, for the readers whose pool holds one: the delivery its push merge
keeps (Macaw.Data.Source.Push.held/1), shared by the pool's readers. deliveries/1 is
it finished.
A monitor over sources and readers by name, calling nothing: {:ok, monitor}, or every
fault at once (spec/00 §5). Options: every: (a pull pool's poll interval and the first
retry, milliseconds, 1000), backoff_max: (the longest retry, 30_000),
stale_after: (milliseconds without an answer before a live reader is :stale, nil
for never) and limit: (the most rows held for a source whose schema has no window
field, Macaw.Data.Source.Push.new/3's, 100_000).
As new/3, the monitor alone, raising Macaw.Data.Error naming the faults.
The monitor with a reader added or replaced, as new/3 takes one: {:ok, monitor} or
its faults under [:readers, name].
The monitor with a source added or replaced under a name, described at the next tick:
{:ok, monitor}, or {[:sources, name], {:type, :source}}. A replaced source's pools
are unsubscribed and start again, its readers kept.
Every pool fetched again at the next tick, push pools included, whatever it holds.
The monitor without a reader; its pool unsubscribed once it has none.
The monitor without a source and its readers, its pools unsubscribed.
Each reader's state: :loading until its first answer, :live, :stale past
stale_after:, or {:down, reason}: its source's describe error, a request its schema
cannot answer ({:request, faults}), a subscribe or fetch error, a snapshot the holding
could not take, or a finishing's faults ({:finish, faults}).
Every subscription stopped; the monitor's end.
One turn of the loop at now: (the owner's monotonic clock in milliseconds, which polls,
retries and staleness are counted on) and at: (the owner's wall clock, a DateTime,
every request's now): {:ok, monitor}, or {[:now], {:type, :integer}}, {[:at], {:type, :datetime}} and {[key], :unknown_key}, every one at once, with nothing done.
Every source, in name order, is described if it is not yet and its retry is due; then
every pool of it is subscribed where it pushes, fetched if it holds nothing, its push
asks a refetch, its request moved or its poll is due, its buffered pushes drained into
its holding, its chunks before the window dropped, and finished for each reader when it
changed.
As tick/2, the monitor alone, raising Macaw.Data.Error naming the faults.
Types
@type name() :: term()
A source's or a reader's name: any term.
@type reader() :: [source: name(), ask: keyword() | map()] | %{source: name(), ask: keyword() | map()}
A reader: the source it reads and its request fields.
@type source() :: Macaw.Data.Source.t() | {Macaw.Data.Source.t(), term()}
A source: described with no context, or a source and its context.
@type status() :: :loading | :live | :stale | {:down, term()}
@type t() :: %Macaw.Data.Monitor{ backoff_max: pos_integer(), cache: Macaw.Data.Schema.Cache.t(), every: non_neg_integer(), limit: integer(), owner: pid(), pools: %{required(term()) => map()}, readers: %{required(name()) => map()}, refs: %{required(reference()) => term()}, sources: %{required(name()) => map()}, stale_after: pos_integer() | nil }
Functions
The monitor with readers' requests changed, %{reader => fields} as new/3 takes a
reader's ask: {:ok, monitor} or the faults under [:readers, name]. A reader whose
request but for budget and reducer moved leaves its pool for the pool of its new
request. A pool all of whose readers move together keeps its subscription and holding,
and is fetched again at the next tick unless it pushes and what it holds answers the
new window: its window field's range starting within what it holds, everything else
asked the same, and under latest a to no earlier. A budget or reducer that changed
refinishes from the holding, and refetches where the source honors :budget and the
pool's largest budget moved.
The monitor with readers' budgets changed, %{reader => budget}, each nil or a
positive integer: ask/2 with only the budget moved.
@spec cache(t()) :: Macaw.Data.Schema.Cache.t()
The schema cache, holding every version the deliveries name, by source name.
@spec deliveries(t()) :: %{required(name()) => Macaw.Data.Delivery.t()}
Each reader's finished delivery, for the readers that have one.
Each reader's pool's last error that left it live, for the readers that have one.
A reason's words (spec/00 §5): this module's, else the library's.
A message the owner received: {:ok, monitor} with a pushed schema cached, or a
snapshot or an increment buffered in its pool's holding for the next tick; :ignore
for a message that is none of the source contract's (spec/01 §4). A push of a
subscription the monitor no longer holds is dropped.
@spec held(t()) :: %{required(name()) => Macaw.Data.Delivery.t()}
Each reader's holding, for the readers whose pool holds one: the delivery its push merge
keeps (Macaw.Data.Source.Push.held/1), shared by the pool's readers. deliveries/1 is
it finished.
A monitor over sources and readers by name, calling nothing: {:ok, monitor}, or every
fault at once (spec/00 §5). Options: every: (a pull pool's poll interval and the first
retry, milliseconds, 1000), backoff_max: (the longest retry, 30_000),
stale_after: (milliseconds without an answer before a live reader is :stale, nil
for never) and limit: (the most rows held for a source whose schema has no window
field, Macaw.Data.Source.Push.new/3's, 100_000).
The faults: {[:sources], {:type, :map}} and {[:readers], {:type, :map}};
{[:sources, name], {:type, :source}} for an entry that is no struct nor {struct, context}; {[:readers, name], {:type, :reader}} for a reader that is no keyword list
or map of source and ask; {[:readers, name, :source], {:unknown_source, source}}
for a source the monitor lacks; {[:readers, name, :ask, key], :unknown_key} for a
field a reader does not ask, and Macaw.Data.Source.Request.new/1's faults under
[:readers, name, :ask]; and an option's {[key], {:type, type}} or {[key], :unknown_key}.
As new/3, the monitor alone, raising Macaw.Data.Error naming the faults.
The monitor with a reader added or replaced, as new/3 takes one: {:ok, monitor} or
its faults under [:readers, name].
The monitor with a source added or replaced under a name, described at the next tick:
{:ok, monitor}, or {[:sources, name], {:type, :source}}. A replaced source's pools
are unsubscribed and start again, its readers kept.
Every pool fetched again at the next tick, push pools included, whatever it holds.
The monitor without a reader; its pool unsubscribed once it has none.
The monitor without a source and its readers, its pools unsubscribed.
Each reader's state: :loading until its first answer, :live, :stale past
stale_after:, or {:down, reason}: its source's describe error, a request its schema
cannot answer ({:request, faults}), a subscribe or fetch error, a snapshot the holding
could not take, or a finishing's faults ({:finish, faults}).
@spec stop(t()) :: :ok
Every subscription stopped; the monitor's end.
One turn of the loop at now: (the owner's monotonic clock in milliseconds, which polls,
retries and staleness are counted on) and at: (the owner's wall clock, a DateTime,
every request's now): {:ok, monitor}, or {[:now], {:type, :integer}}, {[:at], {:type, :datetime}} and {[key], :unknown_key}, every one at once, with nothing done.
Every source, in name order, is described if it is not yet and its retry is due; then
every pool of it is subscribed where it pushes, fetched if it holds nothing, its push
asks a refetch, its request moved or its poll is due, its buffered pushes drained into
its holding, its chunks before the window dropped, and finished for each reader when it
changed.
As tick/2, the monitor alone, raising Macaw.Data.Error naming the faults.