Macaw.Data.Operations.Cache (macaw_data v0.13.0)
View SourceA caller's cache of steps' per-chunk parts (spec/04 §1.5; macaw_data#9 §6): the example caller of the caching contract, a plain value the caller holds and passes back. The operations hold no cache and no hidden state; the frame or the feed that owns this value knows when a chunk leaves its window.
The key of a part is {logical id, version, step key}: Chunk.id/1, where the
chunk sits, which scopes eviction; Chunk.version/1, its content digest (or a step
output's derivation digest), which decides validity; and the part's step key
(Macaw.Data.Operations.Step.parts/4), which carries the context as it resolved for
the chunk. Nothing is hashed on lookup: the version is read from the chunk, computed
once when its state was made, and a chunk whose digest is pending has none, so it
never matches.
run/5 runs a step over a table a part at a time:
- a part held under its key is reused;
- a chunk with a delta whose base's part is held (under
Chunk.base_version/1) is updated from it, where the step's inverse is exact (Step.update/4: counts,bincounts, integer sums); - any other part is computed (
Step.chunk/3);
and the parts are merged (Step.merge/6), which only the combine's or global step's
merge recomputes. The result equals Step.run/4's. prune/2 evicts by logical id.
cache = Cache.new()
{:ok, table, cache} = Cache.run(cache, Sum, node, frame_1, %{})
# the next frame: one new chunk computed, every other part reused
{:ok, table, cache} = Cache.run(cache, Sum, node, frame_2, %{})
cache = Cache.prune(cache, [frame_2])
Summary
Types
The parts held by {id, version, step key}, and how the last run came by its parts:
how many it reused, updated by inverse and computed.
Functions
An empty cache.
The cache with only the parts of chunks the tables still hold (spec/04 §1.5): the
logical id scopes eviction, so a part of a chunk that left the window goes, and so does
a part of a version no held chunk has, but for a chunk's base version, which its inverse
updates start from.
The step module (a column-native one, Step.step?/1) run over table with the parts
cache holds: {:ok, table, cache}, the cache holding this run's parts, or the step's
fault, {:error, {key, reason}}, with the cache unchanged (spec/04 §3.1). The table
equals Step.run(module, node, table, context)'s. With parallel: true in the context,
the parts it computes run as Step.run/4's would (spec/04 §1.6).
As run/5, {table, cache}, raising Macaw.Data.Error naming the fault.
How many parts the cache holds.
Types
@type t() :: %Macaw.Data.Operations.Cache{ last: %{ reused: non_neg_integer(), updated: non_neg_integer(), computed: non_neg_integer() }, parts: %{required({term(), String.t(), term()}) => term()} }
The parts held by {id, version, step key}, and how the last run came by its parts:
how many it reused, updated by inverse and computed.
Functions
@spec new() :: t()
An empty cache.
@spec prune(t(), [Macaw.Data.Table.t()]) :: t()
The cache with only the parts of chunks the tables still hold (spec/04 §1.5): the
logical id scopes eviction, so a part of a chunk that left the window goes, and so does
a part of a version no held chunk has, but for a chunk's base version, which its inverse
updates start from.
@spec run(t(), module(), map(), Macaw.Data.Table.t(), map()) :: {:ok, Macaw.Data.Table.t(), t()} | {:error, Macaw.Data.Operations.Step.fault()}
The step module (a column-native one, Step.step?/1) run over table with the parts
cache holds: {:ok, table, cache}, the cache holding this run's parts, or the step's
fault, {:error, {key, reason}}, with the cache unchanged (spec/04 §3.1). The table
equals Step.run(module, node, table, context)'s. With parallel: true in the context,
the parts it computes run as Step.run/4's would (spec/04 §1.6).
@spec run!(t(), module(), map(), Macaw.Data.Table.t(), map()) :: {Macaw.Data.Table.t(), t()}
As run/5, {table, cache}, raising Macaw.Data.Error naming the fault.
@spec size(t()) :: non_neg_integer()
How many parts the cache holds.