Macaw.Data.Operations.Cache (macaw_data v0.12.0)

View Source

A caller's cache of steps' per-chunk parts (spec/04 §1.5; macaw_data#9 §6): the example caller of the caching contract, a plain value the caller holds and passes back. The operations hold no cache and no hidden state; the frame or the feed that owns this value knows when a chunk leaves its window.

The key of a part is {logical id, version, step key}: Chunk.id/1, where the chunk sits, which scopes eviction; Chunk.version/1, its content digest (or a step output's derivation digest), which decides validity; and the part's step key (Macaw.Data.Operations.Step.parts/4), which carries the context as it resolved for the chunk. Nothing is hashed on lookup: the version is read from the chunk, computed once when its state was made, and a chunk whose digest is pending has none, so it never matches.

run/5 runs a step over a table a part at a time:

  • a part held under its key is reused;
  • a chunk with a delta whose base's part is held (under Chunk.base_version/1) is updated from it, where the step's inverse is exact (Step.update/4: counts, bin counts, integer sums);
  • any other part is computed (Step.chunk/3);

and the parts are merged (Step.merge/6), which only the combine's or global step's merge recomputes. The result equals Step.run/4's. prune/2 evicts by logical id.

cache = Cache.new()
{:ok, table, cache} = Cache.run(cache, Sum, node, frame_1, %{})
# the next frame: one new chunk computed, every other part reused
{:ok, table, cache} = Cache.run(cache, Sum, node, frame_2, %{})
cache = Cache.prune(cache, [frame_2])

Summary

Types

t()

The parts held by {id, version, step key}, and how the last run came by its parts: how many it reused, updated by inverse and computed.

Functions

An empty cache.

The cache with only the parts of chunks the tables still hold (spec/04 §1.5): the logical id scopes eviction, so a part of a chunk that left the window goes, and so does a part of a version no held chunk has, but for a chunk's base version, which its inverse updates start from.

The step module (a column-native one, Step.step?/1) run over table with the parts cache holds: {:ok, table, cache}, the cache holding this run's parts, or the step's fault, {:error, {key, reason}}, with the cache unchanged (spec/04 §3.1). The table equals Step.run(module, node, table, context)'s. With parallel: true in the context, the parts it computes run as Step.run/4's would (spec/04 §1.6).

As run/5, {table, cache}, raising Macaw.Data.Error naming the fault.

How many parts the cache holds.

Types

t()

@type t() :: %Macaw.Data.Operations.Cache{
  last: %{
    reused: non_neg_integer(),
    updated: non_neg_integer(),
    computed: non_neg_integer()
  },
  parts: %{required({term(), String.t(), term()}) => term()}
}

The parts held by {id, version, step key}, and how the last run came by its parts: how many it reused, updated by inverse and computed.

Functions

new()

@spec new() :: t()

An empty cache.

prune(cache, tables)

@spec prune(t(), [Macaw.Data.Table.t()]) :: t()

The cache with only the parts of chunks the tables still hold (spec/04 §1.5): the logical id scopes eviction, so a part of a chunk that left the window goes, and so does a part of a version no held chunk has, but for a chunk's base version, which its inverse updates start from.

run(cache, module, node, table, context)

@spec run(t(), module(), map(), Macaw.Data.Table.t(), map()) ::
  {:ok, Macaw.Data.Table.t(), t()}
  | {:error, Macaw.Data.Operations.Step.fault()}

The step module (a column-native one, Step.step?/1) run over table with the parts cache holds: {:ok, table, cache}, the cache holding this run's parts, or the step's fault, {:error, {key, reason}}, with the cache unchanged (spec/04 §3.1). The table equals Step.run(module, node, table, context)'s. With parallel: true in the context, the parts it computes run as Step.run/4's would (spec/04 §1.6).

run!(cache, module, node, table, context)

@spec run!(t(), module(), map(), Macaw.Data.Table.t(), map()) ::
  {Macaw.Data.Table.t(), t()}

As run/5, {table, cache}, raising Macaw.Data.Error naming the fault.

size(cache)

@spec size(t()) :: non_neg_integer()

How many parts the cache holds.