Changelog

View Source

v0.9.0 (2026-11-01)

On macaw_layers 0.4.0, and a request carries a row order (#92, #94). A capped or budgeted delivery now holds the first rows in the order a reader asked for, where it used to hold a sorted sample. The schema's JSON document carries a version stamp and is read through the kernel's gates (#95). A derived source can hold no URL. sort orders a time column by its instant (#93). A raise in a stored function or a conformance callback is answered without logging an [error] (#83). This release is breaking for an adopter that pins macaw_layers ~> 0.3, matches the fault shapes below, or reads Schema.to_json/1's output itself. The guide below lists what to change, and the entries after it give the detail.

Upgrading from 0.8.x

  1. Move to macaw_layers ~> 0.4.0 (#92). macaw_data 0.9.0 requires it, so an application that pins {:macaw_layers, "~> 0.3.0"} (or ~> 0.3) widens the pin to "~> 0.4.0" and follows macaw_layers 0.4.0's own upgrade notes: Form.read/3 takes urls: and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither change touches what it reads.
  2. Match the new fault shapes.
    • A derived source document's $atom outside the form's atoms is {:unknown_atom, spelling} at its path. It was {:enum, [:identity]}.
    • A URL anywhere in a derived source document ({"$url": …}) is {path, {:url, reason}} at the source's path. A schema document refuses one by type.
    • Schema.from_json/2's malformed JSON is {[], {:json, words}}, where it was {:json, inspect(reason)}. A newer stamp is {[], {:newer_schema, needs, 1}}, a stamp that is no positive integer is {["$version"], {:type, :pos_integer}}, a key beside "$version" and "fields" is {[key], :unknown_key}, and no "fields" is {["fields"], :required}.
    • A request's order that is malformed or that the schema can't answer is {[:order, i], reason} from Request.new/1 and Window.check/2, and {:error, {:order, faults}} from Source.fetch/3.
    • Derived.format_error/1 and Schema.format_error/1 word the kernel's reasons in the kernel's words (Macaw.Layers.Fault). A test that matches the old inspect/1 text needs its expected words updated.
  3. Schema.to_json/1 writes an object (#95): {"$version": 1, "fields": [[name, description], …]}. Where you decode its output yourself, match on the object, not a list. A document written by 0.8.x or earlier still reads: Schema.from_json/2 lifts its list root to version 1. to_map/1 and from_map/1 are unchanged, still the bare pairs, so a derived source's embedded "schema" is unchanged too.
  4. Request.order is new and defaults to [] (#94). A request with no order behaves as before. A source that declares the :order capability must serve its first budget rows in the order and say served.ordered: :source. Otherwise finishing sorts what the source served. Finish.reduce/5 of an ordered delivery keeps the head where it would have reduced by LTTB or M4. Description.capabilities/0 now lists :order, so a test that pins that list needs updating.
  5. sort puts a nil last in either direction (#93). It came first for a time, text or category column ascending, and first for a number column descending. A time column now sorts by its instant, not by struct term order, so an expectation written from the old order changes.

Changes

  • Breaking: the schema's JSON document has a version stamp, read through the kernel's gates (#84, #95, spec/01 §1.11). Schema.to_json/1 writes an object, {"$version": 1, "fields": [[name, description], …]}, so the first change to the form can tell an old document from a new one. to_map/1 and from_map/1 stay the bare pairs, for a document that embeds a schema (a derived source's "schema", unchanged). Schema.from_json/2 reads through Macaw.Layers.Form.gates/3, with no limit or scan code of its own: a newer stamp is refused by name before anything is read, malformed JSON is the kernel's {:json, words} (was {:json, inspect(reason)}), and a key beside "$version" and "fields" is {[key], :unknown_key}. A document written by 0.8.x or earlier still reads: its root is a list, which no version of the object has, so it is lifted to version 1 once its bytes and nesting pass the gates. Match on the object, not a list, where you read to_json/1's output yourself. Schema.format_error/1 words the kernel's reasons where it raised for them.
  • New: a request carries a row order, so a capped delivery holds the first rows in it (#80, #94, spec/01 §2.3–§2.4, §3, §9). Request.order is [{field, :asc | :desc}, …], [] by default. A "slowest first" table under a budget showed a sorted sample, since a source caps before the runtime sees a row (macaw#196). Window.check/2 holds the order's fields to the schema, as {[:order, i], {:unknown_field, name}} or {:invalid_for, type} for :point and :color. Source.fetch/3 refuses such an order before calling the source, {:error, {:order, faults}}, as it refuses parameters. Rows compare by their field's type (Macaw.Data.Source.Order): a time by its instant, a number by value, text by string, a category by its declared order, a missing reading last in either direction, stably.

  • New: the :order capability and served.ordered. A source naming :order serves its first budget rows in the order and says ordered: :source. Otherwise finishing sorts what it served, in a new step before the budget: ordered: :runtime, every row ordered, the head kept; or ordered: :sample, where the source had already cut its rows (reduced_by: :source or capped), so the reader is told they need not be the first. The conformance kit takes order in requests: and checks a source naming :order. A source stating :runtime or :sample is a finding.
  • Changed (behavior): Finish.reduce/5 of an ordered delivery (served.ordered set) keeps its first budget rows, whatever reducer is named, where it reduced by LTTB or M4. A pooled ordered fetch reduces to each reader's own head. Deliveries with no order are reduced as before. Description.capabilities/0 lists eight, :order before :budget.
  • Breaking: on macaw_layers ~> 0.4.0 (was ~> 0.3.0) (#91, #92). An adopter on macaw_layers 0.3.x moves to 0.4.0 with it, following its upgrade notes: Form.read/3 takes urls: and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither touches what it reads (spec/00 §3).
  • Breaking (faults): a derived source document's $atom outside the form's atoms is {:unknown_atom, spelling} at its path, where it was {:enum, [:identity]} (macawlayers 0.3.1, spec/02 §1.5). Match on `{:unknown_atom, }fromDerived.from_map/2andfrom_json/2`.
  • Fixed: a derived source holds no URL (spec/02 §1.5). macaw_layers 0.4.0 makes $url live in every tag table, and a derived source's reader composes the kernel's gates with new/1 rather than Form.read/3, so a {"$url": …} at a window's bound read into a %Macaw.Layers.Url{} that new/1 accepted, under no host's policy. The reader now holds the source it reads to the kernel's closed policy: a URL, whatever its scheme, is {path, {:url, reason}} at the source's path, and its words never quote it. A schema's reader reads no $url, so one is refused by type (spec/01 §1.11).
  • Changed: Derived.format_error/1 words the kernel's own reasons (an unknown tag or atom, a limit, a newer version, a URL) in the kernel's words (Macaw.Layers.Fault), where it showed them with inspect/1.
  • Fixed: sort orders a column by its type, not by term order (#93, #97, spec/04 §4.6). A time column was sorted by %DateTime{}'s struct term order, so by day of month before year: 2026-10-01, 2026-10-02, 2026-09-30, 2025-12-31 ascending. A time now sorts by its instant, as DateTime.compare/2 orders it, a number by its value, and text or a category by its spelling. :color, :point and :term keep term order. The step shares Macaw.Data.Source.Order's comparison (key/2, sort_by/2, new) with a request's row order, so the two order a field alike. Changed (behavior): a nil reading sorts last in either direction. Before, it came first for a time, text or category column ascending, and first for a number column descending. filter's orderings still compare by term order (#96).
  • Fixed: a raise answered by the bounded runner logs no error (#83, spec/02 §6.4, spec/01 §9). Macaw.Data.Bounded spawned its process with :erlang.spawn_opt/2, so a raise in a stored function, in a compile the compiler refused, or in a conformance callback was also reported by the runtime, as an [error] logged a few milliseconds after the caller had its answer, {:error, :raised, module}. A host running its users' functions logged an error for every user mistake. The runner now spawns through :proc_lib.spawn_opt/2: the answer and the exit reasons are unchanged, and nothing is rescued.
  • Changed (internal): the schema's JSON adapter lives in its own hidden module, Macaw.Data.Schema.Form (#85, spec/01 §1.11). Schema.to_map/1, to_json/1, from_map/1 and from_json/2 stay on Macaw.Data.Schema with their docs, so the public API is unchanged.
  • Tests: the dev-only surfex dependency is ~> 0.11.0 (#90), and WorkTest bounds maps:from_list/1's randomly seeded sort charge (#87). Neither changes the package.
  • A value entry's {"$ramp": [name, t]} reads for any name in a theme's grammar (macaw_layers 0.3.1's Color.Form), as $color does, so a schema's ramp color round-trips through JSON.

v0.8.1 (2026-10-31)

An application without Zig starts with no error logged (#76). Every app on macaw_data without Zigler, the default, logged an [error] crash report at each start, though nothing was wrong. Upgrading from 0.8.0 needs no change; from 0.7.0, follow 0.8.0's guide below.

Changes

  • Fixed: an application without the native kernels starts with nothing logged at error level (#76, #77, spec/04 §1.7). Macaw.Data.Native asked the kernels their interface version at the application's start even when the build had only their stub, and the stub's :not_built raise, in the process that asked, logged a crash report. Macaw.Data.Native.loaded?/0 was false, as it should be. Now only built kernels are asked: a build without them is known at compile time and starts quietly. With Zig, native: :optional and :required behave as before.
  • Fixed: the Hex package never carries Zigler's copy of the kernels' source (#77). Zigler writes .Elixir.<Module>.zig beside a module it compiles, so a package built from a checkout that had compiled the kernels would carry it. The package excludes such files. CI now builds the package, boots a minimal app from it without Zigler, and fails on any [error] line, a loaded?/0 that isn't false, or such a file.
  • Changed: each build stages the native kernels in its own directory (#72, #73, spec/04 §1.7). Zigler stages a module in /tmp/<module> by default, one directory for every checkout, worktree and build path on the host, so two builds at once wrote each other's build.zig and module.zig. The kernels now stage under the build's path, <build path>/zigler/, unless ZIGLER_STAGING_ROOT names another root.
  • Tests: suite verdicts are independent of load and build state (#73, #74, #75, #81): a --warnings-as-errors abort on a warning a lazily loaded test module raised is fixed, LadderNativeTest's property is bounded by its work in reductions rather than ExUnit's clock, and the suite's reduction counter is exact but for maps:from_list/1, whose residue it bounds.

v0.8.0 (2026-10-31)

The source contract is a time-varying relation, and a table is chunked (macaw_data#10, #9), with optional native kernels (#38). A push is a numbered changelog of increments, retractions and range replacements, with settled ranges and derived watermarks. A table is a list of immutable chunks with stable ids, and every step runs a chunk at a time, through a caching contract a caller holds. Where Zig is in the build, the hot steps run natively, bit for bit as their Elixir path, and a 50k-row live frame meets visualize's ~17 ms budget. The suite's verdicts no longer depend on how loaded the host is. On macaw_layers ~> 0.3.0, as 0.7.0 was. This release is breaking: the guide below is what an adopter of 0.7.0 changes, and the entries after it are the detail.

Upgrading from 0.7.0

  1. A table is chunked, opaque, and equal by its contents (#17). Compare tables with Macaw.Data.Table.equal?/2, never ==, since the same rows chunked two ways are one value. Read a table through its functions (rows/1, get/2, column/2, chunks/1), never the struct's fields. Table.t() is opaque. Keep functions, PIDs, references and ports out of tables: new/1 refuses them as {[name, i], :not_data}.
  2. Number a push's increments (#12). A source sends Push.increment_message/3, numbered from 1 per subscription, and may send Push.snapshot_message/2 with served.position. Push.delivery_message/2 is gone. Rows are no longer deduplicated: an identical row pushed again is a second row. Guard redelivery by the number, not by the row.
  3. Push.held/1's rows are a Macaw.Data.Table (#19). Read held rows by name with Macaw.Data.Table.get/2, compare a category as a string, and expect a time column's values as DateTimes. Rows with no window value sort last.
  4. sum and mean are exact (#21). Integers stay an integer. With a float among them, the exact sum is rounded once, ties to even, so a float sum may differ from a sequential sum in the last place: [1.0e16, 1.0, -1.0e16] sums to 1.0, where it was 0.0. A golden written from a sequential float sum may need regenerating.
  5. lttb is MinMaxLTTB, and m4 and lttb reduce on one bucket ladder (#22). lttb keeps at most n rows within its stated tolerance, the same under every chunking. The exact algorithm stays as Macaw.Data.Reduce.lttb/3. m4 is exact over pixel columns whose edges move to a bucket edge, at most a quarter of a pixel. A time column reduces under the :time scale, in microseconds, by default; pass scale: :linear for the former seconds.
  6. The row helpers Transforms.answer/4, point_reader/2 and column/2 are gone (#23). No table operation reads rows any more: a transform reads columns, or declares itself a column-native step (#20).
  7. Don't reuse a cache built by an earlier version. An Operations.Cache's m4/lttb partials changed shape twice (#40, #50), and aggregate, sum, bin and spectrum step keys no longer carry a chunk's place (#24). Drop a held cache on upgrade and let it rebuild.
  8. Native kernels are optional; nothing is needed to keep the Elixir path. To build them, list {:zigler, "~> 0.16.0", runtime: false} in your deps and have Zig 0.16.0 in the build (#48). Zigler's zig_doc pins ex_doc to 0.39.1, so a project that lists ex_doc itself may need override: true on it. Without Zig the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path, with the same results. Set config :macaw_data, native: :required to fail the build, and the start, without the kernels.

Changes

  • Spec: the source contract is one model, a relation that varies over time (#11, #10): a delivery is a snapshot, a push the changelog of Z-set increments, Push.held/1 their integral, :as_of a past snapshot, a derived source a map from relations to relations (spec/01 §2.5). The deferred §2.6 and §4.1–§4.4, and spec/02 §8.6, specify what #12–#15 and macaw_data#9 build: numbered increments and the snapshot's handoff, retractions and range replacements, one resolution per subscription, settled ranges, and derived watermarks.
  • Breaking: push increments are numbered, and a snapshot hands off to the stream by position (#12, spec/01 §4.1). A source sends {:increment, seq, delivery} (Push.increment_message/3, numbered from 1 per subscription) and may send {:snapshot, delivery} (Push.snapshot_message/2) whose served.position (new) is the last increment it reflects; Push.delivery_message/2 is gone. Push drops a repeated or older increment whole (counted in ignored, a count by reason), applies increments in the order of their numbers, holds back those past a gap, records the gap and answers it from Push.refetch/1; a snapshot, received or fetched with the request's new subscription: and handed over by Push.snapshot/2, replaces what is held and drops the increments it already reflects. Rows are no longer deduplicated: an identical row pushed again is a second row (tally taps), so redelivery is guarded by the number, not the row. The conformance kit's :push check holds increments to their numbering. Upgrade: number each subscription's increments from 1 and send them with increment_message/3.
  • New: pushed increments carry corrections (#13, spec/01 §4.2–§4.3). A delivery's retracted rows are withdrawn, each matched by identity, the window value it was held at and the fields it names, one copy per unit of weight, never a guessed row: one not held, held only at another window value, or that can't tell rows apart is {:retract, reason, row} in errors, and against a capped or reduced holding an unheld retraction is counted in ignored (:unheld). Its replace ranges, %{from, to, identities, rows}, make the relation over [from, to), limited to identities, exactly the rows given; a change inside a replaced range, a replacement's row outside it and an overlapping replacement are recorded and not applied. Row retractions apply first, then range retractions, then insertions. An increment is merged only into a relation held at its served.resolution; a mismatch is recorded and answered by Push.refetch/1.
  • New: settled ranges (#14, spec/01 §4.4). A delivery's served.complete_until is the source's watermark, a time or a number below which it expects no further change: settled, not final. Push carries the latest claim forward on what it holds and records a smaller one as {:complete_until, :backwards, held, given}, keeping the held claim; a new subscription starts afresh. A description may declare lateness, a hard bound (a Duration over a time window field, a number over a number one); given to Push.new/3 as lateness:, it refuses a change older than the claim minus the bound ({:late, row, bound}) and a replacement reaching past it whole. Without it a change of any age is accepted.
  • New: derived watermarks (#15, spec/02 §8.6). A derived source's output states served.complete_until, the minimum of its inputs' claims, each capped by its request window's upper bound; an input with no claim leaves none, and a change of the watermark alone marks the readers. A description may declare idle_after (milliseconds), which the owner passes to Engine.new/2 by name (idle_after:): a source that has delivered nothing for that long, judged against the now the engine is given, is left out of the minimum until its next delivery, unless every input is idle.
  • Breaking: a table is chunked, and its identity is its contents (#17, macaw_data#9, spec/01 §1.1.1–§1.1.2; an amendment to dco-tek/macaw/macaw#188 D5). A table is an ordered list of immutable chunks of Macaw.Data.Table.chunk_rows/0 (8,192) rows, each column packed per chunk, so a column's form and a dictionary are a chunk's. append/2, trim/2 and slice/3 keep whole chunks and offsets into them, copying nothing, and chunks/1 lists every chunk a table references. Table.t() is opaque. equal?/2 compares contents strictly and digest/1 is the content digest, the kernel's digest of the canonical term {"macaw.table", [{name, type}, …], [values, …]}. The same rows chunked two ways are one value, and compact/1 rewrites any table into canonical chunks. Each chunk carries its own digest (Table.Chunk.digest/1), computed once when it is built. new/1 refuses a value that is not data, a function, a PID, a reference, a port or an improper list, as {[name, i], :not_data}, since it has no digest. Upgrade: compare tables with Macaw.Data.Table.equal?/2, not ==; read storage through chunks/1, never the struct's fields; keep PIDs, references and functions out of tables.
  • New: chunk identity (#18, macaw_data#9, spec/01 §1.1.3). A chunk has a logical id, where it sits, and a version (Chunk.id/1, Chunk.version/1). A table built from rows names its chunks by their content digests. A source's table, built by Macaw.Data.Table.new/2 with origin: %{source, resolution, window, span}, lays its chunks on fixed slots and names them {source, resolution, slot}: a span's start in event time, else a row offset that is a multiple of chunk_rows/0. So ids hold across a rebuild and a restart, and two resolutions never share one. append/2 keeps a source's alignment, writing rows of the last slot into its chunk again, and compact/1 keeps its slots. A description may declare chunk_span, and Description.chunk_span/2 derives one from the resolution otherwise, never from the data. A step's output chunk (Chunk.derive/3, Chunk.combine/3, gathered by Macaw.Data.Table.from_chunks/3) is named after its inputs and step key. It is versioned by a derivation digest and its contents are not hashed, so a chain of steps hashes nothing.
  • Breaking: Push.held/1's rows are a chunked table (#19, macaw_data#9, spec/01 §1.1.4, §2.6). The rows are a Macaw.Data.Table of the source's origin, its columns the schema's fields, packed as Delivery.table/2 packs them, so a category's atom reads back as its string and a time column's values as DateTimes. Every change reaches the table through the new Macaw.Data.Table.apply/2. It writes an open chunk's new contents and puts a settled chunk's changes (wholly before complete_until) in its delta: a deleted set, predicate deletes for replacements, and a side chunk of insertions, all applied on read. Lazy compaction at a quarter of the base keeps the chunk's id and digest and emits nothing. A cold chunk (Table.Chunk.cold/5) is never fetched for a predicate delete; its digest stays pending until Macaw.Data.Table.warm/1. Push.new/3 takes chunk_span:. Rows with no window value now sort last in a source's table, as a push holds them. Rows that don't pack are {:table, faults} in errors. Upgrade: read held rows by name with Macaw.Data.Table.get/2, and compare a category as a string.
  • New: steps run a chunk at a time (#20, macaw_data#9 §5, spec/04 §1.4). A transform may declare itself a column-native step: its class (class/1), the context it reads (reads/0), the context bound over the whole input (bind/3), the context as it resolves for a chunk (resolve/3) and the chunk's part (chunk/3); Macaw.Data.Operations.Step runs it, and Step.parts/4 lists each chunk's view and step key, the step's identity, its context as resolved for the chunk and the run it reads, for a caller's cache. filter, take and window (without latest) are per-row steps: each chunk is passed through, dropped, cut to a run without copying, or gathered (Table.Chunk.gather/5, every column's slots copied, its id {input id, step key}), and the merge is concatenation. None of them transposes: at 50k rows take n: 100 is about 0.01 ms (was 81 ms), filter about 8 ms (was 93 ms). window keys a chunk wholly inside it without now, drops one outside, and binary-searches a sorted window field (Macaw.Data.Table.sorted/1, a source's window field, kept by every per-row step) to cut only its edge chunks. Macaw.Data.Table.from_chunks/3 takes views and sorted:; Chunk.values/4 and Chunk.at/3 read a step's column. Macaw.Data.Table.from/2 names its columns without merging a row map. The row helpers moved to an internal module, Macaw.Data.Table.Edge, and are still called through Macaw.Data.Table.
  • New: the associative combines (#21, macaw_data#9 §5, spec/04 §1.4, §1.2, §5). A step of class :combine declares combine/3, two runs' partials merged (associative, in chunk order), and finish/3, its table; Step.merge/5 folds a table's partials and Step.output/3 names a combine's output chunk over its inputs. sum, aggregate and bin (counting, or with count, sum, mean, min, max or extent) and window with latest are combines: any chunking and any grouping of their partials give equal tables. An aggregate may give the combining form, merge/3. Breaking (values): sum and mean sum exactly: integers stay an integer, and with a float among them the exact sum is rounded once to the nearest float (ties to even), so any order of the readings gives the same bits; a float sum may differ from the sequential sum in the last place (e.g. [1.0e16, 1.0, -1.0e16] sums to 1.0, was 0.0). latest's ties are broken by chunk order, then row order, as before. Aggregates without the combining form (median, quantile, distinct, variance, deviation, slope, delta) still read every reading at once; a quantile stays exact, never a sketch.
  • Breaking: lttb runs MinMaxLTTB, and m4 and lttb reduce on one absolute bucket ladder (#22, macaw_data#9 §5, spec/04 §4.4, §4.5, §4.5.1). Buckets are 2^k wide from origin 0 in the x scale's space, named by the new context keys scale (:linear, :time in microseconds, :log as log2 x; :time by default over a time column) and level (the previous level, for a hysteresis of a factor of 2). A chunk's partial depends only on its rows, the scale's kind and the level, so a zoom within a level reuses every partial, a zoom out folds finer partials (Ladder.coarsen/2), buckets straddling a chunk edge and gaps on one merge, and the count option leaves the step key (the new optional callback keyed/1). lttb's tolerance: at most n rows without gaps, the first and last kept, deterministic, the same under every chunking; its mean interpolation error, over the y extent, is at most twice the exact algorithm's plus 0.005 (measured worst 1.27× sines, 1.37× walks, 1.56× sparse spikes). The exact algorithm stays as Macaw.Data.Reduce.lttb/3. m4's tolerance: exact M4 over pixel columns whose boundaries move to a bucket edge, at most a quarter of a pixel away (half under hysteresis); Reduce.m4/3's answer where bucket and pixel edges meet. A budget's reducer (spec/04 §7) picks as the steps do. A time column reduced under the default :time scale buckets in microseconds; pass scale: :linear for the former seconds.
  • New: the global steps and the reshapes (#23, macaw_data#9 §5, spec/04 §1.4). A step of class :global gets every chunk's part in its finish/3: sort, stack, spectrum, and aggregate and bin with an fn that has no combining form (median, quantile, distinct, variance, deviation, slope, delta). A global output is one chunk named by its step key over its inputs' ids and versions, so a step after it still caches; rows kept across chunks are gathered by Step.gather/5, codes remapped. fold is a per-row reshape ({:reshape, indexes, added}, Table.Chunk.reshape/7: carried columns gathered with repeated indexes, the key and value added), and stack gathers across chunks with repeated indexes. Stack.series/2 stacks data as given. No table operation calls Macaw.Data.Table.rows/1 or Operations.table/2 any more; the row helpers Transforms.answer/4, point_reader/2 and column/2 are gone.
  • New: the caching contract (#24, macaw_data#9 §6, spec/04 §1.5, spec/01 §1.1.4). Macaw.Data.Operations.Cache is the example caller: a plain value the caller holds, keyed by {Chunk.id/1, Chunk.version/1, step key}. Cache.run/5 reuses a held part, updates a changed chunk's part by inverse from its base's part where exact (Step.update/4, a step's new optional update/4 and an aggregate's retract/3: counts, bin counts and integer sums), computes any other, and merges; its result equals Step.run/4's. Cache.prune/2 evicts by logical id. A lookup hashes nothing. A chunk with a delta keeps its base's version (Chunk.base_version/1) and a read view built once per state, so reads through a delta (Chunk.values/4, Chunk.at/3, a gather, window's binary search) no longer apply the delta on every read, and compaction makes the view the base without repacking. Fixed: aggregate, sum, bin and spectrum no longer put the chunk's place (:at, :rows) in their step keys, so a chunk's key holds while other chunks come and go.
  • New: parallel chunks on request (#25, macaw_data#9 §7, spec/04 §1.6, §3.2). parallel: true in the context asks Step.run/4 (so every table operation through Operations.apply/4) and Cache.run/5 to work a step's chunks across the schedulers, one task a part, when the input holds at least two chunks and Step.parallel_rows/0 (16,384) rows; otherwise, and by default, they run inline. The parts reach the merge in chunk order whatever worked them, so a parallel run is bit-identical to the inline one, floating-point sums included. parallel is no step's declared context, so it never enters a step key.
  • Measurement: a synthetic frame loop at 50k rows (#26, macaw_data#9 item 7, spec/04 §1.5): a live window with a chunk's worth of new readings a frame, running take, filter, window → m4, window → lttb and sum by series, recomputed in full and per chunk through Operations.Cache. The suite states the per-chunk cost in reductions (its changed chunks at most a full run over one chunk's readings, its frame at most half the full recompute); mix test --only benchmark prints the wall clock. Measured then, before #39–#54: about 400 ms a frame in full, 190 ms per chunk, of which about 95 ms is the m4 and lttb merges. Results and the native-kernel decision are on macaw_data#9; #54 below gives the release's figures.
  • Faster: m4 and lttb merge their partials in work proportional to the chunks, not the buckets (#39, #40, spec/04 §4.5.1). A partial holds a piece per view, each run's buckets ascending, so Ladder.combine/2 concatenates and reads no bucket; the chunk edges are joined once when the merged partial is read (Ladder.runs/1), and finish/3 folds the buckets into pixel columns (m4) or takes its candidates in input order (lttb) in one pass. Results are unchanged under every chunking and grouping. In #26's frame loop the merges alone went from about 100 ms to about 40 ms a frame, and the per-chunk frame from about 215 ms to about 125 ms. A partial's term changed shape (Ladder.partial/0), so an Operations.Cache built by an earlier version is not to be reused.
  • Faster: reading a column costs its slots, not its bits (#41, #42, spec/01 §1.1.5). An all-valid column decodes without expanding its validity bitmap; a slice cuts a bitmap as one little-endian integer; a gather takes an all-set bitmap without reading a bit; and Chunk.gather/5 and reshape/7 gather a run from the chunk's storage at its offset instead of slicing every column first. Bitmaps live in an internal module, Macaw.Data.Table.Bitmap. Results and digests unchanged.
  • Faster: a chunk's m4/lttb partial is one pass with nothing allocated per row but new extremes (#41, #43, spec/04 §4.5.1). The bucket width is read once a partial, an integer x buckets by a shift (s >>> k, s <<< -k), a number x on a linear or time scale is read without a call, and the open bucket's first, last, least and greatest ride as loop arguments. About half the reductions a row; partials unchanged.
  • Faster: grouping, exact sums and filter without per-row overhead (#41, #44, spec/04 §4.1, §4.10, §5). A chunk's groups are found in one pass, one map update a row, and grouping by one category column reads its dictionary codes and names each group once. Exact sums floats' signed mantissas by exponent and counts them in units of 2^-1074 once, when the value is read. filter scans a packed number column's buffer for an ordering against a number; its passing rows, when they are one run, are now cut from the chunk without copying rather than gathered (spec/04 §4.1), so such an output chunk is the input's, read from a row. Values unchanged.
  • Measurement: the frame loop's pure-Elixir cost is pinned (#41, #45, spec/04 §1.5): a per-chunk frame at most 1,100,000 reductions (1,982,948 before #41) and its merges alone 600,000. spec/04 §1.5 states that a frame's garbage collection is the caller's to size: at the default heap a frame's wall time is mostly collecting the cache's partials, and the benchmark now also reports frames in a process whose minimum heap holds the cache (#46 takes up keeping partials off the heap).
  • New: optional native kernels, built and loaded (#48, #38, macaw_data#9 §8, spec/04 §1.7). Macaw.Data.Native compiles Zig kernels through Zigler 0.16.0 when the build has Zig 0.16.0 (.tool-versions), found through zig env; Zigler is an optional dependency, so an adopter that wants the kernels lists {:zigler, "~> 0.16.0", runtime: false}. Without the toolchain the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path. Native.loaded?/0 is decided once at the application's start (the new application module, Macaw.Data.Application) and held in :persistent_term. config :macaw_data, native: :optional | :required, default :optional: :required fails the build without the toolchain, and the start when the kernels don't load. A call over at most Native.sync_rows/0 rows (8,192, #49) runs on a normal scheduler, a larger one on a dirty CPU scheduler. Kernels are built ReleaseSafe, so a panic is never silent. CI tests with Zig under :required, without Zig, and checks that :required without Zig fails the build. ex_doc is now a dev and test dependency, pinned by Zigler's zig_doc to 0.39.1. #48 itself gives no step a kernel; #49–#53 below do. A kernel never replaces its step's Elixir path, which stays the reference, the path for every form a kernel hands back, and the whole of a build without Zig.

  • Faster: a chunk's m4/lttb ladder partial runs in a native kernel where the kernels are loaded (#49, #38, spec/04 §4.5.2). Ladder.partial/4 hands the view to it when x is int64, float64, a UTC, zoned or naive time or a date, and y is int64 or float64, on a linear or time scale; the kernel reads the packed buffers and validity bitmaps in place and answers the Elixir walk's piece term for term (Ladder.partial_elixir/4, the reference). Where the walk would answer past 64 bits or raise, it hands the view back. Native.sync_rows/0 is 8,192, one canonical chunk: a call's time follows its answer's terms, at worst one bucket a row. Reductions count no kernel's work, so Native.elixir/1 runs a function on the Elixir path (Native.active?/0 false inside it), and the frame loop's reduction statements and pins are measured there.
  • Faster: a ladder partial's piece is one packed binary, off the process heap (#50, #38, #46, spec/04 §4.5.3). The Elixir walk packs its segments (Ladder.Packed: a header with the points' number types, a fixed-width head a segment, 88 bytes a bucket; the external term format for a piece of mixed or past-int64 values), and the native kernel writes the same bytes rather than building a term a bucket. The frame loop's cache went from about 824,000 words on the caller's heap to about 11,000, so a collection no longer copies it; a test holds it under 50,000. combine/2 still only concatenates; runs/1 reads each piece once. Results unchanged. A partial's shape changed again (Ladder.piece/0 is {view, binary}), so an Operations.Cache built by an earlier version is not to be reused.
  • Faster: m4's and lttb's finishes run in native kernels where the kernels are loaded (#51, #38, spec/04 §4.5.4). m4_finish and lttb_finish read the merged partial's packed pieces in place, join them as Ladder.runs/1 does, and answer the refs to gather: m4's pixel columns, and lttb's candidates through Reduce.lttb/3. Their arithmetic is Erlang's, operation for operation (exact integers, an integer converted to the nearest float where it meets one, strict floats), so the refs are the Elixir path's (M4.refs/3, Lttb.refs/2, held equal by a property test). A piece kept as a term, a value past int64 or anything Erlang would raise on is handed back to the Elixir path. A finish over at most Native.finish_buckets/0 (2,048) buckets runs on a normal scheduler, a larger one dirty.
  • Faster: the exact sum by group runs in a native kernel where the kernels are loaded (#52, #38, spec/04 §5.1). sum, and aggregate with sum, mean or count, over an int64 or float64 column grouped by one dictionary column or not at all, hand a chunk's view to group_sum, which reads the values, the codes and both bitmaps in place and answers each group's code, first row, rows, readings and exact-sum state (the internal Macaw.Data.Exact) (integers exact in 128 bits, past int64 as a bignum; floats' mantissas summed by biased exponent), so the part is the Elixir fold's term for term (held equal by a property test). A code past the dictionary or a non-finite float is handed back. A call answering more than Native.sync_entries/0 (2,048) groups and exponent sums answers :dirty before building a term and runs on a dirty scheduler. Sum by series over an 8,192-row chunk: 127,254 → 173 reductions, ~2.3 ms → ~80 µs. kernels.zig is split by kernel family (buffers, ladder, finish, group).
  • Faster: filter's test and the gather of the rows it keeps run in native kernels where the kernels are loaded (#53, #38, spec/04 §4.1.1). filter_mask reads the tested column in place and answers the selection itself: present/absent from any column's validity bitmap, an int64 or float64 column against a number compared exactly as Erlang compares (an integer against a float past 2^53 too), a dictionary of strings by each string's answer decided once with the Elixir path's own test. Column.gather/2 (every gather and reshape) copies fixed-width slots (take) and bitmap bits (take_bits) natively, the same bytes. Held equal to the Elixir path by property tests over every form, test and kind of value. v > 0 over an 8,192-row chunk with rows apart: the mask 62,275 → 30 reductions (~470 → ~75 µs), the whole step 112,003 → 6,507 reductions (~1.2 → ~0.5 ms).
  • Measurement: #26's frame loop with every native kernel (#54, #38, #9, spec/04 §1.5.1). Per chunk 958,680 → ~181,000 reductions and 36–47 → 7–10 ms at the default heap (13–16 → 6–8 ms with the frame heap sized); recomputed in full 1,931,845 → ~186,000 reductions, 55–58 → 9–11 ms. With the kernels the frame meets visualize's ~17 ms budget at the default heap, per chunk and in full. A new default-suite pin holds the frame with the kernels to at most 250,000 reductions per chunk, merges alone and in full.
  • New: Macaw.Data.Native.normal_schedulers/1 (#60, #61, spec/04 §1.7). It runs a function with every kernel call in its process on a normal scheduler, whatever the call's size. A call on a dirty scheduler is charged reductions by its time, about four a microsecond, so a count with one in it moves with the host's load. A normal-scheduler call is charged a fixed few. Use it for a statement of work in reductions with the kernels, such as the frame's (spec/04 §1.5.1), never for serving: a large call then holds its scheduler past the 1 ms rule. scheduler/1, finish_scheduler/1 and the new entry_limit/0 answer for it.
  • Fixed: a bounded call is hung only when it is blocked (#33, #62, #67, spec/02 §6.4, spec/01 §9). The conformance kit's runner and the derived engine's sandbox called any call still running at timeout hung, so on a loaded host a slow compile or a capped CPU-bound call was misjudged {:hung, …}. Now a call with a reductions cap is judged by its count while it works. It is hung only once it has been waiting in receive, its reductions unchanged, for timeout ms on end, and each piece of work gives it the whole bound again. A call with no cap still has only the clock. A wait on another process is a block all the same, so give a stored function a generous timeout:, as spec/02 §6.4 asks.
  • Fixed: the first stored function no longer waits on the code server (#69, spec/02 §6.3). In interactive mode a node's first compile loads Elixir's own modules on first use, and a busy code server could hold a bounded compile past its timeout. The application now compiles and runs one trivial function through the sandbox as it starts, so those modules are loaded before any bounded compile. Starting the application costs that one compile.
  • Docs: spec/00 names the dependencies as they are, with no pinned version: macaw_layers required, and table, nx and zigler optional (#34, #70). Every doc reference resolves, and CI builds the docs with --warnings-as-errors.
  • Tests: the suite is warnings-clean and stable under load (#37, #55, #56, #58, #63, #64, #65): call traces run in sessions of their own, test files compile with --warnings-as-errors, and no test's verdict depends on the host's load or on what other tests run beside it. Fifty loaded full-suite runs, with and without Zig, confirm it.

v0.7.0 (2026-10-30)

Terms are the form (dco-tek/macaw/macaw#188), macaw_data's part. A value's identity and validity are the BEAM term's own; JSON and Arrow are adapters at the edge. On macaw_layers 0.3.0. This release is breaking throughout; the guide below is what an adopter of 0.6.1 changes, and the entries after it are the detail.

Upgrading from 0.6.1

  1. Depend on macaw_layers ~> 0.3.0 (#27); its own upgrade guide applies to any kernel call you make directly.
  2. Every schema version changes (D1). A stored version, cache key or golden written by 0.6.1 differs; regenerate it. A schema's names, states and roles are now strings in the struct (Schema.new/1 normalizes them), so code that pattern-matches :latency inside a schema matches "latency"; every function that takes a name still takes either spelling.
  3. A data color is a color term (D2): a %Macaw.Layers.Color{} literal or a typed color, never a string. Parse CSS text at your input edge with Macaw.Layers.Color.parse!/1, or write ~RGB"#3b6fa8" in code. A role spelled like a CSS name ("orange") is now a valid status role.
  4. A derived source is built from its value (D3). Pass Derived.new/1 a keyword list or a map of terms; read a stored form with Derived.from_map/2 or from_json/2. to_map/1 and to_json/1 now answer {:ok, form} or faults.
  5. The operations registry's digest changes (D4); a compile cache keyed by it recompiles once. Operations.step_digest/2 is new: a checked step's canonical identity.
  6. Category values read back as strings (D5): a :category column holding :ok answers "ok" from rows/1, get/2 and column/2; compare with strings, or with Operations.same?/2. nil, true and false are kept.
  7. Packed columns are faithful and Arrow-compatible (D5). A mixed integer/float column and a zoned time read back exactly as given; anything that held packed buffers (stored binaries) changes with the bitmap order.

Changes

  • Breaking: on macaw_layers ~> 0.3.0 (#27): the kernel's digest is over the value, so every pinned version moved before D1 changed what it digests.
  • Breaking: every schema version changes (D1, #28, dco-tek/macaw/macaw#188). Schema.version is the kernel's digest of the schema's term, not of its JSON form, which is now only an adapter at the edge; a value entry's color no longer goes through the JSON form when Schema.new/1 versions the schema, so a typed color there can't crash it. The struct holds every field name, state and role as its string, normalized once by new/1, so fields, identity/1, window_field/1, compatibility/2's faults and identity/2's maps name fields by their strings; every function that takes a name still takes either spelling. A state named in both spellings in one description is :duplicate.
  • Breaking: data colors are color terms, and a string is never a color (D2, #29, dco-tek/macaw/macaw#187, dco-tek/macaw/macaw#188). A value entry's color, a :color field's row values (a derived source's output), a :color parameter and Schema.color?/1 take a literal %Macaw.Layers.Color{} or a theme-relative typed color ({:color, "good"}, {:alpha, {:color, "bad"}, 0.74}), its names in a theme's name grammar; a string there, "#2e7d32" or "red", is {:type, :color}. new/1 holds a typed color's names as strings, so {:color, :good} and {:color, "good"} are one version. The schema's JSON form writes and reads an entry's color in macaw_layers' color form (Macaw.Layers.Color.Form: {"$rgb": "#2e7d32"}, {"$color": "good"}), making no atom. Macaw.Data.Table.column_types/1 infers :color for a column of color terms; a column of strings stays :text. A status role spelled like a CSS name (:orange) is now a role. Upgrade: read CSS text into a color at your input edge with Macaw.Layers.Color.parse!/1, or write ~RGB"#2e7d32" in source (import Macaw.Layers.Color, only: [sigil_RGB: 2]); a stored schema whose entry colors are strings reads back refused, so convert its colors with parse!/1 before reading it.
  • Breaking: a derived source is built from its value, and its forms are an adapter (D3, #30, dco-tek/macaw/macaw#188). Derived.new/1 takes the value only, a keyword list or a map of its fields as terms; it no longer guesses whether it was given a map form, and a map form's string keys are each :unknown_key. The map and JSON forms are written and read by macaw_layers' JSON adapter (Macaw.Layers.Form), with the kernel's tags and the source's own in one tag table: to_map/1 and to_json/1 answer {:ok, form} or every value without a form, by path; from_map/2 and from_json/2 read through the gates (the limits, the version, the tag table, the keys, making no atom) and then judge the value with new/1. The output schema is used as built, a %Schema{} as given and fields by Schema.new/1, never written to its form and read back; in the form it is Schema.to_map/1's. Derived.Form's private tag scheme is gone. A color, literal or typed, in a schema's value entry reaches a stored function's context as the term it is; before, its tags were respelled as strings. Upgrade: the form changed: an input is {"$input": [name, window]}, a source's name {"$source": name}, per {"$atom": "identity"} or {"$identity": input}, a region an object keyed by field names, and to_json/1 stamps "$version": 1. Nothing in this library stored the old form of pair lists, so no version step reads it: write a stored source again with to_map/1 from its value. Read a map form with from_map/2, not new/1.
  • Breaking: the operations registry's digest/1 is the kernel's term digest of its names and modules, not a SHA-256 of term_to_binary (D4, #31, dco-tek/macaw/macaw#188), so every registry digest changes and a compile cache keyed by one recompiles once. New: Operations.step_digest/2, a checked step's canonical identity, for caches of a step's results.
  • Breaking: a table's packed columns are faithful and laid out as Arrow lays them out (D5, #16, dco-tek/macaw/macaw#188), spec/01 §1.1. Packing never changes a value. A category column of strings and atoms is one Arrow dictionary of strings, so rows/1, get/2 and column/2 read a category's atom back as the string that spells it (:ok and "ok" are one code, read as "ok"), in every form the column takes; nil, true and false stay themselves, beside the names in side bitmaps, so [:ok, true, "warn"] reads back ["ok", true, "warn"]. Validity bitmaps are least-significant-bit first, Arrow's order, not Elixir's <<b::1>> order, so every stored packed binary changes. A DateTime keeps its zone (name, abbreviation and both offsets, in a per-column dictionary, with no time-zone database) and its precision, so a time reads back as given rather than in UTC at microsecond precision. A text column is an offsets buffer and one data buffer, Arrow utf8 when every value is valid UTF-8 and Arrow binary when one is not; a category column holding a string that is not valid UTF-8 is kept as its list. new/1 takes NaiveDateTimes and Dates in a time column as themselves, and refuses a value in a :color column that is not a color term (Schema.color?/1), {[name, i], {:type, :color}}. Upgrade: compare a category value with Operations.same?/2, or as a string.
  • Fixed: a number column mixing integers and floats reads back each as given, through an integral bitmap, or a dense union of int64 and float64 when an integer is beyond ±2^53; an integer beyond int64 keeps its column a list rather than becoming a float. A time column keeps each value's zone and precision, so ~U[2026-01-01 00:00:00Z] and ~U[2026-01-01 00:00:00.000000Z] stay distinct (#16, dco-tek/macaw/macaw#188).

v0.6.1 (2026-10-29)

  • Fixed: compiling macaw_data in an app without the optional table package warned that Table.Reader.impl_for/1 is undefined; CI now compiles it without its optional dependencies (#7).

v0.6.0 (2026-10-29)

  • What a source steps its window by: Window.before/4, region/2 (visualize's box/2: a window as the region it asks of a schema), range/2, and check/2 of the axes alone as well as of a request (dco-tek/macaw/macaw#185, #5).
  • Packing a table and reading it back do about half the work: types inferred only where not given, values checked in one pass, column names merged once, rows built a column step at a time, and time columns converted by the date's fields, a fifth of DateTime's own conversion (dco-tek/macaw/macaw#185, #4).
  • Breaking: a delivery keeps the rows its source gave (dco-tek/macaw/macaw#185). %Delivery{rows, names, types, served, schema, origin}: nothing is packed or rewritten when it is made; Delivery.rows/1 and Macaw.Data.Table.rows/1 of a delivery read the rows as they are (a row list uncopied, tuple rows unchanged); Delivery.table/2 and table!/2 pack them for a reader that wants columns; named_rows/1 names tuple rows and tensors by names:; empty?/1. The table field is gone: read Delivery.table!/1. Finishing and push keep the rows' form. Macaw.Data.Table.unnamed?/1 (#3).

v0.5.4 (2026-10-28)

  • The macaw logo in the docs and the README (macaw#175).
  • Tests check "makes no atom" on the input's own names, never the node's atom count, and the atom gate forbids the count in test/ (#1).

v0.5.3 (2026-10-28)

  • On Hex: depends on macaw_layers ~> 0.2.2 from Hex instead of its git tag, and the README installs from Hex (macaw#155).
  • The README opens with the early-release notice (macaw#174).

v0.5.2 (2026-10-28)

  • Dependencies named by their dco-tek/macaw group paths (macaw#152).
  • Package metadata for Hex: the public home, the changelog link, the files shipped, and docs with ex_doc (macaw_compose#155).

v0.5.1 (2026-10-28)

  • macaw_data v0.5.1: on macaw_layers v0.2.0 (macaw_compose#145)

v0.5.0 (2026-10-27)

  • macaw_data v0.5.0: transforms, table operations and aggregates (macaw_compose#83): the behaviors, the registry, twelve table operations, thirteen statistics, Macaw.Data.Ops in the sandbox, budget reducers by name

v0.4.0 (2026-10-27)

  • macaw_data v0.4.0: derived sources (macaw_compose#82): the model, the sandbox and its gates, host functions and the engine

v0.3.0 (2026-10-27)

  • macaw_data v0.3.0: the source contract (macaw_compose#58): the behavior, description, request and parameters; windows and finishing; push; the conformance kit

v0.2.0 (2026-10-27)

  • macaw_data v0.2.0: the data model (macaw_compose#57): units and quantities, the columnar table, the schema, the delivery

v0.1.0 (2026-10-27)

  • macaw_data 0.1.0: Macaw.Data.Event