Changelog
View Sourcev0.9.0 (2026-11-01)
On macaw_layers 0.4.0, and a request carries a row order (#92, #94). A capped or budgeted delivery now holds the first rows in the order a reader asked for, where it used to hold a sorted sample. The schema's JSON document carries a version stamp and is read through the kernel's gates (#95). A derived source can hold no URL. sort orders a time column by its instant (#93). A raise in a stored function or a conformance callback is answered without logging an [error] (#83). This release is breaking for an adopter that pins macaw_layers ~> 0.3, matches the fault shapes below, or reads Schema.to_json/1's output itself. The guide below lists what to change, and the entries after it give the detail.
Upgrading from 0.8.x
- Move to macaw_layers
~> 0.4.0(#92). macaw_data 0.9.0 requires it, so an application that pins{:macaw_layers, "~> 0.3.0"}(or~> 0.3) widens the pin to"~> 0.4.0"and follows macaw_layers 0.4.0's own upgrade notes:Form.read/3takesurls:and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither change touches what it reads. - Match the new fault shapes.
- A derived source document's
$atomoutside the form's atoms is{:unknown_atom, spelling}at its path. It was{:enum, [:identity]}. - A URL anywhere in a derived source document (
{"$url": …}) is{path, {:url, reason}}at the source's path. A schema document refuses one by type. Schema.from_json/2's malformed JSON is{[], {:json, words}}, where it was{:json, inspect(reason)}. A newer stamp is{[], {:newer_schema, needs, 1}}, a stamp that is no positive integer is{["$version"], {:type, :pos_integer}}, a key beside"$version"and"fields"is{[key], :unknown_key}, and no"fields"is{["fields"], :required}.- A request's order that is malformed or that the schema can't answer is
{[:order, i], reason}fromRequest.new/1andWindow.check/2, and{:error, {:order, faults}}fromSource.fetch/3. Derived.format_error/1andSchema.format_error/1word the kernel's reasons in the kernel's words (Macaw.Layers.Fault). A test that matches the oldinspect/1text needs its expected words updated.
- A derived source document's
Schema.to_json/1writes an object (#95):{"$version": 1, "fields": [[name, description], …]}. Where you decode its output yourself, match on the object, not a list. A document written by 0.8.x or earlier still reads:Schema.from_json/2lifts its list root to version 1.to_map/1andfrom_map/1are unchanged, still the bare pairs, so a derived source's embedded"schema"is unchanged too.Request.orderis new and defaults to[](#94). A request with no order behaves as before. A source that declares the:ordercapability must serve its firstbudgetrows in the order and sayserved.ordered: :source. Otherwise finishing sorts what the source served.Finish.reduce/5of an ordered delivery keeps the head where it would have reduced by LTTB or M4.Description.capabilities/0now lists:order, so a test that pins that list needs updating.sortputs anillast in either direction (#93). It came first for a time, text or category column ascending, and first for a number column descending. A time column now sorts by its instant, not by struct term order, so an expectation written from the old order changes.
Changes
- Breaking: the schema's JSON document has a version stamp, read through the kernel's gates (#84, #95, spec/01 §1.11).
Schema.to_json/1writes an object,{"$version": 1, "fields": [[name, description], …]}, so the first change to the form can tell an old document from a new one.to_map/1andfrom_map/1stay the bare pairs, for a document that embeds a schema (a derived source's"schema", unchanged).Schema.from_json/2reads throughMacaw.Layers.Form.gates/3, with no limit or scan code of its own: a newer stamp is refused by name before anything is read, malformed JSON is the kernel's{:json, words}(was{:json, inspect(reason)}), and a key beside"$version"and"fields"is{[key], :unknown_key}. A document written by 0.8.x or earlier still reads: its root is a list, which no version of the object has, so it is lifted to version 1 once its bytes and nesting pass the gates. Match on the object, not a list, where you readto_json/1's output yourself.Schema.format_error/1words the kernel's reasons where it raised for them. New: a request carries a row order, so a capped delivery holds the first rows in it (#80, #94, spec/01 §2.3–§2.4, §3, §9).
Request.orderis[{field, :asc | :desc}, …],[]by default. A "slowest first" table under a budget showed a sorted sample, since a source caps before the runtime sees a row (macaw#196).Window.check/2holds the order's fields to the schema, as{[:order, i], {:unknown_field, name}}or{:invalid_for, type}for:pointand:color.Source.fetch/3refuses such an order before calling the source,{:error, {:order, faults}}, as it refuses parameters. Rows compare by their field's type (Macaw.Data.Source.Order): a time by its instant, a number by value, text by string, a category by its declaredorder, a missing reading last in either direction, stably.- New: the
:ordercapability andserved.ordered. A source naming:orderserves its firstbudgetrows in the order and saysordered: :source. Otherwise finishing sorts what it served, in a new step before the budget:ordered: :runtime, every row ordered, the head kept; orordered: :sample, where the source had already cut its rows (reduced_by: :sourceorcapped), so the reader is told they need not be the first. The conformance kit takesorderinrequests:and checks a source naming:order. A source stating:runtimeor:sampleis a finding. - Changed (behavior):
Finish.reduce/5of an ordered delivery (served.orderedset) keeps its firstbudgetrows, whatever reducer is named, where it reduced by LTTB or M4. A pooled ordered fetch reduces to each reader's own head. Deliveries with no order are reduced as before.Description.capabilities/0lists eight,:orderbefore:budget. - Breaking: on macaw_layers
~> 0.4.0(was~> 0.3.0) (#91, #92). An adopter on macaw_layers 0.3.x moves to 0.4.0 with it, following its upgrade notes:Form.read/3takesurls:and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither touches what it reads (spec/00 §3). - Breaking (faults): a derived source document's
$atomoutside the form's atoms is{:unknown_atom, spelling}at its path, where it was{:enum, [:identity]}(macawlayers 0.3.1, spec/02 §1.5). Match on `{:unknown_atom, }fromDerived.from_map/2andfrom_json/2`. - Fixed: a derived source holds no URL (spec/02 §1.5). macaw_layers 0.4.0 makes
$urllive in every tag table, and a derived source's reader composes the kernel's gates withnew/1rather thanForm.read/3, so a{"$url": …}at a window's bound read into a%Macaw.Layers.Url{}thatnew/1accepted, under no host's policy. The reader now holds the source it reads to the kernel's closed policy: a URL, whatever its scheme, is{path, {:url, reason}}at the source's path, and its words never quote it. A schema's reader reads no$url, so one is refused by type (spec/01 §1.11). - Changed:
Derived.format_error/1words the kernel's own reasons (an unknown tag or atom, a limit, a newer version, a URL) in the kernel's words (Macaw.Layers.Fault), where it showed them withinspect/1. - Fixed:
sortorders a column by its type, not by term order (#93, #97, spec/04 §4.6). A time column was sorted by%DateTime{}'s struct term order, so by day of month before year:2026-10-01, 2026-10-02, 2026-09-30, 2025-12-31ascending. A time now sorts by its instant, asDateTime.compare/2orders it, a number by its value, and text or a category by its spelling.:color,:pointand:termkeep term order. The step sharesMacaw.Data.Source.Order's comparison (key/2,sort_by/2, new) with a request's row order, so the two order a field alike. Changed (behavior): anilreading sorts last in either direction. Before, it came first for a time, text or category column ascending, and first for a number column descending.filter's orderings still compare by term order (#96). - Fixed: a raise answered by the bounded runner logs no error (#83, spec/02 §6.4, spec/01 §9).
Macaw.Data.Boundedspawned its process with:erlang.spawn_opt/2, so a raise in a stored function, in a compile the compiler refused, or in a conformance callback was also reported by the runtime, as an[error]logged a few milliseconds after the caller had its answer,{:error, :raised, module}. A host running its users' functions logged an error for every user mistake. The runner now spawns through:proc_lib.spawn_opt/2: the answer and the exit reasons are unchanged, and nothing is rescued. - Changed (internal): the schema's JSON adapter lives in its own hidden module, Macaw.Data.Schema.Form (#85, spec/01 §1.11).
Schema.to_map/1,to_json/1,from_map/1andfrom_json/2stay onMacaw.Data.Schemawith their docs, so the public API is unchanged. - Tests: the dev-only
surfexdependency is~> 0.11.0(#90), andWorkTestboundsmaps:from_list/1's randomly seeded sort charge (#87). Neither changes the package. - A value entry's
{"$ramp": [name, t]}reads for any name in a theme's grammar (macaw_layers 0.3.1'sColor.Form), as$colordoes, so a schema's ramp color round-trips through JSON.
v0.8.1 (2026-10-31)
An application without Zig starts with no error logged (#76). Every app on macaw_data without Zigler, the default, logged an [error] crash report at each start, though nothing was wrong. Upgrading from 0.8.0 needs no change; from 0.7.0, follow 0.8.0's guide below.
Changes
- Fixed: an application without the native kernels starts with nothing logged at error level (#76, #77, spec/04 §1.7).
Macaw.Data.Nativeasked the kernels their interface version at the application's start even when the build had only their stub, and the stub's:not_builtraise, in the process that asked, logged a crash report.Macaw.Data.Native.loaded?/0was false, as it should be. Now only built kernels are asked: a build without them is known at compile time and starts quietly. With Zig,native: :optionaland:requiredbehave as before. - Fixed: the Hex package never carries Zigler's copy of the kernels' source (#77). Zigler writes
.Elixir.<Module>.zigbeside a module it compiles, so a package built from a checkout that had compiled the kernels would carry it. The package excludes such files. CI now builds the package, boots a minimal app from it without Zigler, and fails on any[error]line, aloaded?/0that isn't false, or such a file. - Changed: each build stages the native kernels in its own directory (#72, #73, spec/04 §1.7). Zigler stages a module in
/tmp/<module>by default, one directory for every checkout, worktree and build path on the host, so two builds at once wrote each other'sbuild.zigandmodule.zig. The kernels now stage under the build's path,<build path>/zigler/, unlessZIGLER_STAGING_ROOTnames another root. - Tests: suite verdicts are independent of load and build state (#73, #74, #75, #81): a
--warnings-as-errorsabort on a warning a lazily loaded test module raised is fixed,LadderNativeTest's property is bounded by its work in reductions rather than ExUnit's clock, and the suite's reduction counter is exact but formaps:from_list/1, whose residue it bounds.
v0.8.0 (2026-10-31)
The source contract is a time-varying relation, and a table is chunked (macaw_data#10, #9), with optional native kernels (#38). A push is a numbered changelog of increments, retractions and range replacements, with settled ranges and derived watermarks. A table is a list of immutable chunks with stable ids, and every step runs a chunk at a time, through a caching contract a caller holds. Where Zig is in the build, the hot steps run natively, bit for bit as their Elixir path, and a 50k-row live frame meets visualize's ~17 ms budget. The suite's verdicts no longer depend on how loaded the host is. On macaw_layers ~> 0.3.0, as 0.7.0 was. This release is breaking: the guide below is what an adopter of 0.7.0 changes, and the entries after it are the detail.
Upgrading from 0.7.0
- A table is chunked, opaque, and equal by its contents (#17). Compare tables with
Macaw.Data.Table.equal?/2, never==, since the same rows chunked two ways are one value. Read a table through its functions (rows/1,get/2,column/2,chunks/1), never the struct's fields.Table.t()is opaque. Keep functions, PIDs, references and ports out of tables:new/1refuses them as{[name, i], :not_data}. - Number a push's increments (#12). A source sends
Push.increment_message/3, numbered from 1 per subscription, and may sendPush.snapshot_message/2withserved.position.Push.delivery_message/2is gone. Rows are no longer deduplicated: an identical row pushed again is a second row. Guard redelivery by the number, not by the row. Push.held/1's rows are aMacaw.Data.Table(#19). Read held rows by name withMacaw.Data.Table.get/2, compare a category as a string, and expect a time column's values asDateTimes. Rows with no window value sort last.sumandmeanare exact (#21). Integers stay an integer. With a float among them, the exact sum is rounded once, ties to even, so a float sum may differ from a sequential sum in the last place:[1.0e16, 1.0, -1.0e16]sums to1.0, where it was0.0. A golden written from a sequential float sum may need regenerating.lttbis MinMaxLTTB, andm4andlttbreduce on one bucket ladder (#22).lttbkeeps at mostnrows within its stated tolerance, the same under every chunking. The exact algorithm stays asMacaw.Data.Reduce.lttb/3.m4is exact over pixel columns whose edges move to a bucket edge, at most a quarter of a pixel. A time column reduces under the:timescale, in microseconds, by default; passscale: :linearfor the former seconds.- The row helpers
Transforms.answer/4,point_reader/2andcolumn/2are gone (#23). No table operation reads rows any more: a transform reads columns, or declares itself a column-native step (#20). - Don't reuse a cache built by an earlier version. An
Operations.Cache'sm4/lttbpartials changed shape twice (#40, #50), andaggregate,sum,binandspectrumstep keys no longer carry a chunk's place (#24). Drop a held cache on upgrade and let it rebuild. - Native kernels are optional; nothing is needed to keep the Elixir path. To build them, list
{:zigler, "~> 0.16.0", runtime: false}in your deps and have Zig 0.16.0 in the build (#48). Zigler'szig_docpinsex_docto 0.39.1, so a project that listsex_docitself may needoverride: trueon it. Without Zig the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path, with the same results. Setconfig :macaw_data, native: :requiredto fail the build, and the start, without the kernels.
Changes
- Spec: the source contract is one model, a relation that varies over time (#11, #10): a delivery is a snapshot, a push the changelog of Z-set increments,
Push.held/1their integral,:as_ofa past snapshot, a derived source a map from relations to relations (spec/01 §2.5). The deferred §2.6 and §4.1–§4.4, and spec/02 §8.6, specify what #12–#15 and macaw_data#9 build: numbered increments and the snapshot's handoff, retractions and range replacements, one resolution per subscription, settled ranges, and derived watermarks. - Breaking: push increments are numbered, and a snapshot hands off to the stream by position (#12, spec/01 §4.1). A source sends
{:increment, seq, delivery}(Push.increment_message/3, numbered from 1 per subscription) and may send{:snapshot, delivery}(Push.snapshot_message/2) whoseserved.position(new) is the last increment it reflects;Push.delivery_message/2is gone.Pushdrops a repeated or older increment whole (counted inignored, a count by reason), applies increments in the order of their numbers, holds back those past a gap, records the gap and answers it fromPush.refetch/1; a snapshot, received or fetched with the request's newsubscription:and handed over byPush.snapshot/2, replaces what is held and drops the increments it already reflects. Rows are no longer deduplicated: an identical row pushed again is a second row (tally taps), so redelivery is guarded by the number, not the row. The conformance kit's:pushcheck holds increments to their numbering. Upgrade: number each subscription's increments from 1 and send them withincrement_message/3. - New: pushed increments carry corrections (#13, spec/01 §4.2–§4.3). A delivery's
retractedrows are withdrawn, each matched by identity, the window value it was held at and the fields it names, one copy per unit of weight, never a guessed row: one not held, held only at another window value, or that can't tell rows apart is{:retract, reason, row}inerrors, and against acappedor reduced holding an unheld retraction is counted inignored(:unheld). Itsreplaceranges,%{from, to, identities, rows}, make the relation over[from, to), limited to identities, exactly the rows given; a change inside a replaced range, a replacement's row outside it and an overlapping replacement are recorded and not applied. Row retractions apply first, then range retractions, then insertions. An increment is merged only into a relation held at itsserved.resolution; a mismatch is recorded and answered byPush.refetch/1. - New: settled ranges (#14, spec/01 §4.4). A delivery's
served.complete_untilis the source's watermark, a time or a number below which it expects no further change: settled, not final.Pushcarries the latest claim forward on what it holds and records a smaller one as{:complete_until, :backwards, held, given}, keeping the held claim; a new subscription starts afresh. A description may declarelateness, a hard bound (aDurationover a time window field, a number over a number one); given toPush.new/3aslateness:, it refuses a change older than the claim minus the bound ({:late, row, bound}) and a replacement reaching past it whole. Without it a change of any age is accepted. - New: derived watermarks (#15, spec/02 §8.6). A derived source's output states
served.complete_until, the minimum of its inputs' claims, each capped by its request window's upper bound; an input with no claim leaves none, and a change of the watermark alone marks the readers. A description may declareidle_after(milliseconds), which the owner passes toEngine.new/2by name (idle_after:): a source that has delivered nothing for that long, judged against thenowthe engine is given, is left out of the minimum until its next delivery, unless every input is idle. - Breaking: a table is chunked, and its identity is its contents (#17, macaw_data#9, spec/01 §1.1.1–§1.1.2; an amendment to dco-tek/macaw/macaw#188 D5). A table is an ordered list of immutable chunks of
Macaw.Data.Table.chunk_rows/0(8,192) rows, each column packed per chunk, so a column's form and a dictionary are a chunk's.append/2,trim/2andslice/3keep whole chunks and offsets into them, copying nothing, andchunks/1lists every chunk a table references.Table.t()is opaque.equal?/2compares contents strictly anddigest/1is the content digest, the kernel's digest of the canonical term{"macaw.table", [{name, type}, …], [values, …]}. The same rows chunked two ways are one value, andcompact/1rewrites any table into canonical chunks. Each chunk carries its own digest (Table.Chunk.digest/1), computed once when it is built.new/1refuses a value that is not data, a function, a PID, a reference, a port or an improper list, as{[name, i], :not_data}, since it has no digest. Upgrade: compare tables withMacaw.Data.Table.equal?/2, not==; read storage throughchunks/1, never the struct's fields; keep PIDs, references and functions out of tables. - New: chunk identity (#18, macaw_data#9, spec/01 §1.1.3). A chunk has a logical id, where it sits, and a version (
Chunk.id/1,Chunk.version/1). A table built from rows names its chunks by their content digests. A source's table, built byMacaw.Data.Table.new/2withorigin: %{source, resolution, window, span}, lays its chunks on fixed slots and names them{source, resolution, slot}: a span's start in event time, else a row offset that is a multiple ofchunk_rows/0. So ids hold across a rebuild and a restart, and two resolutions never share one.append/2keeps a source's alignment, writing rows of the last slot into its chunk again, andcompact/1keeps its slots. A description may declarechunk_span, andDescription.chunk_span/2derives one from the resolution otherwise, never from the data. A step's output chunk (Chunk.derive/3,Chunk.combine/3, gathered byMacaw.Data.Table.from_chunks/3) is named after its inputs and step key. It is versioned by a derivation digest and its contents are not hashed, so a chain of steps hashes nothing. - Breaking:
Push.held/1's rows are a chunked table (#19, macaw_data#9, spec/01 §1.1.4, §2.6). The rows are aMacaw.Data.Tableof the source's origin, its columns the schema's fields, packed asDelivery.table/2packs them, so a category's atom reads back as its string and a time column's values asDateTimes. Every change reaches the table through the newMacaw.Data.Table.apply/2. It writes an open chunk's new contents and puts a settled chunk's changes (wholly beforecomplete_until) in its delta: a deleted set, predicate deletes for replacements, and a side chunk of insertions, all applied on read. Lazy compaction at a quarter of the base keeps the chunk's id and digest and emits nothing. A cold chunk (Table.Chunk.cold/5) is never fetched for a predicate delete; its digest stays pending untilMacaw.Data.Table.warm/1.Push.new/3takeschunk_span:. Rows with no window value now sort last in a source's table, as a push holds them. Rows that don't pack are{:table, faults}inerrors. Upgrade: read held rows by name withMacaw.Data.Table.get/2, and compare a category as a string. - New: steps run a chunk at a time (#20, macaw_data#9 §5, spec/04 §1.4). A transform may declare itself a column-native step: its class (
class/1), the context it reads (reads/0), the context bound over the whole input (bind/3), the context as it resolves for a chunk (resolve/3) and the chunk's part (chunk/3);Macaw.Data.Operations.Stepruns it, andStep.parts/4lists each chunk's view and step key, the step's identity, its context as resolved for the chunk and the run it reads, for a caller's cache.filter,takeandwindow(withoutlatest) are per-row steps: each chunk is passed through, dropped, cut to a run without copying, or gathered (Table.Chunk.gather/5, every column's slots copied, its id{input id, step key}), and the merge is concatenation. None of them transposes: at 50k rowstake n: 100is about 0.01 ms (was 81 ms),filterabout 8 ms (was 93 ms).windowkeys a chunk wholly inside it withoutnow, drops one outside, and binary-searches a sorted window field (Macaw.Data.Table.sorted/1, a source's window field, kept by every per-row step) to cut only its edge chunks.Macaw.Data.Table.from_chunks/3takes views andsorted:;Chunk.values/4andChunk.at/3read a step's column.Macaw.Data.Table.from/2names its columns without merging a row map. The row helpers moved to an internal module, Macaw.Data.Table.Edge, and are still called throughMacaw.Data.Table. - New: the associative combines (#21, macaw_data#9 §5, spec/04 §1.4, §1.2, §5). A step of class
:combinedeclarescombine/3, two runs' partials merged (associative, in chunk order), andfinish/3, its table;Step.merge/5folds a table's partials andStep.output/3names a combine's output chunk over its inputs.sum,aggregateandbin(counting, or withcount,sum,mean,min,maxorextent) andwindowwithlatestare combines: any chunking and any grouping of their partials give equal tables. An aggregate may give the combining form,merge/3. Breaking (values):sumandmeansum exactly: integers stay an integer, and with a float among them the exact sum is rounded once to the nearest float (ties to even), so any order of the readings gives the same bits; a float sum may differ from the sequential sum in the last place (e.g.[1.0e16, 1.0, -1.0e16]sums to1.0, was0.0).latest's ties are broken by chunk order, then row order, as before. Aggregates without the combining form (median,quantile,distinct,variance,deviation,slope,delta) still read every reading at once; a quantile stays exact, never a sketch. - Breaking:
lttbruns MinMaxLTTB, andm4andlttbreduce on one absolute bucket ladder (#22, macaw_data#9 §5, spec/04 §4.4, §4.5, §4.5.1). Buckets are2^kwide from origin 0 in the x scale's space, named by the new context keysscale(:linear,:timein microseconds,:logaslog2 x;:timeby default over a time column) andlevel(the previous level, for a hysteresis of a factor of 2). A chunk's partial depends only on its rows, the scale's kind and the level, so a zoom within a level reuses every partial, a zoom out folds finer partials (Ladder.coarsen/2), buckets straddling a chunk edge and gaps on one merge, and the count option leaves the step key (the new optional callbackkeyed/1).lttb's tolerance: at mostnrows without gaps, the first and last kept, deterministic, the same under every chunking; its mean interpolation error, over the y extent, is at most twice the exact algorithm's plus 0.005 (measured worst 1.27× sines, 1.37× walks, 1.56× sparse spikes). The exact algorithm stays asMacaw.Data.Reduce.lttb/3.m4's tolerance: exact M4 over pixel columns whose boundaries move to a bucket edge, at most a quarter of a pixel away (half under hysteresis);Reduce.m4/3's answer where bucket and pixel edges meet. A budget's reducer (spec/04 §7) picks as the steps do. A time column reduced under the default:timescale buckets in microseconds; passscale: :linearfor the former seconds. - New: the global steps and the reshapes (#23, macaw_data#9 §5, spec/04 §1.4). A step of class
:globalgets every chunk's part in itsfinish/3:sort,stack,spectrum, andaggregateandbinwith anfnthat has no combining form (median,quantile,distinct,variance,deviation,slope,delta). A global output is one chunk named by its step key over its inputs' ids and versions, so a step after it still caches; rows kept across chunks are gathered byStep.gather/5, codes remapped.foldis a per-row reshape ({:reshape, indexes, added},Table.Chunk.reshape/7: carried columns gathered with repeated indexes, the key and value added), andstackgathers across chunks with repeated indexes.Stack.series/2stacks data as given. No table operation callsMacaw.Data.Table.rows/1orOperations.table/2any more; the row helpersTransforms.answer/4,point_reader/2andcolumn/2are gone. - New: the caching contract (#24, macaw_data#9 §6, spec/04 §1.5, spec/01 §1.1.4).
Macaw.Data.Operations.Cacheis the example caller: a plain value the caller holds, keyed by{Chunk.id/1, Chunk.version/1, step key}.Cache.run/5reuses a held part, updates a changed chunk's part by inverse from its base's part where exact (Step.update/4, a step's new optionalupdate/4and an aggregate'sretract/3: counts,bincounts and integer sums), computes any other, and merges; its result equalsStep.run/4's.Cache.prune/2evicts by logical id. A lookup hashes nothing. A chunk with a delta keeps its base's version (Chunk.base_version/1) and a read view built once per state, so reads through a delta (Chunk.values/4,Chunk.at/3, a gather,window's binary search) no longer apply the delta on every read, and compaction makes the view the base without repacking. Fixed:aggregate,sum,binandspectrumno longer put the chunk's place (:at,:rows) in their step keys, so a chunk's key holds while other chunks come and go. - New: parallel chunks on request (#25, macaw_data#9 §7, spec/04 §1.6, §3.2).
parallel: truein the context asksStep.run/4(so every table operation throughOperations.apply/4) andCache.run/5to work a step's chunks across the schedulers, one task a part, when the input holds at least two chunks andStep.parallel_rows/0(16,384) rows; otherwise, and by default, they run inline. The parts reach the merge in chunk order whatever worked them, so a parallel run is bit-identical to the inline one, floating-point sums included.parallelis no step's declared context, so it never enters a step key. - Measurement: a synthetic frame loop at 50k rows (#26, macaw_data#9 item 7, spec/04 §1.5): a live window with a chunk's worth of new readings a frame, running
take,filter,window→m4,window→lttbandsumby series, recomputed in full and per chunk throughOperations.Cache. The suite states the per-chunk cost in reductions (its changed chunks at most a full run over one chunk's readings, its frame at most half the full recompute);mix test --only benchmarkprints the wall clock. Measured then, before #39–#54: about 400 ms a frame in full, 190 ms per chunk, of which about 95 ms is them4andlttbmerges. Results and the native-kernel decision are on macaw_data#9; #54 below gives the release's figures. - Faster:
m4andlttbmerge their partials in work proportional to the chunks, not the buckets (#39, #40, spec/04 §4.5.1). A partial holds a piece per view, each run's buckets ascending, soLadder.combine/2concatenates and reads no bucket; the chunk edges are joined once when the merged partial is read (Ladder.runs/1), andfinish/3folds the buckets into pixel columns (m4) or takes its candidates in input order (lttb) in one pass. Results are unchanged under every chunking and grouping. In #26's frame loop the merges alone went from about 100 ms to about 40 ms a frame, and the per-chunk frame from about 215 ms to about 125 ms. A partial's term changed shape (Ladder.partial/0), so anOperations.Cachebuilt by an earlier version is not to be reused. - Faster: reading a column costs its slots, not its bits (#41, #42, spec/01 §1.1.5). An all-valid column decodes without expanding its validity bitmap; a slice cuts a bitmap as one little-endian integer; a gather takes an all-set bitmap without reading a bit; and
Chunk.gather/5andreshape/7gather a run from the chunk's storage at its offset instead of slicing every column first. Bitmaps live in an internal module, Macaw.Data.Table.Bitmap. Results and digests unchanged. - Faster: a chunk's
m4/lttbpartial is one pass with nothing allocated per row but new extremes (#41, #43, spec/04 §4.5.1). The bucket width is read once a partial, an integer x buckets by a shift (s >>> k,s <<< -k), a number x on a linear or time scale is read without a call, and the open bucket's first, last, least and greatest ride as loop arguments. About half the reductions a row; partials unchanged. - Faster: grouping, exact sums and
filterwithout per-row overhead (#41, #44, spec/04 §4.1, §4.10, §5). A chunk's groups are found in one pass, one map update a row, and grouping by one category column reads its dictionary codes and names each group once.Exactsums floats' signed mantissas by exponent and counts them in units of 2^-1074 once, when the value is read.filterscans a packed number column's buffer for an ordering against a number; its passing rows, when they are one run, are now cut from the chunk without copying rather than gathered (spec/04 §4.1), so such an output chunk is the input's, read from a row. Values unchanged. - Measurement: the frame loop's pure-Elixir cost is pinned (#41, #45, spec/04 §1.5): a per-chunk frame at most 1,100,000 reductions (1,982,948 before #41) and its merges alone 600,000. spec/04 §1.5 states that a frame's garbage collection is the caller's to size: at the default heap a frame's wall time is mostly collecting the cache's partials, and the benchmark now also reports frames in a process whose minimum heap holds the cache (#46 takes up keeping partials off the heap).
New: optional native kernels, built and loaded (#48, #38, macaw_data#9 §8, spec/04 §1.7).
Macaw.Data.Nativecompiles Zig kernels through Zigler 0.16.0 when the build has Zig 0.16.0 (.tool-versions), found throughzig env; Zigler is an optional dependency, so an adopter that wants the kernels lists{:zigler, "~> 0.16.0", runtime: false}. Without the toolchain the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path.Native.loaded?/0is decided once at the application's start (the new application module, Macaw.Data.Application) and held in:persistent_term.config :macaw_data, native: :optional | :required, default:optional::requiredfails the build without the toolchain, and the start when the kernels don't load. A call over at mostNative.sync_rows/0rows (8,192, #49) runs on a normal scheduler, a larger one on a dirty CPU scheduler. Kernels are built ReleaseSafe, so a panic is never silent. CI tests with Zig under:required, without Zig, and checks that:requiredwithout Zig fails the build.ex_docis now a dev and test dependency, pinned by Zigler'szig_docto 0.39.1. #48 itself gives no step a kernel; #49–#53 below do. A kernel never replaces its step's Elixir path, which stays the reference, the path for every form a kernel hands back, and the whole of a build without Zig.- Faster: a chunk's
m4/lttbladder partial runs in a native kernel where the kernels are loaded (#49, #38, spec/04 §4.5.2).Ladder.partial/4hands the view to it when x is int64, float64, a UTC, zoned or naive time or a date, and y is int64 or float64, on a linear or time scale; the kernel reads the packed buffers and validity bitmaps in place and answers the Elixir walk's piece term for term (Ladder.partial_elixir/4, the reference). Where the walk would answer past 64 bits or raise, it hands the view back.Native.sync_rows/0is 8,192, one canonical chunk: a call's time follows its answer's terms, at worst one bucket a row. Reductions count no kernel's work, soNative.elixir/1runs a function on the Elixir path (Native.active?/0false inside it), and the frame loop's reduction statements and pins are measured there. - Faster: a ladder partial's piece is one packed binary, off the process heap (#50, #38, #46, spec/04 §4.5.3). The Elixir walk packs its segments (
Ladder.Packed: a header with the points' number types, a fixed-width head a segment, 88 bytes a bucket; the external term format for a piece of mixed or past-int64 values), and the native kernel writes the same bytes rather than building a term a bucket. The frame loop's cache went from about 824,000 words on the caller's heap to about 11,000, so a collection no longer copies it; a test holds it under 50,000.combine/2still only concatenates;runs/1reads each piece once. Results unchanged. A partial's shape changed again (Ladder.piece/0is{view, binary}), so anOperations.Cachebuilt by an earlier version is not to be reused. - Faster:
m4's andlttb's finishes run in native kernels where the kernels are loaded (#51, #38, spec/04 §4.5.4).m4_finishandlttb_finishread the merged partial's packed pieces in place, join them asLadder.runs/1does, and answer the refs to gather:m4's pixel columns, andlttb's candidates throughReduce.lttb/3. Their arithmetic is Erlang's, operation for operation (exact integers, an integer converted to the nearest float where it meets one, strict floats), so the refs are the Elixir path's (M4.refs/3,Lttb.refs/2, held equal by a property test). A piece kept as a term, a value past int64 or anything Erlang would raise on is handed back to the Elixir path. A finish over at mostNative.finish_buckets/0(2,048) buckets runs on a normal scheduler, a larger one dirty. - Faster: the exact sum by group runs in a native kernel where the kernels are loaded (#52, #38, spec/04 §5.1).
sum, andaggregatewithsum,meanorcount, over an int64 or float64 column grouped by one dictionary column or not at all, hand a chunk's view togroup_sum, which reads the values, the codes and both bitmaps in place and answers each group's code, first row, rows, readings and exact-sum state (the internal Macaw.Data.Exact) (integers exact in 128 bits, past int64 as a bignum; floats' mantissas summed by biased exponent), so the part is the Elixir fold's term for term (held equal by a property test). A code past the dictionary or a non-finite float is handed back. A call answering more thanNative.sync_entries/0(2,048) groups and exponent sums answers:dirtybefore building a term and runs on a dirty scheduler. Sum by series over an 8,192-row chunk: 127,254 → 173 reductions, ~2.3 ms → ~80 µs. kernels.zig is split by kernel family (buffers, ladder, finish, group). - Faster:
filter's test and the gather of the rows it keeps run in native kernels where the kernels are loaded (#53, #38, spec/04 §4.1.1).filter_maskreads the tested column in place and answers the selection itself:present/absentfrom any column's validity bitmap, an int64 or float64 column against a number compared exactly as Erlang compares (an integer against a float past 2^53 too), a dictionary of strings by each string's answer decided once with the Elixir path's own test.Column.gather/2(every gather and reshape) copies fixed-width slots (take) and bitmap bits (take_bits) natively, the same bytes. Held equal to the Elixir path by property tests over every form, test and kind of value.v > 0over an 8,192-row chunk with rows apart: the mask 62,275 → 30 reductions (~470 → ~75 µs), the whole step 112,003 → 6,507 reductions (~1.2 → ~0.5 ms). - Measurement: #26's frame loop with every native kernel (#54, #38, #9, spec/04 §1.5.1). Per chunk 958,680 → ~181,000 reductions and 36–47 → 7–10 ms at the default heap (13–16 → 6–8 ms with the frame heap sized); recomputed in full 1,931,845 → ~186,000 reductions, 55–58 → 9–11 ms. With the kernels the frame meets visualize's ~17 ms budget at the default heap, per chunk and in full. A new default-suite pin holds the frame with the kernels to at most 250,000 reductions per chunk, merges alone and in full.
- New:
Macaw.Data.Native.normal_schedulers/1(#60, #61, spec/04 §1.7). It runs a function with every kernel call in its process on a normal scheduler, whatever the call's size. A call on a dirty scheduler is charged reductions by its time, about four a microsecond, so a count with one in it moves with the host's load. A normal-scheduler call is charged a fixed few. Use it for a statement of work in reductions with the kernels, such as the frame's (spec/04 §1.5.1), never for serving: a large call then holds its scheduler past the 1 ms rule.scheduler/1,finish_scheduler/1and the newentry_limit/0answer for it. - Fixed: a bounded call is hung only when it is blocked (#33, #62, #67, spec/02 §6.4, spec/01 §9). The conformance kit's runner and the derived engine's sandbox called any call still running at
timeouthung, so on a loaded host a slow compile or a capped CPU-bound call was misjudged{:hung, …}. Now a call with a reductions cap is judged by its count while it works. It is hung only once it has been waiting inreceive, its reductions unchanged, fortimeoutms on end, and each piece of work gives it the whole bound again. A call with no cap still has only the clock. A wait on another process is a block all the same, so give a stored function a generoustimeout:, as spec/02 §6.4 asks. - Fixed: the first stored function no longer waits on the code server (#69, spec/02 §6.3). In interactive mode a node's first compile loads Elixir's own modules on first use, and a busy code server could hold a bounded compile past its timeout. The application now compiles and runs one trivial function through the sandbox as it starts, so those modules are loaded before any bounded compile. Starting the application costs that one compile.
- Docs: spec/00 names the dependencies as they are, with no pinned version:
macaw_layersrequired, andtable,nxandzigleroptional (#34, #70). Every doc reference resolves, and CI builds the docs with--warnings-as-errors. - Tests: the suite is warnings-clean and stable under load (#37, #55, #56, #58, #63, #64, #65): call traces run in sessions of their own, test files compile with
--warnings-as-errors, and no test's verdict depends on the host's load or on what other tests run beside it. Fifty loaded full-suite runs, with and without Zig, confirm it.
v0.7.0 (2026-10-30)
Terms are the form (dco-tek/macaw/macaw#188), macaw_data's part. A value's identity and validity are the BEAM term's own; JSON and Arrow are adapters at the edge. On macaw_layers 0.3.0. This release is breaking throughout; the guide below is what an adopter of 0.6.1 changes, and the entries after it are the detail.
Upgrading from 0.6.1
- Depend on macaw_layers
~> 0.3.0(#27); its own upgrade guide applies to any kernel call you make directly. - Every schema version changes (D1). A stored version, cache key or golden written by 0.6.1 differs; regenerate it. A schema's names, states and roles are now strings in the struct (
Schema.new/1normalizes them), so code that pattern-matches:latencyinside a schema matches"latency"; every function that takes a name still takes either spelling. - A data color is a color term (D2): a
%Macaw.Layers.Color{}literal or a typed color, never a string. Parse CSS text at your input edge withMacaw.Layers.Color.parse!/1, or write~RGB"#3b6fa8"in code. A role spelled like a CSS name ("orange") is now a valid status role. - A derived source is built from its value (D3). Pass
Derived.new/1a keyword list or a map of terms; read a stored form withDerived.from_map/2orfrom_json/2.to_map/1andto_json/1now answer{:ok, form}or faults. - The operations registry's digest changes (D4); a compile cache keyed by it recompiles once.
Operations.step_digest/2is new: a checked step's canonical identity. - Category values read back as strings (D5): a
:categorycolumn holding:okanswers"ok"fromrows/1,get/2andcolumn/2; compare with strings, or withOperations.same?/2.nil,trueandfalseare kept. - Packed columns are faithful and Arrow-compatible (D5). A mixed integer/float column and a zoned time read back exactly as given; anything that held packed buffers (stored binaries) changes with the bitmap order.
Changes
- Breaking: on macaw_layers
~> 0.3.0(#27): the kernel's digest is over the value, so every pinned version moved before D1 changed what it digests. - Breaking: every schema version changes (D1, #28, dco-tek/macaw/macaw#188).
Schema.versionis the kernel's digest of the schema's term, not of its JSON form, which is now only an adapter at the edge; a value entry's color no longer goes through the JSON form whenSchema.new/1versions the schema, so a typed color there can't crash it. The struct holds every field name, state and role as its string, normalized once bynew/1, sofields,identity/1,window_field/1,compatibility/2's faults andidentity/2's maps name fields by their strings; every function that takes a name still takes either spelling. A state named in both spellings in one description is:duplicate. - Breaking: data colors are color terms, and a string is never a color (D2, #29, dco-tek/macaw/macaw#187, dco-tek/macaw/macaw#188). A value entry's
color, a:colorfield's row values (a derived source's output), a:colorparameter andSchema.color?/1take a literal%Macaw.Layers.Color{}or a theme-relative typed color ({:color, "good"},{:alpha, {:color, "bad"}, 0.74}), its names in a theme's name grammar; a string there,"#2e7d32"or"red", is{:type, :color}.new/1holds a typed color's names as strings, so{:color, :good}and{:color, "good"}are one version. The schema's JSON form writes and reads an entry's color in macaw_layers' color form (Macaw.Layers.Color.Form:{"$rgb": "#2e7d32"},{"$color": "good"}), making no atom.Macaw.Data.Table.column_types/1infers:colorfor a column of color terms; a column of strings stays:text. A status role spelled like a CSS name (:orange) is now a role. Upgrade: read CSS text into a color at your input edge withMacaw.Layers.Color.parse!/1, or write~RGB"#2e7d32"in source (import Macaw.Layers.Color, only: [sigil_RGB: 2]); a stored schema whose entry colors are strings reads back refused, so convert its colors withparse!/1before reading it. - Breaking: a derived source is built from its value, and its forms are an adapter (D3, #30, dco-tek/macaw/macaw#188).
Derived.new/1takes the value only, a keyword list or a map of its fields as terms; it no longer guesses whether it was given a map form, and a map form's string keys are each:unknown_key. The map and JSON forms are written and read by macaw_layers' JSON adapter (Macaw.Layers.Form), with the kernel's tags and the source's own in one tag table:to_map/1andto_json/1answer{:ok, form}or every value without a form, by path;from_map/2andfrom_json/2read through the gates (the limits, the version, the tag table, the keys, making no atom) and then judge the value withnew/1. The output schema is used as built, a%Schema{}as given and fields bySchema.new/1, never written to its form and read back; in the form it isSchema.to_map/1's.Derived.Form's private tag scheme is gone. A color, literal or typed, in a schema's value entry reaches a stored function's context as the term it is; before, its tags were respelled as strings. Upgrade: the form changed: an input is{"$input": [name, window]}, a source's name{"$source": name},per{"$atom": "identity"}or{"$identity": input}, a region an object keyed by field names, andto_json/1stamps"$version": 1. Nothing in this library stored the old form of pair lists, so no version step reads it: write a stored source again withto_map/1from its value. Read a map form withfrom_map/2, notnew/1. - Breaking: the operations registry's
digest/1is the kernel's term digest of its names and modules, not a SHA-256 ofterm_to_binary(D4, #31, dco-tek/macaw/macaw#188), so every registry digest changes and a compile cache keyed by one recompiles once. New:Operations.step_digest/2, a checked step's canonical identity, for caches of a step's results. - Breaking: a table's packed columns are faithful and laid out as Arrow lays them out (D5, #16, dco-tek/macaw/macaw#188), spec/01 §1.1. Packing never changes a value. A category column of strings and atoms is one Arrow dictionary of strings, so
rows/1,get/2andcolumn/2read a category's atom back as the string that spells it (:okand"ok"are one code, read as"ok"), in every form the column takes;nil,trueandfalsestay themselves, beside the names in side bitmaps, so[:ok, true, "warn"]reads back["ok", true, "warn"]. Validity bitmaps are least-significant-bit first, Arrow's order, not Elixir's<<b::1>>order, so every stored packed binary changes. ADateTimekeeps its zone (name, abbreviation and both offsets, in a per-column dictionary, with no time-zone database) and its precision, so a time reads back as given rather than in UTC at microsecond precision. A text column is an offsets buffer and one data buffer, Arrowutf8when every value is valid UTF-8 and Arrowbinarywhen one is not; a category column holding a string that is not valid UTF-8 is kept as its list.new/1takesNaiveDateTimes andDates in a time column as themselves, and refuses a value in a:colorcolumn that is not a color term (Schema.color?/1),{[name, i], {:type, :color}}. Upgrade: compare a category value withOperations.same?/2, or as a string. - Fixed: a number column mixing integers and floats reads back each as given, through an integral bitmap, or a dense union of int64 and float64 when an integer is beyond ±2^53; an integer beyond int64 keeps its column a list rather than becoming a float. A time column keeps each value's zone and precision, so
~U[2026-01-01 00:00:00Z]and~U[2026-01-01 00:00:00.000000Z]stay distinct (#16, dco-tek/macaw/macaw#188).
v0.6.1 (2026-10-29)
- Fixed: compiling macaw_data in an app without the optional
tablepackage warned that Table.Reader.impl_for/1 is undefined; CI now compiles it without its optional dependencies (#7).
v0.6.0 (2026-10-29)
- What a source steps its window by:
Window.before/4,region/2(visualize'sbox/2: a window as the region it asks of a schema),range/2, andcheck/2of the axes alone as well as of a request (dco-tek/macaw/macaw#185, #5). - Packing a table and reading it back do about half the work: types inferred only where not given, values checked in one pass, column names merged once, rows built a column step at a time, and time columns converted by the date's fields, a fifth of
DateTime's own conversion (dco-tek/macaw/macaw#185, #4). - Breaking: a delivery keeps the rows its source gave (dco-tek/macaw/macaw#185).
%Delivery{rows, names, types, served, schema, origin}: nothing is packed or rewritten when it is made;Delivery.rows/1andMacaw.Data.Table.rows/1of a delivery read the rows as they are (a row list uncopied, tuple rows unchanged);Delivery.table/2andtable!/2pack them for a reader that wants columns;named_rows/1names tuple rows and tensors bynames:;empty?/1. Thetablefield is gone: readDelivery.table!/1. Finishing and push keep the rows' form.Macaw.Data.Table.unnamed?/1(#3).
v0.5.4 (2026-10-28)
- The macaw logo in the docs and the README (macaw#175).
- Tests check "makes no atom" on the input's own names, never the node's atom count, and the atom gate forbids the count in
test/(#1).
v0.5.3 (2026-10-28)
- On Hex: depends on
macaw_layers ~> 0.2.2from Hex instead of its git tag, and the README installs from Hex (macaw#155). - The README opens with the early-release notice (macaw#174).
v0.5.2 (2026-10-28)
- Dependencies named by their
dco-tek/macawgroup paths (macaw#152). - Package metadata for Hex: the public home, the changelog link, the files shipped, and docs with
ex_doc(macaw_compose#155).
v0.5.1 (2026-10-28)
- macaw_data v0.5.1: on macaw_layers v0.2.0 (macaw_compose#145)
v0.5.0 (2026-10-27)
- macaw_data v0.5.0: transforms, table operations and aggregates (macaw_compose#83): the behaviors, the registry, twelve table operations, thirteen statistics, Macaw.Data.Ops in the sandbox, budget reducers by name
v0.4.0 (2026-10-27)
- macaw_data v0.4.0: derived sources (macaw_compose#82): the model, the sandbox and its gates, host functions and the engine
v0.3.0 (2026-10-27)
- macaw_data v0.3.0: the source contract (macaw_compose#58): the behavior, description, request and parameters; windows and finishing; push; the conformance kit
v0.2.0 (2026-10-27)
- macaw_data v0.2.0: the data model (macaw_compose#57): units and quantities, the columnar table, the schema, the delivery
v0.1.0 (2026-10-27)
- macaw_data 0.1.0: Macaw.Data.Event