Changelog
View Sourcev0.18.0 (2026-11-16)
Macaw.Data.Window.check/2 answers every malformed axis as a fault, a column of number pairs is inferred :point, Macaw.Data.Table.get/2 reads a present key with one match, and a stale kernel library is refused at start (#294, #295, #297, #298, #300). It changes behavior for a caller that relied on a malformed region key or box passing, and for a reader of an inferred column's type. So this is a minor.
Upgrading from 0.17.x
- A column of
{x, y}number pairs is now:point.Macaw.Data.Table.from/2withouttypes:packs it as:pointwhere it was:term, andMacaw.Data.Table.column_type/2answers:point. Its rows read back equal. A caller that wants the old packing passestypes: %{field => :term}. - A region key that is no name is refused.
Macaw.Data.Source.Request.new/1answers{[:region, key], {:type, :name}}, andMacaw.Data.Window.check/2answers{[:region, key], {:window, :unknown_field}}. A box over a:pointwindow field that is not a box is{[:box], {:type, :box}}fromcheck/2, where it passed. An empty region still passescheck/2. - Every other change is internal or a fault where there was a raise:
check/2no longer raises on a malformed region, range orlatest, andget/2answers as before, faster.
Changes
Macaw.Data.Table.get/2reads a present key with one match (#294). It looked the key up throughkey/2and then fetched it; it now matches the name as given and looks for the other spelling only on a miss. Over 10,000 reads a present key costs 3 reductions a read where it cost 6 (the loop's own counted), and the other spelling 7 where it cost 9 (a string name) and 9 where it cost 11 (an atom name). The answers are unchanged.Macaw.Data.Table.column_types/1infers:pointfor a column of{x, y}number pairs (#298), any range, the test a:pointcolumn packs by.Macaw.Data.Table.from/2withouttypes:therefore packs such a column as:pointwhere it was:term: its rows read back equal, and onlyMacaw.Data.Table.column_type/2answers differently. A list[x, y]is not a point.Macaw.Data.Window.check/2names a malformed axis instead of raising (#295). A region that is no map, a range that is no pair, a region key that is no name, alatestthat is no boolean (nilamong them) and a box over a:pointwindow field that is none each raisedFunctionClauseError, or passed in the box's case. Each is now a fault, the sameSource.Request.new/1gives, and the two share one shape check.Request.new/1also refuses a region key that is no name,{[:region, key], {:type, :name}}. An empty region still passescheck/2.- A stale kernel library is refused at start (#300). The kernels answer the CRC-32 of the Zig sources they were built from,
digest/0, and the start compares it with the sources this build compiled beside: another digest is{:stale, built, loaded}, a failed start undernative: :requiredand a warning with the Elixir path under:optional. The cause is gone too: the toolchain check,__mix_recompile__?/0, moved to a module with no NIF, since Mix loaded the kernels' module to ask it and so bound the previous library on the first build after a kernel change. - The kernels' 1 ms bound is judged by the scheduler thread's CPU time (#297). Each test of the rule read a wall clock around the call, which a loaded host stretches past 1 ms (the weekly stress job failed one run in 54). Each timed kernel now has a probe twin,
<kernel>_timed, built only undernative_probes: true, that answers{answer, ns}fromCLOCK_THREAD_CPUTIME_ID. No change to an adopter's build: its kernels export no probe. - Test and CI only (#291, #292, #293, macaw#320): a property's map over a finite key list never retries; the suite loads every module first, as a release does;
Native.Kernels' build settings are a recorded exception to the family's rule 3 (spec/04 §1.7); the stress script names each failed run's failing tests. Nothing changes for a host.
v0.17.0 (2026-11-12)
A reducer named per reader and finishing chunk by chunk through a cache, one parsed format for numbers and times, the loop over a set of sources as a value, and generated, combined and telemetry sources that pass the conformance kit (#239, #246, #247, #248, #249, #250, #251, #260, #261, #269, #271, #273, #275, #277, #278, #279, #282). It adds public API and changes behavior: a source's table refuses rows out of its window field's order, and the kit holds a subscription to its resolution and position. So this is a minor.
Upgrading from 0.16.x
- Breaking: a source's table whose origin names a window field is sorted by it, with a chunk span or without one.
Macaw.Data.Table.new/2refuses rows out of that field's order with{[:origin, :window], {:unsorted, i}}, andMacaw.Data.Table.append/2of such a table refuses earlier rows with{:error, {:unsorted, 0}}. A caller handing a source's table unsorted rows sorts them first; a late row isMacaw.Data.Table.apply/2's. - Breaking for a holder that read
Macaw.Data.Source.Push.held/1'snoteororderedafter an increment.noteis the last applied answer's (nilwhen that stated none), asstaleis, andorderedisnil. - Breaking for the conformance kit's users: the
:push,:deliveriesand new:monitorchecks can fail a source that passed before. An increment states its subscribed fetch's resolution; a fetch taken withsubscription:set states the subscription's position and no other fetch states one. Such a source was already broken for every holder. - Breaking only for a caller that read a reduced chunk's id's form: a reduced table's chunks are
{source id, {:reduced, name, count}}. Ids are opaque. - New:
Macaw.Data.Source.Request'sreduce({:m4, width},{:lttb, n},:none) and the served factreduced_with; finishing through aMacaw.Data.Operations.Cache(Macaw.Data.Source.Finish.finish/5,reduce/5), withMacaw.Data.Table.Chunk.pick/5. Every existing call answers as it did. - New:
Macaw.Data.Format(spec/05): d3-format numbers, strftime times, and a field's default from its measure. - New:
Macaw.Data.Monitor,Macaw.Data.Source.Signal,Macaw.Data.Source.CombinedandMacaw.Data.Source.Telemetry, which needs the new optional:telemetrydependency (without itstart_link/1answers{:error, :no_telemetry}). - Fixed:
Macaw.Data.Table.compact/1,append/2andapply/2of a table that is not a source's keep its sorted mark only while it is true.
Changes
Macaw.Data.Source.Combinedkeeps the stated resolution across a seam (#282, #284, spec/01 §10.2). Two parts at one resolution on offset grids (a history at 10 from 0, a live part at 10 from 55) made a union holding 50 and 55, closer than the 10 it states, which the conformance kit's:deliveriescheck reports ({:resolution, probe, 5, 10}). A later part now keeps only its rows at least the first part's spacing before the fresher part's served start (a:timefield's in seconds, as the kit reads it): only rows in(served − spacing, served)are dropped, none where the grids align and at most one of each series. Coverage andcappedare judged before the drop. A reader draws a gap by the field'scontinuity, never the resolution, and the seam between parts at one resolution stays under two spacings wide. Over a 10,000-row window (a 2,000-row live part in front of a history), a fetch costs 998,911 reductions, 9.6 ms, against 1,064,289, 11.7 ms, before over tables, and 4,210,045, 46.4 ms, against 4,418,503, 50.4 ms, over rows of a:timefield: the seam's one comparison per row replaces the bound's.- Not breaking against a release:
Combinedis new in this release.
- Not breaking against a release:
Macaw.Data.Source.Combinedrefuses a later part whose rows are finer than the first part's resolution (#278, #283, spec/01 §10.2, §4.3). A combination states the first part'sserved.resolution, its subscription's, so a history at resolution 1 behind a live part at 10 made a delivery whose rows lay closer together than it said, which the conformance kit's:deliveriescheck reports ({:resolution, probe, gap, resolution}). Where the first part's resolution has a spacing (a positive number in the window field's units, or:secondto:weekover a:timefield, as the kit reads it), every later part that kept rows must state a resolution at least as wide, a number read in the first's units; one finer, or one with no spacing to compare, is{:error, {:combined_resolution, position, theirs, first}}. A coarser history (a rollup behind raw live data) combines; a part keeping no rows, and a first part stating no resolution, impose nothing. Refusing keeps the kit's rule true and the subscription's one resolution, and fails at the fetch, the first point a resolution is known; stating the finest would tell a pull and a subscribed reader two spacings, and coarsening would need an aggregation no part was asked for.- Not breaking against a release:
Combinedis new in this release.
- Not breaking against a release:
A push holding's
noteis the last applied answer's, as itsstaleis, and the holding states noordered(#279, #281, spec/01 §4).Macaw.Data.Source.Pushreplaced the holding'sserved.stalewith each applied increment's but kept the snapshot'sserved.note, so a stale snapshot noting "upstream down; last good answer" followed by a live increment heldstale: falsebeside that note, and aMacaw.Data.Monitorreader finished from the holding was told the answer was live and the upstream down; an increment's own note was dropped. An applied increment's note now replaces the holding's, and one stating none clears it; an increment not applied leaves it. Every other fact the holding carries is now stated in §4:capped,reduced_by,reduced_withandparamsstay the snapshot's (an increment's own are not read),window,resolution,positionandcomplete_untilas before, andorderedisnilfrom the snapshot on, since the holding keeps its rows in the window field's order, never a request's (a snapshot'sordered: :sourcewas false ofheld/1).- Breaking for a holder that read
held/1'snoteafter an increment (it was the snapshot's; it is now the last applied answer's,nilwhen that stated none) or itsordered(nownil). A source whose answers stay stale or coarsened says so on each increment.Macaw.Data.Monitorrefinishes every step but the budget and statesordereditself, so its readers' order is unchanged.
- Breaking for a holder that read
Macaw.Data.Source.Combinedis described by its parts (#277, #280, spec/01 §10.2, §2.1). A combination's description held the first part's schema and its ownsupportsalone, so the parts'paramswere dropped andMacaw.Data.Source.fetch/3, which checks a request's parameters against the description, refused every parameter the parts take ({:error, {:params, [{["floor"], :unknown_param}]}}); and it was never viewer independent, and stated nolateness,idle_afterorchunk_span. Each description fact now has one rule: the first part'sparams, and every later part must declare each of them taking every value the first's takes (of its type or a:term, within its closedvaluesif any,nilwhere the first's default isnil), else describe and fetch answer{:error, {:combined_params, faults}}, each{position, name, :missing | {:narrower, first, later}};viewer_independentonly when every part declares it;lateness,idle_afterandchunk_spanthe first part's, its subscription's and its units'. The conformance kit passes a combination of parameterized, viewer-independent parts asked a parameter.- Not breaking against a release:
Combinedis new in this release.
- Not breaking against a release:
Macaw.Data.Source.Combinedstates what its parts said of their answers (#275, #276, spec/01 §10.2, §3). A combination built its delivery from the window,resolution,capped,positionandcomplete_untilalone, so a part answeringstale: true(its upstream failed, it served its last good answer) made a combination that saidstale: false, drawn as live, and every part'snoteandparamswere dropped. Each served fact now has one rule:stalewhen any part asked answered stale, rows or none; the parts'notes, each once, the freshest first, joined by"; "; the first part'sparams,positionandresolution;cappedalso when the oldest part asked says it cut, though it reached the window's start;complete_untilthe least claim of the parts asked (a later part's in the first's units), none when any claims none, where it was the first part's alone and so claimed history settled that the history never claimed, but still the first part's for a snapshot taken for a subscription; and noreduced_by,reduced_with,orderedororigin, which no part is asked. A part's fact no delivery takes (astalethat is no boolean) is{:error, {:combined_delivery, faults}}, not dropped. The conformance kit passes a combination whose first part is stale, and it is stale on every probe.- Not breaking against a release:
Combinedis new in this release.
- Not breaking against a release:
Macaw.Data.Source.Combinedasks its parts the window alone, and finishing reduces the union (#273, #274, spec/01 §10.2). A combination passed a request'sbudget,orderandreduceto every part as they were, so a part naming:budgetreduced its own slice to the whole budget (reduced_by: :source) and the combination's delivery saidreduced_by: nil: the conformance kit's:deliveriescheck found{:unrecorded_reduction, probe, count, total}for a combination of budget-honoring parts, and a union of two parts could hold twice the budget before finishing. A part's request now has nobudget,orderorreduce, as it already had no region orlatest; finishing orders and reduces the union once, by the reader's reducer.as_of,params,viewerandnowreach every part as they are, and the combination now names:as_ofwhen every part does. Over a 4,000-row window (a 1,000-row live part in front of a 4,000-row history, both reducing bylttbunder a budget), a fetch finished byMacaw.Data.Source.Finish.finish/3costs 238,897 reductions, 1.3 ms, at no budget (unchanged); at a budget of 1,000, 943,179 reductions, 11.3 ms, against 1,098,518, 12.1 ms, when the history reduced first and finishing again; at 100, 871,640, 7.7 ms, against 815,076, 7.2 ms. AMacaw.Data.Monitorreader at 1,000 is reduced to 1,000 rows,reduced_by: :runtime.- Not breaking against a release:
Combinedis new in this release.
- Not breaking against a release:
New: the conformance kit's ninth check,
:monitor(#248, #267, spec/01 §9). Each request, at each budget, is read through aMacaw.Data.Monitoras one reader: a tick makes it:live, and its delivery is the source's answer finished for it alone; a source naming:push, afterpush:, stays live with no error and holds more. Its findings are{:monitor_status, probe, status},{:monitor_rows, probe, got, expected},{:monitor_push, probe, why}and{:monitor_faults, probe, faults}.- Breaking for a source that passed the kit before and whose fetches and increments disagree: an increment at another resolution than its subscribed fetch (spec/01 §4.3), or a snapshot stating a position its increments do not follow, now fails
:monitor. The kit's own honest test source did the first, its increments stating no resolution; it states its fetch's now.
- Breaking for a source that passed the kit before and whose fetches and increments disagree: an increment at another resolution than its subscribed fetch (spec/01 §4.3), or a snapshot stating a position its increments do not follow, now fails
The conformance kit holds a fetch's
served.positionto whether it was taken for a subscription (#271, #272, spec/01 §3, §4.1, §9). A snapshot fetched for a subscription (subscription:set) states the number of the last increment it reflects, and any other answer states none. The kit judged a pushed snapshot's position and never a fetched one, so a source answeringposition: nilto the snapshot a holder refetches after a gap passed every check, thoughMacaw.Data.Source.Pushrefuses that snapshot ({:snapshot, :no_position}) and holds nothing. The kit's own honest test source was such a source; it states the subscription's position now.:pushnow holds the subscription's fetched snapshot to a non-negative integer position ({:subscribed_position, got}) and numbers the increments after it past it, as after a pushed snapshot.:deliveriesnow holds every probe's answer, which names no subscription, toposition: nil({:position, probe, got}): such a position belongs to no subscription's numbering, and a holder handed the answer (Macaw.Data.Monitordoes this for a pull pool) takes it as its own.- Breaking for the kit's users: a source that passed
:pushbefore can fail it now if its subscribed fetch states no position, or states one its increments do not number past. A source that passed:deliveriescan fail it if it states a position on an answer not taken for a subscription. Such a source was already broken for every holder. State the subscription's position on a subscribed fetch andnilon every other answer.
- Breaking for the kit's users: a source that passed
The conformance kit's
:pushcheck holds every increment to the subscription's resolution (#269, #270, spec/01 §4.3, §9). It checked each pushed message alone and against the numbering, never against the resolution a subscription keeps, so a source whose fetch statesresolution: 1and whose increments state none passed:pushwhile every holder (Macaw.Data.Source.Push) refused each increment and refetched, live data never merging. The check now fetches the snapshot a holder hands off to (the first request, no budget,subscription:the check's reference) after what arrived with the handshake; the first snapshot, one pushed on subscribe or that fetch, fixes the resolution, and each increment at another is{:resolution, expected, got}(nilequal only tonil). A subscribed fetch that answers no delivery is a:pushfinding too.- Breaking for the kit's users: a source that passed
:pushbefore can fail it now: one whose increments state another resolution than its subscribed fetch or its pushed snapshot (or none beside one), and one whose fetch made withsubscription:set answers an error. Such a source was already broken for every holder; state the subscription's resolution on every increment.
- Breaking for the kit's users: a source that passed
New:
Macaw.Data.Monitor, the loop over a set of sources as a value (#248, #266, spec/01 §6.3; from visualize'sVisualize.Feed.Monitor). Its owner holds it in its state:Macaw.Data.Monitor.new/3takes sources and readers by name, each reader a source and its request fields, and answers{:ok, monitor}or every fault (new!/3);Macaw.Data.Monitor.tick/2(now:the owner's monotonic milliseconds,at:every request'snow) subscribes where a source pushes, polls where it cannot, drains buffered pushes in one merge, trims a sliding window's chunks and finishes;Macaw.Data.Monitor.handle_message/2buffers the pushes the owner receives;Macaw.Data.Monitor.deliveries/1andstatus/1answer each reader. Readers of one request but for budget and reducer are one pool: one subscription and one fetch atFinish.pooled/1, finished once, each reader reduced its own way through the pool'sMacaw.Data.Operations.Cache, a property holding each reader's delivery equal to the answer finished for it alone.ask/2andbudget/2move readers, a pool moving whole keeping its holding; failures back off fromevery:tobackoff_max:. Ten readers of one request cost 1,267,466 reductions (21 ms) on their first tick against 4,696,384 (72 ms) fetched and finished alone, pinned at half; a tick with nothing due costs 2,189 to 2,466 reductions (0.06 to 0.09 ms), pinned at 5,000.new/3joins spec/00 §5's readers. Nothing of a page runtime is in it: pooling across pages, out-of-process fetching and notification stay macaw#108's.New:
Macaw.Data.Source.Telemetry, an in-process:telemetrycollector with one push source per metric (#250, #265, spec/01 §10.3, spec/00 §3; from visualize'sVisualize.Feed.Telemetry).Macaw.Data.Source.Telemetry.start_link/1takesTelemetry.Metricsdefinitions, read by their fields (notelemetry_metricsdependency), and judges every definition and option in the caller before anything starts, answering{:error, faults}(start_link!/1raises them);Macaw.Data.Source.Telemetry.sources/1andsource/2are the metrics' sources: rows%{t, value}and the tags, a schema with the UCUM unit, quantity andmetric_kind, the application's facts fromreporter_options[:macaw](facts_key:names another key), a window ofDateTimes up to the request'snow, pushes numbered per subscription, and snapshots at their position. It passes the conformance kit, andstart_link/1joins spec/00 §5's readers.:telemetryis a new optional dependency (nx already required it). Without it the module compiles warning-free andstart_link/1answers{:error, :no_telemetry};ci/adopter_boot.exs's adopter, built with no optional dependency, now checks both.telemetry_metricsis a test-only dependency.
New:
Macaw.Data.Source.Combined, one source over several, routed by window (#249, #264, spec/01 §10.2; from visualize'sVisualize.Feed.Combine).Macaw.Data.Source.Combined.new/2takes parts freshest first, each a source or{source, context}, and answers{:ok, combined}or its faults (new!/2raises them). The first part is asked the window and each later part the rest before what the fresher ones served, its rows kept in[from, served)so a row at a seam is the fresher part's; later parts' rows are read in the first schema's names and units; tables are concatenated as their chunks; pushes and position are the first part's (and the served facts as #276 states them). It passes the conformance kit, a property holds the union over gaps, open bounds and every kind of time, andnew/2joins spec/00 §5's readers.- Unlike visualize's, a part serving from after the window's end hands on that end inclusive: visualize asked the next part up to the fresher one's served start, past the window, and dropped a row at the window's
tothat no fresher part held (the property found it).
- Unlike visualize's, a part serving from after the window's end hands on that end inclusive: visualize asked the next part up to the fresher one's served start, past the window, and dropped a row at the window's
New:
Macaw.Data.Source.Signal, generated sources whose rows are a pure function of the tick (#250, #263, spec/01 §10.1; from visualize'sVisualize.SignalsandVisualize.Feed.Signals).Macaw.Data.Source.Signal.new/3answers{:ok, signal}or every fault at once (new!/3raises them) for a kind (:sine,:square,:random,:discrete), its fields (the tick first) and the wave'swindow:,period:,min:,max:andseed:;Macaw.Data.Source.Signal.rows/2androws/3are visualize's rows for the same tick, bit for bit. As a pull source it honors:windowover ticks up to the present tick, fixed (tick:,Macaw.Data.Source.Signal.at/2) or read from the request'snow(every:,origin:), capped pastlimit:, its rows a chunked table of the signal's origin. It passes the conformance kit, andnew/3joins spec/00 §5's readers.New: finishing reduces a table chunk by chunk through the caller's cache (#247, #259, spec/01 §2.4, spec/04 §1.5; from visualize's frame-time work).
Macaw.Data.Source.Finish.reduce/5given aMacaw.Data.Operations.Cachein the reducer name's place, and the newMacaw.Data.Source.Finish.finish/5given one after its options, answer{:ok, delivery, cache}: a table's series are reduced bym4orlttbfrom per-chunk ladder partials the cache holds by chunk id and version, folded into a coarser level when the extent or count calls for one, so a warm call costs the chunks that changed and the merge. The answer is the whole reduction's, rows, chunk names and served facts, held by a property over generated tables, series, gaps, chunkings and cut views. Any other delivery, reducer or window field reduces whole, the cache as it was. On visualize#998's loop (50,000 readings, one 1,000-row frame, the newest 50,000 seconds to 1,000):Reducer, path Whole Warm, through the cache m4, Elixir2,055,820 r, 89–97 ms 610,353 r, 25 ms m4, kernels829,569 r, 30–32 ms 207,793 r, 7–8 ms lttb, Elixir1,867,580 r, 44–60 ms 575,783 r, 19–23 ms lttb, kernels681,288 r, 23–27 ms 149,407 r, 4–5 ms The warm costs are pinned (700,000 on the Elixir path, 250,000 with the kernels, each at most a third of the whole). Prune the cache with the held table (
Macaw.Data.Operations.Cache.prune/2).A reduced table's chunks are named stably (#247, #258, spec/01 §1.1.3, §2.4). Finishing's budget step gathered the rows it kept into chunks named by those rows (
{source id, {:subset, rows}}), so a reduced chunk's id changed whenever its kept rows did. Each source chunk's kept rows are now one chunk named{source id, {:reduced, name, count}}(Macaw.Data.Table.Chunk.pick/5, new), for the reducer and count ofserved.reduced_with({:reduced, :none, budget}for the ceiling's stride), its version the derivation digest of the source chunk's version and the slots kept: the id holds on every reduction and the version moves only with that chunk's kept rows. Rows, served facts and cost are unchanged. A host's per-chunk cache keyed by id and version reuses exactly what it reused before (visualize#998's loop under a budget of 4,000: 8 to 10 parts a frame bym4, none bylttb, whose picks move with the window), and its eviction by id now follows the source chunk.- Breaking only for a caller that read a reduced chunk's id's form; ids are opaque, and nothing in this library reads one's form.
New: a request names its reducer, per reader (#246, #257, spec/01 §2.3, §2.4, §3, §6.1, spec/04 §7; from visualize's frame-time work).
Macaw.Data.Source.Requestgainsreduce:{name, count}, a registered reducer by spelling and its own count ({:m4, width},{:lttb, n}, held as{"m4", width}),:none, ornil(as before).Macaw.Data.Source.Request.new/1checks its shape ({[:reduce], {:type, :reducer}},{[:reduce], {:type, :pos_integer}}), and finishing checks the name against the registry before any step ({[:reduce], {:unknown, :reducer, name}},{[:reduce], {:least, least}}).- A named reducer reduces each series and number field at its own count, a line at its pixel width, the request's budget still capping the union;
:nonenever reduces, the budget then only a ceiling; a budget alone reduces exactly as before. - New served fact
reduced_with:{name, count}of the reducer that shaped the rows,nilwhere none did.Macaw.Data.Delivery.new/2refuses another value ({[:served, :reduced_with], {:type, :reducer}}), and the conformance kit's:deliveriescheck finds one a source states withoutreduced_by: :source({:reduced_with, probe, value}). - Pooling with mixed reducers:
Macaw.Data.Source.Finish.pooled/1takes requests beside budgets (a:nonereader, or one with no budget, wants every row), andMacaw.Data.Source.Finish.reduce/5takes a reader's request in its budget's place, so each reader of one fetch is reduced its own way. - Not breaking: every existing call answers as it did; a reduced delivery's
servedcarries the newreduced_with.
- A named reducer reduces each series and number field at its own count, a line at its pixel width, the request's budget still capping the union;
A source's table whose origin names a window field is sorted by it, with a chunk span or without one (#239, #256, spec/01 §1.1.3, §2.6; from visualize#1063).
Macaw.Data.Table.new/2markedsorted:only in event time (a window field and a span), so a push whose description names nochunk_spanand whose resolution gives none held a table in its window field's order thatMacaw.Data.Table.sorted/1called unsorted, and every reader of the newest row read every row. It now marks the window field whenever the origin names one, and the mark holds throughapply/2,append/2,trim/2andcompact/1and through finishing. Reading the newest reading of a 50,000-row spanless holding (last: 1) costs 105 reductions, 0.002 ms, where it cost 508,317 reductions, 5.7 to 7.9 ms. The order is checked on the packed window column, so the snapshot's drain costs 5,152,672 reductions (4,752,843 before), about the same 90 to 115 ms.- Breaking:
Macaw.Data.Table.new/2with an origin naming a window field and no span refuses rows out of that field's order,{[:origin, :window], {:unsorted, i}}at the first,nillast, as it does in event time, andMacaw.Data.Table.append/2of such a table refuses earlier rows,{:error, {:unsorted, 0}}(a late row isMacaw.Data.Table.apply/2's). A caller handing a source's table unsorted rows sorts them first. Macaw.Data.Source.Pushholds a snapshot's rows at their instants: the rows were ordered by the raw window value before the edge converted it, so a time given as an ISO string sorted as missing and was held out of order (with a span, refused as unsorted). They are ordered as the table holds them.
- Breaking:
New:
Macaw.Data.Format, a d3-format specifier read once into data and applied to any number of values (#251, #252, spec/05 §1 to §3).Macaw.Data.Format.parse/2answers{:ok, %Macaw.Data.Format{}}, a struct with no function in it, or{:error, faults}(parse!/2raises them);Macaw.Data.Format.number/2writes any term by it, a non-number asNaN. The notation is d3-format's whole grammar (fill, align, sign,$and#, zero fill, width, grouping, precision,~, and thee f g r s % p d b o x X c ntypes), ported from visualize'sVisualize.Format.formatter/1and byte-identical to it over visualize's 64 d3 cases. A tie rounds away from zero on the exact double, as JavaScript does, where visualize's port rounded some to even (.1eof 0.125 is1.3e-1,.2sof 125 is130); an integer keeps its exact digits under the integer types; a width past 1,024 is refused.Macaw.Data.Format.Localeis a d3 locale definition (decimal, thousands, grouping, currency, minus, nan, percent, and the ISO 4217 codes written by a symbol),Macaw.Data.Format.Locale.new/1building one overMacaw.Data.Format.Locale.english/0. Both readers join spec/00 §5's list.The common formats write through the BIFs (#251, #253, spec/05 §4). A format of type
f,%ordwith no width, no~and the-sign (,.2f,.0%,d,$,.2f) writes a float, or underdan integer, from one:erlang.float_to_binary/2orInteger.to_string/1call, its integer digits grouped by slicing that binary, the string built once; every other format and value keeps the general path, and a property holds the two equal. 1,000 values, median of 15 runs (load 12), against visualize's port and the bare BIF:Format visualize's port General path Now BIF alone ,.2f155,384 r, 5.70 ms 116,195 r, 2.57 ms 13,300 r, 0.33 ms 4,007 r, 0.060 ms d64,304 r, 4.24 ms 62,310 r, 1.26 ms 10,314 r, 0.23 ms 1,007 r, 0.058 ms .0%61,721 r, 4.15 ms 58,629 r, 1.91 ms 12,008 r, 0.22 ms 4,007 r, 0.060 ms $,.2f161,211 r, 6.37 ms 122,023 r, 3.26 ms 13,301 r, 0.35 ms 4,007 r, 0.060 ms The property found
fand%raisingArgumentErrorfor a float past 10²⁵⁵ (the BIF writes at most 256 characters); such a float is now written from its exact integer digits.New:
Macaw.Data.Format.parse_time/2andMacaw.Data.Format.time/2, a strftime pattern read once and every kind of time written by it (#251, #254, spec/05 §5). The directives are visualize'sVisualize.Format.time/2table, written byte for byte as it writes them over its cases (an unknown directive and a trailing%are text); the onesCalendar.strftime/3has are written by it,%L,%U,%w,%Z,%zand%pby this module. A UTC or zonedDateTime, aNaiveDateTime, aDate(midnight, no zone) and integer Unix microseconds (UTC) are all written; any other value is the locale'sNaN.Macaw.Data.Format.Localegainsperiods,days,short_days,monthsandshort_months. 1,000 times by%Y-%m-%d %H:%M: 87,677 r, 0.78 ms, againstCalendar.strftime/2's 83,677 r, 0.73 ms and visualize's 258,419 r, 2.84 ms.New: a field's default format from its measure, a d3 string only overriding it (#251, #255, spec/05 §6).
Macaw.Data.Format.for_measure/2answers aMacaw.Data.Format.Measurefrom a measureMacaw.Data.Quantity.resolve/1completes, andMacaw.Data.Format.measure/2writes a value by it: a:durationin the compound form (1h 24m,840ms, orstyle: :magnitude's2m30s), a:currencyby its code ($1,240.00,¥1,240,1,240.000 BHD), a:ratioas a percent from any dimensionless unit, a:countrounded and grouped, any other quantity rescaled within its unit's family under the system (1.50 MB,3.00 MiB,176.37 lbunder:us), and a number with no measure by an axis's notation (plain, grouped, SI;domain:for one notation across ticks). Ported from visualize'sVisualize.Quantity.format/3,Visualize.Format.currency/2,duration/1,2andVisualize.Format.Auto, byte for byte over their cases, but for what #251 chose: one value is rescaled within its family under:as_postedtoo, a ratio in1is a percent, and a plain number is written as an axis writes it. spec/03's sentences that left the format to the drawing library now point at spec/05 §6.Fixed:
Macaw.Data.Table.compact/1of a table that is not a source's (a step's output fromfrom_chunks/3, a finished delivery's rows) keeps the column it is sorted by (#260, #262, spec/01 §1.1.3). It rebuilt the table throughnew/2, which marks only an origin's window field, soMacaw.Data.Table.sorted/1of the compacted table wasniland every reader that binary-searches the sorted column (a window,last, the ladder's extent, finishing's columns) scanned instead. A property now holds every layout function that keeps the rows' order (compact/1,slice/3,trim/2,warm/1, ascending subsets, an in-orderappend/2) to keeping the mark, with and without an origin, and to the rows being in its order.Fixed:
Macaw.Data.Table.append/2andMacaw.Data.Table.apply/2of a table that is not a source's no longer keep a sorted mark the rows no longer satisfy (#261, #268, spec/01 §1.1.3). Both keptMacaw.Data.Table.sorted/1whatever rows they added, so rows appended or inserted before the last ([1, 2, 3, 0]still marked"x") made every reader that binary-searches the sorted column (finishing's window,last, the ladder's extent, thewindowstep's newest reading) answer wrong rows, silently. The mark is now kept exactly while it is true and dropped otherwise: anappend/2whose rows start at or after the table's last and are in the column's order (by their own mark, or checked), anapply/2that inserts nothing, merges under a schema whose window field is that column, or lays in-order rows after the held ones. Rows are never refused for it. Only the rows added and the held table's last row are read: appending 100 rows to a 50,000-row table costs 1,245 reductions, 0.015 ms (55, 0.001 ms before, when nothing was checked), 261 when they are marked; inserting 100 rows costs 13,375 reductions (12,239), about the same 0.13 to 0.19 ms; the check's cost is pinned per row added and is the same over 1,000 held rows as 50,000. A source's table, whose paths already keep its order, is unchanged (apply/2of 100 rows 16,178 reductions,append/222,140). The #260 property now covers rows in any order, with retractions and schemas, and holds every reader that trusts the mark to its answer on the same rows unmarked.- Not breaking: the mark is not part of the contents (
equal?/2,digest/1); only a mark that was false is withdrawn, which costs the reader a scan instead of a wrong answer.
- Not breaking: the mark is not part of the contents (
v0.16.0 (2026-11-11)
A documented entry for a host's own counted process, packed :point and :color columns, and a drain into a holding laid by row count that joins its rows after the last chunk's (#224, #231, #236, #238). Macaw.Data.Native.enter/1 is new public API and now refuses a scope it doesn't know, so this is a minor.
Upgrading from 0.15.x
- New:
Macaw.Data.Native.enter/1is the public entry for a host that counts reductions in a process it spawned:enter(scope)putsscope's flags in the calling process for the rest of its life, so call it only in a process that ends with the scope, and usescope/2anywhere else. Breaking for a typo only:
enter/1raisesFunctionClauseErrorfor a scope outside:elixir | :normal_schedulers, asscope/2does.enter(:bogus)answered:okand left the process counting on the wrong path. A caller passing:elixiror:normal_schedulerschanges nothing.:pointand:colorcolumns are packed, and a drain into a holding laid by row count joins its rows after the last chunk's. Every answer, chunk id and digest is the one 0.15.0 gives;row/2on such a column costs the same wherever the row lies, and digests and drains cost a fraction of what they did. No call changes; the tables are below.Macaw.Data.Schema.new/1andfrom_map/1answer a structstatus:besideorder:orvalues:as{[name, :status], {:type, :status}}, where they raisedProtocol.UndefinedError. A caller that rescued nothing gets a fault where it crashed.- The spec names no versions (#238). Nothing in the package changes.
Changes
New:
Macaw.Data.Native.enter/1is documented, the public entry for a host's own counted process (#224, #237, spec/04 §1.7). Since 0.14.0Native.elixir/1andnormal_schedulers/1run their function in a process of their own, so a host that counts reductions in a spawned process, as visualize'sVisualize.Work.reductions/2does, enters the scope in that process instead.enter(scope)puts the flags ofscope, and whichever of the two the process already holds, in the calling process for the rest of its life: it is only for a process that ends with the scope, andscope/2is the scope for any other caller, since nothing clears the flag. It now refuses a scope outside:elixir | :normal_schedulerswith aFunctionClauseError, asscope/2does;enter(:bogus)put{nil, true}in the caller's dictionary and answered:ok, so a typo counted on the wrong path with no sign.A drain into a holding laid by row count joins its rows after the last chunk's as they lie (#231, #233, spec/01 §1.1.4). A holding whose chunks are aligned by row count (no window field, a
:pointwindow, or a number window at no resolution, so with no span) tookMacaw.Data.Table.apply/2's append path, which read every held row of the last chunk and packed them again with the increment's. Now the rows filling it are packed by themselves and joined after the held ones column by column (Chunk.written_from/4, as an open chunk's in-order rows have been since #213), and the last held row's window value is read from its column's last slot rather than by reading the chunk. A column the join can't keep exact (a list form among them) still writes the chunk from all its rows, so every chunk, id and digest is the one the full rewrite builds, which a byte-equality property against the frozen copy holds over every such layout, columns of every form and a last chunk with room for all, some or none of the rows. A 1,000-row drain into 6,000 held rows, kernels loaded, median of 15 runs (load 13):Holding Before After number window, no span 1,025,878 r, 13.0 ms 237,832 r, 2.7 ms no window field 683,820 r, 10.1 ms 147,780 r, 2.8 ms :pointwindow (its column a list until #234)1,366,285 r, 24.8 ms 1,126,277 r, 15.2 ms A
:pointcolumn is packed: two number columns and a validity bitmap (#231, #234, spec/01 §1.1). It was always a list, soMacaw.Data.Table.row/2read it from its chunk's start, its digest encoded each point by itself, and a drain into its chunk wrote the chunk again from all its rows. Each coordinate now takes the form a number column of it takes (int64, float64, integral or a dense union), so integer and float coordinates read back as given; a coordinate past int64 keeps the column a list, as it keeps a number column one. No answer changes, and no digest: the canonical streams are built from the coordinates' buffers and are the list form's byte for byte, which a property holds over generated columns and a replay of every point column the suite hashed (6,458 distinct) confirmed. One 8,192-row chunk of float points beside a float column, kernels loaded, ms the median of single calls (load 3 to 8):Measure Before After row/2at row 0 / 4,096 / 8,00088 / 4,184 / 8,088 r; 0.001 / 0.016 / 0.035 ms 126 r at each; 0.001 to 0.002 ms Macaw.Data.Table.digest/1328,424 r, 10.5 ms 80,399 r, 2.0 ms a 1,000-row drain into 7,000 held, a :pointwindow1,077,844 r, 12.9 ms 335,823 r, 5.3 ms the same, a number window beside a point column 1,289,525 r, 17.2 ms 313,447 r, 4.4 ms A
:colorcolumn is packed: its distinct terms once, an int32 code per value and a validity bitmap (#231, #235, spec/01 §1.1). It was always a list, with the list's costs. The terms are kept in the order they first appear and told apart as a map key tells them, so{:ramp, :viridis, 1}and{:ramp, :viridis, 1.0}, or0.0and-0.0, stay two, and each value reads back as the very term given. The form has its own tag, so grouping and filtering, which read a category's dictionary as strings, take their general path for it as before. No answer changes, and no digest: the canonical streams encode each distinct term once and are the list form's byte for byte, which a property holds over generated columns and a replay of every color column the suite hashed (4,536 distinct) confirmed. One 8,192-row chunk of four distinct colors beside a float column, kernels loaded, ms the median of single calls (load 8 to 9):Measure Before After row/2at row 0 / 4,096 / 8,00088 / 4,184 / 8,088 r; 0.001 / 0.019 / 0.031 ms 90 r at each; 0.001 ms Macaw.Data.Table.digest/1606,770 r, 16.2 ms 126,730 r, 1.6 ms a 1,000-row drain into 7,000 held, a number window beside a color column 1,901,577 r, 61.2 ms 377,461 r, 8.0 ms Fixed:
Macaw.Data.Schema.new/1andfrom_map/1answer a structstatus:besideorder:orvalues:as the status's fault,{[name, :status], {:type, :status}}, where they raisedProtocol.UndefinedError(#236, #240, spec/00 §5, spec/01 §1.3). A struct is a map tois_map/1, and the two cross-checks of a status againstorder:andvalues:enumerated it; they now read only the plain mapstatus:accepts. This also removes a random CI failure: the reader property reached it only when a generated description drew a struct at:statusand anorder:orvalues:beside it (first at seed 112837). The property now puts each struct at every description key, beside each cross-checked companion, on every run.Docs: the spec names libraries, never versions (#238, #241, as dco-tek/macaw/macaw#297). Versions live in mix.exs,
.tool-versionsand this file: spec/00 §4 and the spec index say what is done and cite its issues, and where a version marked a change the spec cites the issue. A new test fails on a library version in spec/:vX.Y, a library named before a version, a~>requirement, or a release aftersince,fromoruntil.
v0.15.0 (2026-11-10)
A ceiling that refuses a budget it can't use, many rows of a table read by index in one gather, and a push drain that reads each incoming row once (#199, #205, #215). A budget that is no positive integer is now refused, which is breaking, and there is new public API, so this is a minor.
Upgrading from 0.14.x
- Breaking:
Macaw.Data.Source.Finish.ceiling/2refuses a budget that is neithernilnor a positive integer:{:error, [{[:budget], {:type, :pos_integer}}]}, answered with anyhost:andpage:faults, the budget's first, andceiling!/2raises it.ceiling(0, host: 10)was{:ok, {0, false}}, and an atom budget read as the ceiling itself. A caller that hands it a budget it has not judged judges it first, or reads the fault; a budget fromMacaw.Data.Source.Request.new/1is already judged, so a request's budget never meets it. - New:
Macaw.Data.Table.rows_at/2androws_at!/2read the rows at many indexes in one gather, in the order given, each the rowMacaw.Data.Table.row/2answers. Use it in place ofrow/2in a loop: over a:termcolumnrow/2reads from its chunk's start, so a loop of it is quadratic in the chunk's rows. Macaw.Data.Source.Push.drain/2answers what it answered, chunk for chunk and digest for digest, and costs less: a 1,000-row increment into a nearly full open chunk 439,545 → 243,533 reductions, and the first drain after a snapshot 785,932 → 591,168. No call changes; the table is below.ci/raises.exsis the family's current copy (#223). Nothing in the package changes.
Changes
A push drain reads each incoming row once (#215, #228, #229, spec/01 §1.1.4, §4).
Macaw.Data.Table.apply/2's insertions built each row's slot as aDateTimeto group it, read its instant again to sort it, and matched every key of every row against every column. Now each row's window value is read once, from the increment's packed window column where packing has computed it. Its slot key stands for its slot exactly: aDateTimeand a naive time at one instant are still two slots, so no chunk id or digest moves. Each slot's term is built once, from one of its rows, and an increment's keys are matched to the header once. Every chunk, fault and count is the one the earlier code gives, which byte-equality properties against a frozen copy of it hold over mixed kinds of time and number, rows into settled slots, and keys spelled as strings, atoms or neither. A 1,000-row drain on the live loop's holding, kernels loaded, median of 21 runs (load 2.8 before, 6.6 after):Drain Before After open chunk nearly empty 402,655 r, 5.9 ms 206,643 r, 2.6 ms open chunk nearly full 439,545 r, 4.8 ms 243,533 r, 2.6 ms first after the snapshot 785,932 r, 10.5 ms 591,168 r, 8.2 ms late, from 500 s before the open chunk's last row 611,563 r, 7.2 ms 351,539 r, 4.3 ms Slotting and sorting fell by about 112,000 reductions, matching keys to the header once by about 44,000, and reading positions from the packed column by about 40,000 more.
New:
Macaw.Data.Table.rows_at/2androws_at!/2read many rows by index in one gather (#205, #226, spec/01 §1.1).{:ok, rows}in the order given, repeats kept, each row the oneMacaw.Data.Table.row/2answers andnilpast the end; an index that is no non-negative integer is{[k], {:type, :non_neg_integer}}, every fault at once, and indexes that are no proper list{[], {:type, :indexes}}. It never raises (spec/00 §5 lists it among the readers).row/2reads a:termcolumn, which a chunk holds as a list, from the chunk's start, so reading rows one at a time is quadratic in the chunk's rows;rows_at/2costs the same per row wherever it lies.row/2itself is unchanged. 50,000 rows × 9 columns, kernels loaded, median of 9 runs:Read All packed One :termcolumnrow/2over 1,000 ascending indexes450,553 r, 13.7 ms 4,445,107 r, 33.3 ms rows_at/2, the same 1,00095,584 r, 4.4 ms 96,975 r, 7.9 ms rows_at/2, the same 1,000 shuffled139,971 r, 6.8 ms 201,852 r, 8.2 ms row/2over every row22,501,767 r, 1,236 ms 223,962,959 r, 2,284 ms rows_at/2over every row3,655,400 r, 277 ms 3,555,351 r, 296 ms A gather of rows that repeat, in another order, reads each row it names (#205, #226). The internal gather
Macaw.Data.Table.rows_at/2is built on took a chunk's kept rows as a contiguous run when they spanned as many places as they held, which rows that repeat can do:[10, 10, 12, 13]read rows 10 to 13. No released path passed a repeated index (every caller's were distinct), so no answer of 0.14.0 changes.Breaking:
Macaw.Data.Source.Finish.ceiling/2refuses a budget that is neithernilnor a positive integer (#199, #225, spec/01 §6.1). Its last clause answered whatever wasn't above the limit, soceiling(0, host: 10)was{:ok, {0, false}},ceiling(-1, …)andceiling(1.5, …)answered themselves, andceiling(:nope, host: 10)was{:ok, {10, true}}, an atom ordering above every number. Each is now{:error, [{[:budget], {:type, :pos_integer}}]}, answered with thehost:andpage:faults at once, the budget's first, andceiling!/2raises it. A budget fromMacaw.Data.Source.Request.new/1was already judged the same way, and no caller in the family is affected: macaw's surface fetch passes a budget macaw_component has already judged.CI: the family's current
ci/raises.exs(#223). The family's raise check counts a raise inside a__using__or__before_compile__quote as compile time since 2026-11-09, somix hex.publish's check against the family reference refused 0.14.0 until it was published with the earlier copy. The new copy finds the same raise sites inlib/, every one allowlisted;ci/is not in the package.
v0.14.0 (2026-11-10)
macaw_layers 0.5.0, native kernels that answer their faults as terms, a push drain that costs the rows an increment brings, and a table whose kernel digest is its contents (#101, #202, #204, #216, #218, #220). An adopter takes macaw_layers 0.5.0 with it, a few calls that raised now answer faults, Macaw.Data.Native.elixir/1 and normal_schedulers/1 run their function in a process of its own, and there is new public API, so this is a minor.
Upgrading from 0.13.x
- Take macaw_layers 0.5.0 too:
{:macaw_layers, "~> 0.5.0"}. 0.14.0 does not resolve against 0.4.x, and a project that calls macaw_layers itself takes that release's own changes. The move to 0.5.0 changes no digest by itself. - Calls that raised
ArgumentErrorfrom the kernel now answer faults, so a test rescuing it changes:Macaw.Data.Operations.step_digest/2:{:error, {path, :not_data}}for a node holding a function, PID, reference, port or improper list;Macaw.Data.Operations.Step.parts/4, and soStep.run/4andMacaw.Data.Operations.Cache.run/5:{:error, {key, :not_data}}.Macaw.Data.Schema.from_json/2,Macaw.Data.Derived.from_json/2andMacaw.Data.Derived.from_map/2:limits:that are no limits answer faults under[:limits], such as{[:limits, :depth], {:limit_value, 64}}.
Macaw.Data.Table.new/2refuses an origin source or resolution that is not data,{[:origin, key], :not_data}. It accepted one, and a sort or merge over the table raised later.Macaw.Data.Native.elixir/1andnormal_schedulers/1run their function in a process of their own. The function must not use the caller'sself(), mailbox or process dictionary (only$callersis carried): asend(self(), …)inside it reaches its own process, and areceivewaits there. Its reductions are not the caller's, so a count taken around either call sees only the plumbing; take it inside the function. A raise, throw or exit reaches the caller as before.Macaw.Data.Native.scope/2answers{:ok, result}or{:error, {:down, reason}}instead of passing an exit on.- A kernel fault that is this library's defect raises
RuntimeError, naming the kernel, where anErlangErrorsuch as:ShortBufferwas raised. Out of memory in a kernel is answered by the Elixir path. Nothing a caller passes reaches the raise, so only code rescuingErlangErroraround a step changes. - Kernel digests of a term holding a table or a chunk move:
Macaw.Layers.Document.digest/1andencode/1followMacaw.Data.Table.equal?/2, so equal tables chunked otherwise give one digest. A step identity whose:termoption holds a table moves with them. Anything you stored keyed on such a digest is recomputed;Macaw.Data.Table.digest/1of a table with no table among its values, and every chunk digest, is unchanged. - New:
Macaw.Data.Source.Push.cache/1,position/1anddrained/1; read them in place of the holder's struct fields.Macaw.Data.Native.scope/2andscope!/2. Macaw.Data.Source.Push.drain/2answers what it answered and costs the rows an increment brings: a 1,000-row increment into a nearly full open chunk 1,551,511 → 439,513 reductions (33.1 → 5.7 ms), and the first drain after a snapshot 10,266,302 → 785,865 (171.7 → 7.4 ms). Each kernel call costs 2 to 4 reductions more andMacaw.Data.Table.Chunk.gather/517 more; the tables are below.
Changes
A native kernel answers a fault as a term, never as a Zig error (#101, #208, spec/04 §1.7). Zigler turned a kernel's error return into an
ErlangErrorthrough its error-return trace, whose debug-info globals it initializes on first use without a lock. A kernel now answers{:fault, reason}(:short_buffer,:not_a_binary,:not_a_piece,:unknown_form,:out_of_memory), and a compile-time check refuses one that returns an error union. Out of memory is answered by the Elixir path. Any other fault is a defect in this library's own packing and raises aRuntimeErrornaming the kernel and its arguments' shapes, where anErlangErrorsuch as:ShortBufferwas raised before. The kernels' interface version is 8. Each kernel call counts 2 to 4 reductions more (a ladder partial over 8,192 rows: 39 → 42; the column sum: 215 → 219); its time is within noise.A native kernel frees what it allocated when an allocation fails (#101, #209, spec/04 §1.7).
group_sumleaked its hash index whenTable.init's second allocation failed, found by extla's allocation model. Out of memory only, so no working call changes. Every kernel allocation now goes through one counted path, and the suite fails each allocation of each kernel in turn and finds nothing live at any return.Macaw.Data.Native.elixir/1andnormal_schedulers/1run their function in a process of their own (#101, #210, spec/04 §1.7). Each put its flag in the caller's process and cleared it after the function, so a raise the caller caught left that process on the Elixir path, or on normal schedulers, for the rest of its life. Now the flag lives and ends with the function's own process, through the newNative.scope/2({:ok, result}or{:error, {:down, reason}}) andscope!/2, which passes a raise, throw or exit on to the caller. The scopes nest. Behaviour change: the function's reductions are its own process's, not the caller's, so a caller counting reductions around either call counts only the call's plumbing; count inside the function's process instead. The function must not depend on the caller's mailbox orself(). The suite'sMacaw.Data.Test.Work.reductions/2takesnative: :elixir | :normal_schedulers, and its counts are 7 reductions lower than through the old scopes, which counted their own flag (a 2,000-row ladder partial: 18,879 → 18,872 on the Elixir path, 42 → 35 with the kernel).New:
Macaw.Data.Source.Push.cache/1,position/1anddrained/1(#202, #212, spec/01 §4), so a holder reads its subscription's schema cache, the number of the last message taken and how many snapshots and increments the lastdrain/2applied without the struct's fields.drain/2answers as before. New public API, so the next release is 0.14.0.Macaw.Data.Source.Push.drain/2costs the rows an increment brings, not the rows already held (#204, #213, spec/01 §1.1.4, §4).Macaw.Data.Table.apply/2wrote an open chunk again whole on every increment: every held row read, sorted and packed. Now only the increment is sorted, and rows starting at or after the chunk's last row are packed by themselves and joined after its columns as they lie, wherever the join is exactly what packing every row gives; any other chunk is written whole as before. The first drain after a snapshot folded the whole window column twice to find the served window's ends; it reads the column's positions instead, with the same answer. No answer changes: every chunk is byte for byte the one written before, digest and version included.Macaw.Data.Test.LiveLoop, a 1,000-row increment into the 50,000-row holding, kernels loaded, median of 11 drains:Drain Reductions ms open chunk nearly empty (56 rows) 496,536 → 402,622 6.2 → 4.9 half (4,056) 1,098,685 → 422,709 23.5 → 6.0 nearly full (7,056) 1,551,511 → 439,513 33.1 → 5.7 first after the snapshot 10,266,302 → 785,865 171.7 → 7.4 A late increment rewrites only the held rows after the first it moves (#204, #214, spec/01 §1.1.4). The first held row later than the increment's earliest is found by binary search over the open chunk's packed window column; the rows before it are kept as they lie, and only those after it are read, merged with the increment and packed. A dictionary column now joins at any split, its kept rows' dictionary cut to the strings they hold. No answer changes. A late 1,000-row increment into the nearly full open chunk (7,056 rows), starting before its last row by:
Starts Reductions ms 500 s 1,567,584 → 611,530 43.8 → 8.3 3,000 s 1,571,582 → 911,552 43.4 → 19.6 7,000 s 1,575,583 → 1,243,565 39.3 → 33.2 Depends on macaw_layers 0.5.0 (#216, #217, spec/00 §5), whose functions answer values where they raised;
{:macaw_layers, "~> 0.5.0"}. Every digest is unchanged: each of the 18,985 distinctMacaw.Layers.Document.digest/1calls the 0.4.0 suite made answers the same bytes under 0.5.0, and no golden or stored digest moves. Where a call now answers a fault:Macaw.Data.Operations.step_digest/2answers a node holding a term that is not data (a function, a PID, a reference, a port, an improper list) as{:error, {path, :not_data}}, its path in the node, andMacaw.Data.Operations.Step.parts/4(soStep.run/4andMacaw.Data.Operations.Cache.run/5) as{:error, {key, :not_data}}; each raisedArgumentErrorfrom the kernel. A nodeMacaw.Data.Operations.validate/3checked holds none.Macaw.Data.Table.new/2refuses an origin source or resolution that is not data,{[:origin, key], :not_data}(spec/01 §1.1.3). It was accepted, and a sort or merge over the table then raised, since a combined chunk's id is the digest of its inputs' ids.Macaw.Data.Schema.from_json/2,Macaw.Data.Derived.from_json/2andMacaw.Data.Derived.from_map/2answerlimits:that are none as faults under[:limits]({[:limits, :depth], {:limit_value, 64}},{[:limits], {:type, :keyword}}), reading nothing, where the kernel raisedArgumentError.- The kernel's digest of a term this library built (a chunk's derivation and combined id, a step's identity over a checked node, a registry's names, a schema's fields) is one named invariant in
ci/raises_allowlist.exs;Url.Policy.new!([]), configuration written in code, is the one!twin of the kernel called. - The digest's answer costs a gather 17 reductions more (
Macaw.Data.Table.Chunk.gather/5×1,000 on one chunk, median of 21, load 4 to 8):
Path Reductions ms Chunk.gather/5×1,000 (a derivation digest each)255,006 → 272,006 22 → 27 filterover 50,000 rows, 84 chunks38,110 → 40,297 2.3 → 2.4 sortover 50,000 rows (a combined id)3,187,023 → 3,187,599 142 → 140 Document.digest/1×10,000 on a derivation term1,300,007 → 1,590,007 25 → 27 A table's digest uses the kernel's encoder, not a copy of it (#218, #219, spec/01 §1.1.2).
Macaw.Data.Table.digest/1encoded its header and every value no packed form holds with a byte-for-byte copy of the kernel's canonical encoder, kept while macaw_layers held it private; it now callsMacaw.Layers.Document.encode/1, so a change to the kernel's encoding moves table digests withMacaw.Layers.Document.digest/1's. No digest moves: the 86,841 table digests the suite computed on 0.13.1's code (45,899 distinct, 2,574 integers beyond int64, 40,863 other terms and 28,867 zoned times among their values) replay identical, and no golden or stored digest changes. Two things follow from one encoder:- A struct in a column is encoded as the kernel encodes it, by its canonical term (
Macaw.Layers.Digest, macaw_layers 0.5.0), where the copy encoded its map. No struct in this library or its family implemented the protocol then, so no digest moved (a table and a chunk do since #220, below); a caller's struct that does now has one digest however it is laid out.Macaw.Data.Table.new/2judges a struct by that canonical term too, so one whose canonical term is not data is{[name, i], :not_data}, never a raise when the table is hashed. - The copy's own
ArgumentErroris gone; a fault from the encoder over a term this library built is the one named invariantci/raises_allowlist.exsalready held.
The kernel's encoder tracks a fault's path, so a value it encodes costs more: 4 reductions a big integer, 7 a small tuple or map. A zoned time column now encodes each of its zones once rather than a zone per value, from its packed zones, so it costs a quarter of what it did. Kernels loaded, the digest of one 10,000-value column, ms the median of 21, load 1 to 3:
Column Reductions ms integers beyond int64 186,550 → 226,584 6.3 → 6.3 tuples, maps, atoms and lists ( :term)520,745 → 594,808 9.2 → 9.6 zoned times 976,010 → 231,948 17.3 → 2.7 int64 16,059 → 16,078 0.10 → 0.10 A packed column's digest hashes no such value and costs what it did. A gather, a filter and a sort encode nothing, so they cost what they did (100 gathers of 100 rows: 96,607 reductions; a filter and a sort over 50,000 rows: 209,026 and 3,713,531; each before and after); the digest of their result pays the per-value cost above for its
:termand big-integer columns (a 50,000-row table with one of each and a number column: 2,785,252 → 3,485,633).Macaw.Data.Source.Push.drain/2of 1,000 rows into a nearly full open chunk: 439,512 → 439,545.- A struct in a column is encoded as the kernel encodes it, by its canonical term (
The kernel's digest of a table is of its contents, not its layout (#220, #221, spec/01 §1.1.2).
Macaw.Data.TableandMacaw.Data.Table.ChunkimplementMacaw.Layers.Digest, soMacaw.Layers.Document.digest/1andencode/1of any term holding one followMacaw.Data.Table.equal?/2: the same rows chunked otherwise, compacted, sliced out of a longer table or held as a source's table give one digest, wherever the table sits in the term. They hashed the struct's fields before, so equal tables laid out differently got different digests, and a cache keyed on a term holding a table missed where it should have hit.- A table's canonical term is
{"macaw.table", [{name, type}, …], digests}, the content digests of its rowsMacaw.Data.Table.chunk_rows/0at a time: the digests a canonical table of the same rows keeps in its chunks. A table of whole canonical chunks reads them and hashes nothing, and fetches no cold chunk; any other layout hashes its rows a run at a time. A chunk's is{"macaw.table.chunk", [{name, type}, …], digest}, its content digest, never its id. A cold chunk, which held a function, now has a digest. - Digests that move: every kernel digest over a term holding a table or a chunk: a step's identity (
Macaw.Data.Operations.step_digest/2,Macaw.Data.Operations.Step.parts/4) and the step key under derivation digests and combined chunk ids where a:termoption holds a table, a chunk id whose origin source or resolution holds one, andMacaw.Data.Table.digest/1of a table whose column holds a table value. Nothing in this library or its family persists or compares such a digest across versions: its caches live in process memory, and a schema's version is over fields, which hold no table. The family'slib/digests no table (macaw's only kernel digest is a layout template's). Every otherMacaw.Data.Table.digest/1and every chunk digest is unchanged: each of the 22,341 distinctMacaw.Layers.Document.digest/1and table-encoding calls the suite made on the previous code (18,816 and 3,525) replays identical, and none held a table. - Cost, 50,000 rows by four columns (UTC time, integer, float, category), app started, median of 7, load about 4 before and 18 after:
Kernel digest of the table Reductions ms canonical chunks ( new/1,from/2,compact/1)318,405 → 496 1.4 → 0.007 other chunking (an append of two pieces) 318,405 → 368,257 1.3 → 3.7 Macaw.Data.Table.digest/1, for comparison395,067 → 395,067 3.5 → 3.5 - A table's canonical term is
v0.13.1 (2026-11-06)
A sliding Macaw.Data.Source.Push.trim/2 costs almost nothing (#201). It decides from each chunk's last value and no longer decodes and compares the chunk's rows, so a holder that trims every frame stops paying for it.
Upgrading from 0.13.0
No working call changes, and there is no new API: trim/2 answers what it answered. Each cost below is before to after.
A whole
trim/2, 50,000DateTimerows in 7 chunks (the first 1,792 rows, then 8,192), median of 9:fromReductions ms at the first row 25,892 → 139 1.228 → 0.003 late in the first chunk 159,711 → 139 4.209 → 0.005 past three chunks, late in the fourth 2,173,879 → 467 55.142 → 0.011 past every chunk 4,144,984 → 768 147.217 → 0.018 A frame,
Macaw.Data.Test.LiveLoopplus a trim each frame, kernels loaded, median of 15 frames from one run per version, 0.13.0 to 0.13.1. The ms are noisy between runs, so read the reductions.Setup Reductions ms no budget 2,005,232 → 1,555,543 60.74 → 44.63 budget 2,000, lttb2,665,483 → 2,196,671 75.64 → 66.46 budget 2,000, m42,552,940 → 2,139,423 68.09 → 60.83
Changes
Macaw.Data.Source.Push.trim/2reads each chunk's last value (#201, spec/01 §4). It decoded every chunk it walked and compared each value until one wasn't beforefrom, so its cost followed the rows. On a 50,000-row holding a sliding trim costs about 290 reductions where it cost about 437,000, and it keeps what it kept.
v0.13.0 (2026-11-05)
Errors are values past the readers: every reader has a ! twin, an improper list and arithmetic past a float's range answer faults, and Macaw.Data.Reduce, Macaw.Data.FFT and Macaw.Data.Source's dispatch answer values (#184, #169, #181, #185, #193, #192, #191, #195). A breaking change for a direct caller of Macaw.Data.Reduce or Macaw.Data.FFT, and new ! twins.
Upgrading from 0.12.x
Breaking:
Macaw.Data.Reduce.lttb/3,Macaw.Data.Reduce.m4/3,Macaw.Data.FFT.magnitudes/1andMacaw.Data.FFT.window/2answer{:ok, value}or{:error, reason}. Each answered a bare list. Calllttb!/3,m4!/3,magnitudes!/1orwindow!/2for the old answer.lttb/2andm4/2change with them. Thelttbandm4steps keep the shape of their answers.New
!twins:Macaw.Data.Table.from!/2,Macaw.Data.Schema.from_map!/1,Macaw.Data.Schema.from_json!/2andMacaw.Data.Derived.from_map!/2.Calls that worked near a float's range are now refused, with a fault in place of the answer.
8.0e307is the bound called half the largest float below.Macaw.Data.Reduce.lttb/3:{:values, :overflow}whencount × |x|,count × |y|or4 × x range × y rangereaches8.0e307.Macaw.Data.Reduce.m4/3:{:values, :overflow}for anxat or above8.0e307, and{:width, width}for a width at or above it.Macaw.Data.FFT.fft/1:{:values, :overflow}when the absolute parts sum to8.0e307or more.Macaw.Data.FFT.magnitudes/1refuses a term whose square passes a float's range.Macaw.Data.FFT.spectrum/3:{:values, :overflow}when the absolute sum of the readings reaches1.0e152, and{:every, :overflow}for an interval whose frequencies pass a float's range.- The
lttbandm4steps:{[i, "fields"], :overflow}for anxat or above8.0e307, a number at or above1.0e301under atimescale or past a float's range underlog, and forlttba line whose triangles or bucket means could pass the range.
Macaw.Data.Operations.validate/3refuses what is no data in a step's:termoption. A function, a PID, a reference, a port or an improper list infilter'svalueis{[i, "value"], {:type, :term}}.validate/3accepted them andapply/4raised, so a test that only validated such a node changes.Macaw.Data.Source.Finish.ceiling/2refuses ahost:orpage:that is no positive integer,{[key], {:type, :pos_integer}}, every fault at once.host: 10.5andhost: 0answered before, andhost: :nopeapplied no limit at all.ceiling(5, host: 0)is now an error, so a caller using0for "nothing" must say so itself.Macaw.Data.Source's dispatch answers{:error, :not_a_source}for a term that is no source, fromdescribe/2,fetch/3,subscribe/2and adescriber/1lookup, holding nothing of the term.unsubscribe/2answers:okfor such a term. Adescriber/1lookup of a non-tuple entry answered{:ok, entry}as if it were a schema, which was a bug. A test assertingFunctionClauseErrorfrom any of these changes.An
:overflowfault reads "is beyond a float's range" where it read "sums beyond a float's range". A test on those words changes.An improper list answers a fault at every reader, where it raised.
Macaw.Data.Table.new/2has{[], {:type, :columns}}andMacaw.Data.Operations.validate/3{["nodes"], {:type, :nodes}}, both new.Macaw.Data.Derived.from_map/2still raises on an improper list inside a map form: that is the kernel's to refuse (spec/00 §5).spec/04 §4.4's
lttbtolerance is wider for a sparse-spike series: twice the exact algorithm's error plus0.005, plus1/nfor each peak kept beyond the exact algorithm's. No picked row changes.Cost at 50k rows, before to after. The direct calls pay for the new extent check; the steps skip it.
Call Reductions ms lttbstep, kernels,n10001,079,886 → 1,099,226 (+1.8%) within noise m4step, kernels, width 10001,111,055 → 1,112,078 within noise lttbstep, Elixir path,refsn1000144,929 → 132,427 (-8.6%) 5.4 → 4.4 Macaw.Data.Reduce.lttb/3direct,n1000465,853 → 670,227 (+44%) 25.9 → 20.5 Macaw.Data.Reduce.m4/3direct, width 1000699,005 → 903,355 (+29%) 16.7 → 19.3 Macaw.Data.FFT.fft/1, 32,768 samplesabout 5.41M → 5.55M (+2.6%) within noise, about 110 Macaw.Data.FFT.spectrum/3, 32,768 samplesabout 6.21M → 6.39M (+3%) within noise, about 145 Macaw.Data.Table.source/150,018 → 53,143 (+6%) 0.63 → about 1.1
Changes
Macaw.Data.Source.Finish.ceiling/2refuses ahost:orpage:that is no positive integer (#195),{[key], {:type, :pos_integer}}, every fault at once. A term that isn't a number orders above every number, sohost: :nopeorhost: "10"applied no limit andceiling(nil, host: :nope)answered{:nope, false}.host: 10.5andhost: 0, which used to answer, are now refused.Macaw.Data.Source.describe/2,fetch/3andsubscribe/2answer{:error, :not_a_source}for a term that is no source (#191), where they raisedFunctionClauseErrororUndefinedFunctionError.describer/1answers it for an entry that is no{source, context}pair: such an entry either raised or, for one that wasn't a tuple, answered{:ok, entry}as if it were a schema.unsubscribe/2answers:okfor such a term, and for a module withsubscribe/2and nounsubscribe/2. A wrongDescriptionorRequest, or apidthat isn't one, stays a programming error (spec/00 §5).Macaw.Data.Reduce.lttb/3,m4/3,Macaw.Data.FFT.magnitudes/1andwindow/2answer{:ok, value}or{:error, reason}, a breaking change (#192). Each had a bare list. Their twinslttb!/3,m4!/3,magnitudes!/1andwindow!/2answer it as before. Each module words its reasons informat_error/1.Reduce.lttb/3andm4/3also refuse what used to raiseFunctionClauseError, as values:{:n, n},{:width, width},{:data, :list},{:data, :pair}and{:accessor, accessor}. No project in the family calls these directly.- The engines answer numbers past a float's range as
{:values, :overflow}(#192).Reduce.lttb/3holds thelttbstep's bounds andReduce.m4/3refuses anxat half the largest float or past it, as them4step does.FFT.fft/1refuses readings whose absolute parts sum to half the largest float or past it. Each can refuse a call that happened to work near those bounds.FFT.spectrum/3answers an improper list of values, or of options, as a fault. - spec/04 §4.4 states what
lttb's tolerance holds for (#193). A random property failed about once in 2,000 sparse-spike series atnof 20. The step kept more peaks thanMacaw.Data.Reduce.lttb/3, and the error metric charges a kept peak. No picked row changes: the bound adds1/nfor each peak kept beyond the exact algorithm's. Macaw.Data.Table.from!/2,Macaw.Data.Schema.from_map!/1,Macaw.Data.Schema.from_json!/2andMacaw.Data.Derived.from_map!/2, the twins spec/00 §5 says every reader has (#185). Each raisesMacaw.Data.Errorwith exactly its reader's faults. A test reads the spec's list and asserts every twin is exported.- A step's arithmetic past a float's range is
:overflow, neverArithmeticError(#181).mean,median,quantile,variance,deviation,slopeanddeltaanswer it assumdid,bin's edges atfield,window's bounds atfromandto, andspectrumatfield,everyandtime. What a float holds is the BEAM's answer, to the bit, except in one case below. A fault of:overflowreads "is beyond a float's range" where it read "sums beyond a float's range". lttbandm4answer{[i, "fields"], :overflow}where their arithmetic could pass a float's range (#181). Anxat or above half the largest float, a number at 1.0e301 under atimescale or past a float's range underlog, and forlttba line whose triangles or bucket means could pass it. A line near those bounds that happened to work is now refused. Inside them, the rows kept are the same.Macaw.Data.FFT.spectrum/3refuses readings whose absolute sum reaches 1.0e152 (#181),{:error, {:values, :overflow}}, and an interval whose frequencies are past a float's range,{:error, {:every, :overflow}}. Past that sum a term's square could pass the range, and one check over the readings replaces one at every butterfly. So a spectrum whose terms happened to stay small enough, and which worked, is now refused.- An improper list answers its position's fault at every reader (#184, #169). One guard judges a list, and a test makes each list of a valid input of each reader improper on every run.
Macaw.Data.Table.new/2has{[], {:type, :columns}}andMacaw.Data.Operations.validate/3{["nodes"], {:type, :nodes}}. - A step's
:termoption refuses what is no data (#187). A function, a PID, a reference, a port or an improper list infilter'svalueis{[i, "value"], {:type, :term}}fromvalidate/3andapply/4, whereapply/4raisedArgumentErrorfrom the step's digest.validate/3accepted them before, so a test that only validated such a node changes.
v0.12.1 (2026-11-04)
A typed parameter can default to no value, and a push holder, a stack and a schema's map form answer as faults what they raised (#172, #166, #178, #177, #182). default: nil on a typed Macaw.Data.Parameter means optional with no value. Macaw.Data.Source.Push.drain/2 answers every delivery Macaw.Data.Delivery.new/2 accepts, Macaw.Data.Source.Push.trim/2 keeps a box window, Macaw.Data.Stack.generate/2 answers a value it can't stack or a sum past a float's range, and Macaw.Data.Schema.from_map/1 answers a struct at its path. Two random CI failures go with them. This is a patch, and some calls behave differently, including a few that worked: read the guide below.
Upgrading from 0.12.0
Macaw.Data.Source.Push.new/3refuses a badcache:,limit:,lateness:orchunk_span:at the call, including values that were never read before. A badlimit:on a source with a window field, a badchunk_span:on a source with no window field and a badlateness:on a source with nocomplete_untilall worked, and now answer{:error, faults}. The owner accepted this in a patch because only nonsense option values are affected. Pass a value of the right type or drop the option.Macaw.Data.Stack.series/2raisesMacaw.Data.Errorinstead ofArithmeticError, andMacaw.Data.Stack.generate!/2's message changed. A test that asserts either one changes with it.Macaw.Data.Source.Description.chunk_span/2answersnilfor extreme resolutions, so those chunks align by row count.- A float window value past 2^53 spans is its own slot.
- A struct that enumerates as
{key, value}pairs is no description inMacaw.Data.Schema.from_map/1. AMapSetof them read as one, and is now a fault. Macaw.Data.Stack.generate/2costs more onwiggle, for its checked arithmetic. 50k rows x 3 keys went from 5,404,131 to 9,180,325 reductions; the time in ms is within noise.- New API:
default: nilon a typed parameter,Macaw.Data.Stack.stacked/2andstacked!/2.
Changes
Added:
default: nilon a typedMacaw.Data.Parameteris optional with no value (#172, #175, spec/01 §2.2). A number has nothing to stand for "none", and a sentinel collides with a real value.Macaw.Data.Parameter.check/1accepts it on any type, besidevaluesthat don't list it, andMacaw.Data.Parameter.validate/2answersnilwhen it is absent and takesnilgiven as the same thing.Macaw.Data.Parameter.conforms?/2also takes a parameter:nilconforms to one whose default isnil, and to nothing else but:term.conforms?(type, nil)is unchanged. A required parameter and one with any other default still refusenil, and listingnilin thevaluesof a type it is no value of is still a fault.bindableis unchanged. No stored document's digest changes: nothing digested holds a parameter, and no described source could declare this before.Macaw.Data.Source.Conformance's:deliveriescheck asks each such parameter givennil, on the first request, so a source that declares one and can't readnilis a finding.Fixed:
Macaw.Data.Source.Push.drain/2answers every deliveryMacaw.Data.Delivery.new/2accepts, where it raised (#166, #176, spec/01 §4). A snapshot or an increment of tuple rows withoutnames:, or of tuple rows of another width than their names, is{:table, faults}inerrors,Macaw.Data.Delivery.named_rows/1's faults, as rows that don't pack already are. A refused snapshot leaves what is held and the position as they were. A refused increment consumes its number and applies nothing, neither its retractions and replacements nor itscomplete_untilandstale. A property over generated deliveries found the same class elsewhere on the path, each now answered:- An increment under a schema whose fields the held rows don't pack under raised
MatchError. It is{:table, faults}, and the holding keeps its schema. Macaw.Data.Table.new/2with a source'sorigin:raisedArithmeticErroron a number in a window column whose span is aDuration, or on a time under a number span. It is the column's own type fault.- A retraction whose window value its span can't slot raised. It is
{:retract, :not_held, row}. - A row map key that spells nothing (a tuple, a reference) raised from
Macaw.Data.Table.apply/2, and fromMacaw.Data.Table.get/2as a name. Such a key is no column. A list of structs as an increment's rows raised there too; they read by their fields, as a snapshot's do. - A
Dateascomplete_untilunder alatenessof hours or less raisedArgumentError. The bound is that midnight less the lateness. - A number resolution, claim, lateness or window value past what a float's arithmetic holds raised
ArithmeticError.Macaw.Data.Source.Description.chunk_span/2derives no span from a resolution under a nanosecond a row (over a time field) or under1.0e-280of the field's units (over a number one), or over1.0e280of either, so those chunks are aligned by row count; a value past a float's precision over its span is its own slot; a span reaching back past the earliestDateTimestarts there.
Nothing that worked changes: every resolution, value and row a push could hold before reads and lays out the same.
- An increment under a schema whose fields the held rows don't pack under raised
Fixed:
Macaw.Data.Stack.generate/2answers a value it can't stack as a fault, where it raisedArithmeticError(#178, #179, spec/04 §4.8). This also removes a random CI failure: the reader property hit this only when a generated row used a stack key, so its verdict depended on the seed. The property now generates rows under the stack's own keys.- A value that is no number, from the default value function or a caller's, is
{[key, j], {:reads, :number}}, rowjunderkey, every one at once.generate!/2raises those faults. The default function still readsnilas0. A caller's function answeringnilanswers no number. - A sum past a float's range raised too, and is
:overflow:{[key, j], :overflow}for a point,{[:order], :overflow}for a series' sum an order reads. Thestackstep raised the same way fromMacaw.Data.Operations.apply/4and answers{[i, "fields"], :overflow}. - Added:
Macaw.Data.Stack.stacked/2andstacked!/2, the value form over a list of any data.series/2keeps its bare list and is nowstacked!/2, so it raisesMacaw.Data.Errorwhere it raisedArithmeticError.
Nothing that worked changes. One stack does no arithmetic, so it takes any value and answers it as it is, as it did: one key under the
:noneoffset and an order of:none,:reverseor{:keys, _}. Every sum a float holds is the BEAM's own answer, to the bit, and integers of any size stay exact.generate!/2's message now starts "not a source it stacks".- A value that is no number, from the default value function or a caller's, is
Fixed:
Macaw.Data.Source.Push.trim/2answers a holding whose served window is a box, where it raisedCaseClauseError(#177, #180, spec/01 §4). The chunks wholly beforefromdrop and the box stays as it is, since it has no start to narrow.Changed:
Macaw.Data.Source.Push.new/3refuses an option value it can't use, at the call, andnew!/3raises those faults (#177, #180, spec/01 §4). A call with a bad option value that only failed later now answers{:error, faults}at once.limit:must be an integer,{[:limit], {:type, :integer}}. A float,nil, an atom or a string raised at a drain under a schema with no window field:ArithmeticError, or for a floatFunctionClauseErroronce the rows passed it.lateness:must benil, a number or aDuration,{[:lateness], {:type, :lateness}}. Anything else recorded{:lateness, :incompatible, claim}on every increment after a claim. ADurationholding no integers raised.chunk_span:must benil, a positive number or a positiveDurationof fixed length,{[:chunk_span], {:type, :chunk_span}}. Anything else was{:table, faults}on every delivery under a window field. ADurationholding no integers raised, also fromMacaw.Data.Source.Description.new/1andMacaw.Data.Table.new/2'sorigin:, which now answer their fault. An integer span past what a float holds raisedArithmeticErrorover a float window value, here and fromorigin:, and now lays chunks out as it does over an integer.cache:must be aMacaw.Data.Schema.Cache,{[:cache], {:type, :cache}}. Anything else raised at a drain.
Some such calls never reached the failure, and are refused all the same.
new/3knows no schema and no rows, so it can't tell: a badlimit:under a source with a window field, wherelimitis not read; a floatlimit:no holding reached; a badchunk_span:under a source with no window field; a badlateness:under a source that states nocomplete_until. Pass a value of the type above, or drop the option. Every other working value stays: alimit:of0or below still holds nothing, a negativelateness:still tightens the bound, aDurationof any sign or length is still alateness:, and a value of the wrong kind for the window field (a number span over a:timefield) is still the delivery's fault.Fixed:
Macaw.Data.Schema.from_map/1answers a struct as a description, or as a value entry, as a fault at its path, where it raisedProtocol.UndefinedErrororFunctionClauseError(#182, #183, spec/01 §1.11). It is{[name], {:type, :description}}, or{[name, :values], {:type, :values}}, andMacaw.Data.Derived.from_map/2answers it under[:schema]. This also removes a random CI failure: the reader property hit this only when a generated pair was a binary name then a struct, so its verdict depended on the seed. The property now generates both on every run. One call that answered changes: a struct that enumerates as{key, value}pairs, aMapSetof them, read as a description or an entry by those pairs, and is now the same fault.
v0.12.0 (2026-11-03)
A source says its answer is stale, an inverted window is refused, a push holder and a time column read through public seams, and errors are values to the end (#148, #122, #123, #149, #139, #130, #161, #162). A delivery's served.stale says a source answered from a store it could not refresh, carried through finishing, pooling, a push and a derived source's output, so a surface can draw stale rows as stale. Macaw.Data.Window.check/2 and Macaw.Data.Source.fetch/3 refuse a window or region that runs backward. Macaw.Data.Source.Push.trim/2, take_errors/1 and pending?/1, Macaw.Data.Derived.Columns.time_kind/2 and Macaw.Data.Table.Chunk.time_kind/4 replace reads of a struct's fields and of internal storage forms. Every ! twin raises Macaw.Data.Error in words, carrying its value twin's reason, and the readers of a caller's rows (Macaw.Data.Table.rows/1, Macaw.Data.Delivery.rows/1, Macaw.Data.Table.column_types/1, time_columns/1, Macaw.Data.Stack.generate/2) answer {:ok, _} or {:error, faults}, with properties over malformed input proving no reader of untrusted input raises. The family's CI scripts are vendored and a drifted copy fails CI and the release. This release is breaking for an adopter that builds a served map by hand, matches a derived function's context exactly, asks a backward window, asserts ArgumentError from a twin, reads rows/1, column_types/1, time_columns/1 or generate/2 as a bare value, or passes Macaw.Data.Table.from/2 an option it ignored. The guide below lists what to change, and the entries after it give the detail.
Upgrading from 0.11.x
- A delivery's
servedmap has astalekey (#148, #150).Macaw.Data.Delivery.new/2fills it,falseunless the source givesserved: [stale: true], so a source built on it changes nothing. A served map built by hand, aroundDelivery.new/2, or a match that lists its keys exactly, addsstale. Astalethat is not a boolean is the fault{[:served, :stale], {:type, :boolean}}, andMacaw.Data.Source.Conformance's:deliveriescheck finds it malformed. A reader that draws a delivery readsserved.staleto draw a stale answer as stale. - A derived source's output is stale when any input's delivery is, and its function's context has
"stale"(#148, #151). An idle input counts, so a derived output over a source serving its last good answer stays stale until that input delivers a fresh one. A change ofstalealone marks the readers, as a changed answer. A function of two arguments that compared an input's context map whole, or read its keys as a list, sees the new"stale"key beside"capped"; a map pattern of the keys it reads is unaffected. - An inverted request window is refused (#122, #152).
Macaw.Data.Window.check/2refuses a window whosefromis after itsto,{[:window], {:window, :inverted}}, as it refused an inverted region range, comparing two times of any kinds on the instant scale.Macaw.Data.Source.Finish.finish/4and a derived source's request ({[:from, i, :window, :window], {:window, :inverted}}) refuse what they finished to no rows, andMacaw.Data.Source.fetch/3refuses an inverted window or region range before calling the source,{:error, {:window, faults}}. A caller that asked a backward window and relied on an empty answer orders its ends or matches the fault; an equal pair and an open end still pass. - A
!twin raisesMacaw.Data.Error, notArgumentError(#130, #157). A test that assertsassert_raise ArgumentErroron a twin (Macaw.Data.Table.new!/2,Macaw.Data.Delivery.new!/2,Macaw.Data.Source.Request.new!/1and the others) assertsMacaw.Data.Error. The exception'sreasonholds the value twin's error term, so a test can match that term rather than a message. Macaw.Data.Table.rows/1andMacaw.Data.Delivery.rows/1answer{:ok, rows}(#130, #158). A caller that reads rows it knows are valid (a table, a deliveryMacaw.Data.Delivery.new/2built) calls the new twinrows!/1:Macaw.Data.Table.rows!(table)is whatMacaw.Data.Table.rows(table)was. A caller that reads a source it was given matches{:ok, rows}and{:error, faults},Macaw.Data.Table.source/1's faults (a delivery's under[:rows]), where it used to catch nothing while the call raisedArgumentError.Macaw.Data.Table.from/2refuses an option it does not take (#159). It takestypes:alone, and any other key is{:error, [{[key], :unknown_key}]}where it was ignored. Drop the option.types:that is no map is{[:types], {:type, :map}}where it raisedBadMapError, here and throughMacaw.Data.Delivery.table/2.Macaw.Data.Table.column_types/1,Macaw.Data.Table.time_columns/1andMacaw.Data.Stack.generate/2answer{:ok, value}(#161, #163). A caller whose source is known valid (a table, rows it built) calls the twin,column_types!/1,time_columns!/1orgenerate!/2, for what the function returned before; a pipelinestack |> Stack.generate(data)becomesstack |> Stack.generate!(data). A caller reading a source it was given matches{:error, faults},Macaw.Data.Table.source/1's faults.Stack.generate/2of a tensor or tuple rows needsMacaw.Data.Table.named/2first, or a value function (Stack.value/2) that reads them.- Read a push and a time column through their public functions (#123, #149, #153, #154, #155). Nothing breaks, but a holder that read or rewrote
%Macaw.Data.Source.Push{}fields callsMacaw.Data.Source.Push.take_errors/1(which also empties them),pending?/1andtrim/2instead, and a reader that matched a chunk's internal storage forms for a time column's kind callsMacaw.Data.Derived.Columns.time_kind/2orMacaw.Data.Table.Chunk.time_kind/4. The CI changes (#160, #164, #165) change nothing a caller sees.
Changes
Breaking:
Macaw.Data.Table.rows/1andMacaw.Data.Delivery.rows/1answer{:ok, rows}or{:error, faults}, each with arows!/1twin (#130, #158, spec/01 §1.1, §3, spec/00 §5). A sourceMacaw.Data.Table.source/1refuses is answered with its faults where it raised: a column map whose columns differ in length ({[name], {:length, n}}), a column that is no list ({[name], {:type, :column}}), a tensor of rank other than 1 or 2 ({[], {:rank, rank}}) and a struct with noTable.Reader({[], {:type, :source}}, which used to raiseProtocol.UndefinedError).Delivery.rows/1puts them under[:rows].Macaw.Data.Delivery.empty?/1andnamed_rows/1read rows the same way. The raise guard's two invariant entries forTable.Edgeare gone.Macaw.Data.Table.time_columns/1,Macaw.Data.Table.column_types/1andMacaw.Data.Stack.generate/2answer such a source too (#161, below).Breaking:
Macaw.Data.Table.column_types/1,Macaw.Data.Table.time_columns/1andMacaw.Data.Stack.generate/2answer{:ok, value}orMacaw.Data.Table.source/1's{:error, faults}, each with a!twin,column_types!/1,time_columns!/1andgenerate!/2(#161, #163, spec/01 §1.1, spec/04 §4.8, spec/00 §5). They read a caller's source throughrows!/1and raised on onesource/1refuses, a column map of unequal columns, a tensor of rank 3, any other term, from functions without!. A table answers{:ok, _}as well, its declared types and time columns.Stack.generate/2under the default value function, which reads a datum's key by name, answers rows with no names (a tensor's, tuple rows) as{[:names], :required}, where it raisedBadMapError; a value function of the caller's still reads them.Macaw.Data.Stack.series/2is unchanged. A test now lists every call inlib/of a twin that reads rows (Macaw.Data.Table.rows!/1,Macaw.Data.Delivery.rows!/1,Macaw.Data.Delivery.named_rows!/1) with why its argument can't be refused, since the raise guard sees only literal raise sites. Known:Macaw.Data.Source.Push.drain/2raises on a snapshot or increment of tuple rows without names (#166).Fixed: no reader of untrusted input raises on input of the right type but the wrong shape (#130, #159, spec/00 §5). Properties over generated malformed input, run over 40 seeds, check
Macaw.Data.Table.new/2,Macaw.Data.Table.from/2,Macaw.Data.Schema.new/1,Macaw.Data.Schema.from_map/1,Macaw.Data.Schema.from_json/2,Macaw.Data.Source.Description.new/1,Macaw.Data.Source.Request.new/1,Macaw.Data.Derived.new/1,Macaw.Data.Derived.from_map/2,Macaw.Data.Source.Push.new/3andMacaw.Data.Source.Finish.ceiling/2. They found these, each now a fault:Macaw.Data.Table.new/2given a list holding something other than{name, type, values}columns, such as row maps, raisedFunctionClauseError. It is{[i], {:type, :column_spec}}at the element's index, and a column's length is the first column's.Macaw.Data.Source.Request.new/1andMacaw.Data.Source.Description.new/1given a list of anything but pairs, or a struct, raisedArgumentErrorfromMap.new/1. It is{[], {:type, :keyword}}. A request'sregion:given as a struct is{[:region], {:type, :region}}.Macaw.Data.Table.from/2andMacaw.Data.Delivery.table/2giventypes:that is no map raisedBadMapError. It is{[:types], {:type, :map}}.- The twins' formatter,
Macaw.Data.Fault, wrote a path holding a reference or a tuple throughString.Chars. It now inspects such a step.
Breaking:
Macaw.Data.Table.from/2refuses an option it does not take,{[key], :unknown_key}, where it ignored it (#159). It takestypes:alone.Known: a map form holding a struct still raises inside macaw_layers'
Macaw.Layers.Form.from_map/3, reached throughMacaw.Data.Derived.from_map/2. The kernel will refuse it (macaw_layers#37).Breaking: every
!twin raisesMacaw.Data.Errorin words, with its value twin's{:error, reason}term on the exception'sreason(#130, #157, spec/00 §5). Thirteen twins and findings wrote their faults withinspect/1:Macaw.Data.Table.new!/2,Macaw.Data.Source.Request.new!/1,Macaw.Data.Source.Description.new!/1,Macaw.Data.Derived.Engine.source!/3,Macaw.Data.Source.Push.new!/3,Macaw.Data.Source.Push.trim!/2,Macaw.Data.Source.Finish.finish!/4,Macaw.Data.Source.Finish.reduce!/5,Macaw.Data.Source.Finish.ceiling!/2,Macaw.Data.Source.Order.sort!/3,Macaw.Data.Delivery.table!/2,Macaw.Data.Derived.Functions.registry!/1, and the conformance kit's:description_invalidand:probefindings. They now word them throughMacaw.Data.Fault.sentence/2, the library's one fault-to-sentence formatter:"not a request: budget: must be a positive integer; now: is required". The other twins keep their module'sformat_error/1words.Macaw.Data.Quantity.label!/1words its reason too. AnArgumentErrorhas no field to carry the term, which is why the exception changed. A stored function's!call is still the runner's{:error, :raised, module}, now withMacaw.Data.Erroras the module.Clarified: a
FunctionClauseErrorfrom a guard on an argument's type (a map where a list is taken) is a programming error and outside the errors-are-values rule (#130, spec/00 §5). An argument of the right type but the wrong shape is input, and answers a fault.Changed: the raise guard,
ci/raises.exs, is the Macaw family's, vendored byte for byte from claude-tools'macaw_ci/raises.exs(#130, #160, spec/00 §5). It also findsexit, theKernel.and:erlang.forms andMix.raise, names every raising function inci/raises_allowlist.exsas%{at: {module, name, arity}, class, reason}, and verifies each class. So each!twin raises its ownMacaw.Data.Error, where five modules' twins shared a private helper. The derived engine's defsource refuses bad code from its macros. The native kernels' build failure is the module's body. The kernels' interface probe sends its answer back instead of exiting with it. A stored function's!call toMacaw.Data.Opscompiles to the internal__run__!/5, on the allowlist's excluded list beside__run__/5. Nothing a caller sees changes.Changed: the family's other CI scripts are vendored byte for byte from claude-tools'
macaw_ci/, so the family's drift check finds none missing or differing (#162, #164, spec/00 §3). CI's newauditjob runsci/advisories.exsovermix hex.audit: a locked package with a security advisory outsideci/advisories_allowlist.exs(empty), or a retired one, fails the pipeline.mix hex.publishfirst runsci/release_guard.exs, which refuses a dirty tree, a HEAD that is not the release tag and a tag not on origin.ci/spec_links.exsis the reference copy: it also checks links into another repository's directories and files beyond its spec, repository homes and GitLab issue links, counts each kind, and refuses a link that leaves the repository; CI runs its self-test first. Nothing a caller sees changes.Added: a vendored family script that drifts fails, in CI and before a release (#162, #165, spec/00 §3). The family's drift check ran by hand from claude-tools, which CI can't reach, and nothing in this repository ran it, so the advisory gate and the release guard went missing and the link check fell behind.
ci/vendored_manifest.exsholds each vendored copy's SHA-256, and CI's newvendoredjob (ci/vendored.exs) fails on a copy edited here or missing.mix hex.publishfirst runsci/vendored.exs --referenceagainst the family's reference ($MACAW_CI_REFERENCE, else~/Projects/claude-tools/macaw_ci), which fails on a copy the reference has moved past, a family script not vendored, one the family dropped, and a reference that isn't there.elixir ci/vendored.exs --vendorvendors the reference and rewrites the manifest.Added:
served.stale, by which a source says its answer is stale (#148, #150, spec/01 §3). A source that answers from a store it could not refresh (its upstream failed, and it serves its last good answer) givesMacaw.Data.Delivery.new/2served: [stale: true], beside anotefor a person if it likes; it isfalseby default, and any other value is the fault{[:served, :stale], {:type, :boolean}}, worded "not a boolean". It is a fact of one answer, ascappedis, not a capability insupports, so a reader can draw the answer as stale rather than live (macaw's fetch, a follow-up there).Macaw.Data.Source.Finishcarries it through every step unchanged, and each reader of a pooled fetch keeps it throughMacaw.Data.Source.Finish.reduce/5(spec/01 §2.4, §6.1).Macaw.Data.Source.Push's holding takes a snapshot'sstale, and each applied increment's replaces it (spec/01 §4).Macaw.Data.Source.Conformance's:deliveriescheck finds astalethat is not a boolean malformed, as it doescapped; adown:source answering a stale delivery is:answered_down, as any answer is (spec/01 §9). A served map built by hand, aroundDelivery.new/2, now needs thestalekey.Added: a derived source's output is
stalewhen any input's delivery is (#148, #151, spec/02 §8.7), its rows being drawn from that input's, and a function of two arguments reads each input's"stale"in its context beside"capped"(Macaw.Data.Derived.Context, spec/02 §2.3). A derived input passes its output's on down a chain. A change ofstalealone is a changed answer and marks the readers, as a watermark-only change does. Unlike the watermark, an idle input still counts: a source that keeps serving its last good answer delivers nothing new, so it goes idle, and leaving it out would draw its stale rows as live. A stored function that matched its context's input map exactly sees the new key.Changed: an inverted request window is refused, as an inverted region range is (#122, #152, spec/01 §2.3).
Macaw.Data.Window.check/2compares a window's ends once they are of the window field's type, two times of any kinds on the instant scale, and afromafter itstois{[:window], {:window, :inverted}}; an equal pair (an instant) and an open end pass. SoMacaw.Data.Source.Finish.finish/4and a derived source's request ({[:from, i, :window, :window], {:window, :inverted}}) refuse what they used to finish to no rows.Macaw.Data.Source.fetch/3refuses an inverted window or region range before calling the source,{:error, {:window, faults}}.Macaw.Data.Source.Conformance's:errorscheck asks an inverted number window and an inverted pair of instants of two kinds. A caller that asked a backward window and relied on an empty answer now matches the fault.Added:
Macaw.Data.Source.Push.take_errors/1andMacaw.Data.Source.Push.pending?/1, so a holder reads its subscription without the struct's fields (#123, #153, spec/01 §4).take_errors/1answers{errors, push}, the errors recorded since the last take in the order recorded, and a push holding none, so a long-lived subscription's errors no longer grow without bound.pending?/1is whetherMacaw.Data.Source.Push.drain/2has anything buffered to apply: a snapshot or an increment received since the last drain, or increments waiting on a gap.Added:
Macaw.Data.Source.Push.trim/2andtrim!/2, so a holder whose window slides drops what is before it without the struct (#123, #154, spec/01 §4, §4.2). It answers{:ok, push}without the held chunks wholly beforefromin the window field's order, throughMacaw.Data.Table.trim/2, so nothing is copied and every chunk kept is the same chunk by id and version; a chunk straddlingfromis kept whole. The served window starts atfrom, or the held start where later, its upper end andcomplete_untilunchanged. Afromnot of the window field's type is{:error, [{[:from], {:type, type}}]}, and it is a no-op with nothing held, no window field, a:pointone or anilfrom. The push keeps the trimmed start: a later retraction of an unheld row before it, or a replace range wholly before it, is counted inignoredas:trimmedinstead of being an error; a late insertion before it is still placed; a snapshot clears it.Added: a time column's kind is public, read from its storage form with nothing decoded (#149, #155, spec/01 §1.1.5, spec/02 §2.2).
Macaw.Data.Derived.Columns.time_kind/2answers{:ok, kind}:DateTime(UTC or zoned),NaiveDateTimeorDatewhen every chunk holding a value holds that kind, else:mixed; a column with no value present isDateTime. Its faults are{:unknown_column, name},{:wrong_type, name, [:time], actual}and{:type, :table}, andtime_kind!/2raises their words.Macaw.Data.Table.Chunk.time_kind/4answers a chunk run's kind, andMacaw.Data.Table.Chunk.instants/4, which visualize already read, is documented and public. They replace a match on the internalChunk.storage/4forms.time_kind/2reads a 50,000-row, ten-chunk column in 948 reductions (instants/2takes 113,949). It is not on the sandbox allowlist: excluded as:atoms, since it answers a module.Clarified: where
Macaw.Data.Ops' faults are placed (#139, #156, spec/04 §6). Nothing changes. A transform call runs a one-step pipeline, so its faults are at[0, key]and its name's, or anopoption's, at[0, "op"]: 0.10.0's rule for a fault naming an op is the step's, the transform's case. An aggregate call is no step, so its faults name none:[]for an unknown name ({[], {:unknown, :aggregate, name}}),[key]for an option and["values"]for a reading. From a stored function the paths are the same with the reasons spelled, and an unknown name is refused at its line.
v0.11.0 (2026-11-02)
Every kind of time on one instant scale, and a table gathered and reduced on its columns across chunks (#124, #127, #141, #128, #132, #133, #140). A window bound, a window position, latest, the budget's reducers and min/max/extent read a Date, a NaiveDateTime and a DateTime on one scale, Macaw.Data.Source.Order.instant/1's integer Unix microseconds, so mixing kinds of time answers a value where it raised or was refused. Rows in another order across chunks are gathered by their slots, a one-field order sorts its column's keys, the m4/lttb ladder reads a time x as integers, and a budget is reduced over the table's own columns: on visualize#998's 50,000-row loop the finish of a frame reduced by lttb costs about a quarter of the reductions it did before #146 on the Elixir path and a tenth with the kernels, keeping the same rows. This release is breaking for an adopter that reads Macaw.Data.Window.position/1 of a date or naive time, expects a budget over a Date or NaiveDateTime window field to keep 0.10.0's rows, matches the {:window, :mixed} refusal, or relies on a mixed-kind window or min/max/extent failing. The guide below lists what to change, and the entries after it give the detail.
Upgrading from 0.10.x
Macaw.Data.Window.position/1answers integer Unix microseconds for every kind of time (#127, #135). ADatewas Gregorian days and aNaiveDateTimefloat Gregorian seconds; aDateTimewas and is its Unix microseconds, and a number is itself. A caller that read a date's position as days or a naive time's as seconds divides by 86,400,000,000 or 1,000,000 (and shifts the epoch), or callsMacaw.Data.Source.Order.instant/1and converts.- A
DateorNaiveDateTimeseries under a budget may keep different rows (#127)."lttb"and"m4"preselect on absolute power-of-two buckets, so the rescaled x moves the bucket edges. A series now keeps the same rows whichever kind of time it is given as, the rows a UTC series keeps, and the rows thelttbandm4transforms keep over the time column. An expectation or golden written from 0.10.0's reduction of a date or naive series changes; aDateTimeor number series is unchanged.:latestnow keeps the later of two naive times of one identity however close. - A request window compares times of different kinds by instant, and
{:window, :mixed}is gone (#124, #134). A bound of another kind of time than the window field's values trims by instant whereMacaw.Data.Window.compare/2raisedFunctionClauseError, throughMacaw.Data.Source.Finish.finish/4, theMacaw.Data.Windowtrims andMacaw.Data.Derived.window/3.Macaw.Data.Window.check/2no longer refuses a region range whose ends are different kinds of time, andMacaw.Data.Source.Conformanceno longer calls such a served window malformed: drop any match on{:window, :mixed}, since only an inverted range is refused. A value with no reading on a bound's scale (a number or a string in a window of instants) lies in no window, asnildoes.Macaw.Data.Window.comparable?/2is new. min,maxandextentover a time column of mixed kinds answer a value (#141, #143).Macaw.Data.Operations.apply/4raisedArgumentErrorandMacaw.Data.Operations.aggregate/4answered the fault{["values"], {:reads, :comparable}}. Both now compare by instant and answer the reading itself, of its own kind (of two at one instant, the first). Code that matched that fault for mixed kinds of time matches the value; a number among times, or a time among numbers, is still that fault.
Changes
Changed: a table's budget is reduced over its own columns (#132, #146, spec/01 §2.4).
Macaw.Data.Source.Finish.finish/4andMacaw.Data.Source.Finish.reduce/5packed each series and number field into a fresh table of positions, values and indexes before the reducer ran. The reducer now reads each series' columns where they lie: x the window field's column, whose time buffer holds the positions, y the field's, beside the rows' indexes. The kept rows are the same. A window field whose column is neither:timenor:numberis still read value by value. The ladder's extent over an unsorted time x reads its instants' least and greatest in one pass (Macaw.Data.Operations.Transforms.Ladder). Reductions before and after, on visualize#998's loop (Macaw.Data.Test.LiveLoop: 50,000 UTC rows, two number fields, budget 4,000), Elixir path (kernels):- the frame's finish, reduced by
lttb: 8,788,073 to 2,272,032 (8,218,433 to 754,860), ~349 to ~153 ms (~298 to ~28 ms); - the frame's finish, reduced by
m4: 8,255,767 to 1,738,752 (8,116,467 to 674,868); - the whole frame, finish and
m4andlttbthrough the cache, underlttb: 9,212,945 to 2,696,904 (8,356,191 to 891,523); reduce/5of a 50,000-row table not known sorted: 8,785,631 to 2,853,474 (8,212,293 to 1,339,034);- 50 series of 1,000 rows: 5,484,998 to 4,573,772.
- the frame's finish, reduced by
Fixed: a gathered chunk is named at its view's first slot, so two gathers of one chunk share an id and a version only where they read the same rows (#132, #145, spec/01 §1.1.3).
Macaw.Data.Table.Chunk.gather/5andMacaw.Data.Table.Chunk.reshape/7named their output{input id, step key}, versioned by{[input version], step key}, while their callers count the kept rows from the view's first row.Macaw.Data.Source.Finish.finish/4's budget, them4andlttboutputs andlatestgather so. Two views of one chunk cut at different first rows that kept the same counted rows then got one id and one version over different rows. A cache keyed by them, asMacaw.Data.Operations.Cacheis, could reuse another view's part: a window's edge chunk is cut a little later every frame. A view from slot 0 keeps its name{input id, step key}; any other's step key is now{step key, {:from, first}}.Pinned: a finished table keeps the held table's chunks (#132, spec/01 §2.4).
Macaw.Data.Source.Finish.finish/4's window keeps every chunk inside it by id and version and cuts only the two edge chunks to views, and a delivery no step acts on is the held table. This was already so in 0.10.0: on visualize#998's loop throughMacaw.Data.Source.Push,Macaw.Data.Source.Finishand anMacaw.Data.Operations.Cacheofm4andlttb(Macaw.Data.Test.LiveLoop), an unreduced frame reuses 10 of its 14 parts (8 when a new chunk starts). Reduced bym4under a budget of 4,000 a frame reuses 8 to 10; reduced bylttb, whose picks move with the window, none. The budget itself costs about 8.3 to 8.9 million reductions a frame on 50,000 rows, the cost visualize measured on its board (#146).Changed: the
m4andlttbladder's Elixir walk (Macaw.Data.Operations.Transforms.Ladder.partial/4without the native kernels) reads a packed time x column's instants from its buffer, integer microseconds onMacaw.Data.Source.Order.instant/1's scale, instead of building aDateTimeper row and callingDateTime.to_unix/2(#133, #144, spec/04 §4.5.2). Partials are equal term for term to before and to the kernel's. The unsorted x extent reads the same instants. On visualize#998's loop shape (aMacaw.Data.Source.Pushholding of 50,000 UTC rows, one 1,000-row increment a frame,Macaw.Data.Source.Finish.finish/4's window, thenm4andlttbthroughMacaw.Data.Operations.Cache), a frame's reductions on the Elixir path before and after:- cached (10 of 14 parts reused): 1,351,771 to 666,731, ~64 to ~40 ms;
- full (no cache): 5,228,160 to 1,427,844, ~172 to ~47 ms;
- one sorted 8,192-row UTC chunk's partial at level 0: 411,913 to 100,008.
With the kernels loaded the frame is unchanged (~8 ms).
Changed: rows of a table in another order across its chunks, a full
:orderamong them, are gathered by their slots (#128, #137, #138, spec/01 §2.4). Each column's pieces are joined into one storage where they share a form a join takes (int64, float64, date, UTC and naive buffers, text, binary and dictionary columns), its validity and precision with it, and gathered in the order. Any other column is read and packed again, that column alone. The output is chunked atMacaw.Data.Table.chunk_rows/0, and every result isMacaw.Data.Table.equal?/2to before. On 50k rows, reductions before and after on the Elixir path (with the kernels in brackets):- a random reorder of an int64, a float64 and a UTC column: 4,088,217 to 1,228,251 (448,160), ~154 to ~10 ms;
- finishing by a full order by cpu: 5,316,721 to 2,856,710 (2,076,736), ~210 to ~46 ms. Its sort is 1,471,715 of that;
- a random reorder of a text and a category column: 2,064,177 to 1,156,091 (896,060), ~45 to ~43 ms.
Text and binary pieces join their data, each piece's offsets moved past the data before it, and dictionary pieces join their codes, moved to one dictionary of every piece's strings where the dictionaries differ (#138). Gathering a text column's single slots reads both offsets in one match, which halves a text reorder's work.
Changed: finishing a table by a one-field
:ordersorts the column's keys, so a full order of a 50k-row table costs less than one transpose (1,705,513 reductions; #128, #140, #142, spec/01 §2.4).Macaw.Data.Source.Finishpairs each row index with its key (a time buffer's instants as they lie, any other column's values keyed by the field's type) and sorts them stably asMacaw.Data.Source.Order.sort_by/2sorts one key, instead of reading each index through closures (Macaw.Data.Source.Order.sort_with/4, which an order of several fields still uses). The order is exactlyMacaw.Data.Source.Order.sort/3's. Gathering a buffer's slots without the kernels copies a slot a step, a run at a time, instead of listing the runs first. On 50k rows, reductions before and after on the Elixir path (with the kernels in brackets):- finishing by a full order by cpu: 2,856,710 to 1,500,713 (2,076,736 to 1,356,709); its sort 1,471,715 to 754,291;
- finishing by a full order by time, descending: 3,908,922 to 1,228,659 (3,752,829 to 1,084,694);
- an order's head under a budget of 1,000: 1,559,266 to 816,397 (1,528,425 to 810,388);
- a random reorder of an int64, a float64 and a UTC column: 1,228,254 to 588,159 (448,160, unchanged);
- a random reorder of a text and a category column: 1,156,091 to 940,059 (896,060, unchanged).
Wall time moved within the noise of a loaded machine (a full order by cpu ~80 to ~110 ms either way; by time ~130 to ~45 ms).
Breaking:
Macaw.Data.Window.position/1places every kind of time onMacaw.Data.Source.Order.instant/1's one scale, integer Unix microseconds (#127, #135, spec/01 §2.4). ANaiveDateTimewas float Gregorian seconds, which can't hold a microsecond at today's magnitude, and aDatewas Gregorian days. A number is still itself and aDateTimestill its Unix microseconds. Upgrade: a caller that read a naive position as seconds or a date's as days divides by 1,000,000 or 86,400,000,000, or callsMacaw.Data.Source.Order.instant/1and converts.Fixed:
:latestinMacaw.Data.Source.Finish.finish/4keeps the later of two naive times of one identity however close, where naive times a few microseconds apart shared a position and the later in order was kept (#127). A column mixing kinds of time is placed on one axis, wherelatestcompared microseconds against days.Changed (behavior): under a budget, a
DateorNaiveDateTimewindow field's series may reduce to different rows than in 0.10.0 (#127)."lttb"and"m4"preselect on absolute power-of-two buckets, so rescaling x moves the bucket edges. A series now keeps the same rows whichever kind of time it is given as, and the rows thelttbandm4transforms keep over the time column on their time scale. ADateTimeseries is unchanged. Latest and the budget read a naive or a date column's buffer as they read a UTC one. On a 50k-row table, reductions before and after:- latest over a naive column: 2,009,465 to 609,472 (~57 to ~30 ms);
- latest over a date column: 1,567,973 to 609,472 (~55 to ~31 ms);
- a budget of 1,000 over a naive column: 7,085,026 to 5,685,114;
- a budget of 1,000 over a date column: 6,643,478 to 5,685,069;
- a UTC column's latest (609,472) and budget (5,685,114) unchanged.
Fixed: a request window whose bound is another kind of time than the window field's values no longer raises
FunctionClauseErrorfromMacaw.Data.Window.compare/2(#124, #134, spec/01 §2.4). Times compare on one scale whatever their kinds,Macaw.Data.Source.Order.instant/1's, as sort andfiltercompare them: aNaiveDateTimeorDatebound trimsDateTimerows, and a list column of mixed kinds, by instant.Macaw.Data.Source.Finish.finish/4,Macaw.Data.Window.trim/3,Macaw.Data.Window.trim_region/2,Macaw.Data.Window.newer/3,Macaw.Data.Window.before/4andMacaw.Data.Derived.window/3answer a value for every requestMacaw.Data.Window.check/2passes. A value with no reading on a closed bound's scale (a number or a string in a window of instants) lies in no window, asnildoes.Macaw.Data.Window.latest/2,Macaw.Data.Window.earliest/2andMacaw.Data.Window.intersect/2keep the first of two bounds that share no scale.Added:
Macaw.Data.Window.comparable?/2, whether two values share the scaleMacaw.Data.Window.compare/2compares them on: both numbers, or both instants of any kinds (#124).Fixed: an
aggregatestep'smin,maxorextentover a time column of mixed kinds no longer raisesArgumentErrorfromMacaw.Data.Operations.apply/4(#141, #143, spec/04 §5). A typed:timecolumn passed the step's reading check by its type, and the fold raised on the second kind. Now these statistics compare times of any kinds on one scale,Macaw.Data.Source.Order.instant/1's, asMacaw.Data.Window.compare/2does (#124). They answer the reading itself, of its own kind, and of two readings at one instant the first.Macaw.Data.Operations.aggregate/4agrees: it answered the fault{["values"], {:reads, :comparable}}for mixed kinds and now answers the value. A number among times, or a time among numbers, is still that fault. A property now checks thatMacaw.Data.Operations.apply/4never raises over a mixed-kind time column for every aggregate, grouped or not.Changed:
Macaw.Data.Window.check/2no longer refuses a region range whose two ends are different kinds of time ({:window, :mixed}). They compare on the instant scale, so only an inverted range is refused (#124). A served window of two kinds of time is no longer malformed toMacaw.Data.Source.Conformanceeither.Changed: finishing a table reads a packed time column (UTC, zoned, naive or date) against window bounds of any kind of time as the buffer lies (#124). On a 50k-row UTC table a naive half window costs 319,993 reductions (~8 ms), as a UTC one does (319,867, ~6 ms), and the sorted table's binary search 6,479. Same-kind windows are unchanged (319,860 before).
Fixed (docs): two v0.10.0 entries open by saying a wrong-kind filter value raises: the ordering entry (#96, #100) and the
eq/neentry (#102, #109). Since #118 (#114) neither raises.Macaw.Data.Operations.apply/4answers the fault{[i, "value"], {:reading, type}}once, before any row is read, andMacaw.Data.Operations.apply!/4raises it, as the v0.10.0 upgrade guide says (#131, #136). The same stale wording is corrected in spec/04, where a step's bad reading is a fault:window({:reads, :time}),latestwithout a schema or an identity ({:needs, :schema},{:needs, :identity}),lttb's y,stackandbin({:reads, :number}). It is also corrected forMacaw.Data.Derived.Sandbox.compile/2without a registry ({nil, {:operations, :registry}}) and inMacaw.Data.Operations.Aggregates' docs.
v0.10.0 (2026-11-02)
Errors are values, a derived source reads columns, and a table is finished on its columns (#114, #107, #105). A public function answers bad input with {:error, reason} and keeps its old return in a ! twin; a CI guard holds lib/ to it (spec/00 §5). A derived source may take its inputs as tables (input: :columns) and read them through Macaw.Data.Derived.Columns, which never makes a row. A derived source's output is a table, and the engine windows and finishes a table input on its columns. filter compares by the column's type, and eq/ne compare a time by its instant (#96, #102). This release is breaking for an adopter that calls any function whose return gained {:ok, _}, matches a derived output's rows, writes a derived source document that a 0.9.x reader must read, or relies on filter across kinds. The guide below lists what to change, and the entries after it give the detail.
Upgrading from 0.9.x
A function that answered its value now answers
{:ok, value}or{:error, reason}(#114, #117–#121). Where the input is known to be good, call the!twin, which keeps the 0.9.x return and raisesArgumentErrorin the module'sformat_error/1words. Otherwise match the result. The functions, each with its twin:Macaw.Data.Delivery:new/2,named_rows/1.Macaw.Data.Source.Finish:finish/4,reduce/5,ceiling/2({:ok, {budget, clamped?}}).Macaw.Data.Source.Order:sort/3.Macaw.Data.Source.Push:new/3.Macaw.Data.Derived.Engine:source/2,3(twinsource!/3).Macaw.Data.Derived.Functions:registry/1.Macaw.Data.Operations:registry/1,step_digest/2.Macaw.Data.Operations.Step:run/4,parts/4.Macaw.Data.Operations.Cache:run/5({:ok, table, cache}).Macaw.Data.Ops:aggregate/2,3,transform/2,3. A stored function that wants the bare value calls the!twin, which the sandbox allows.Macaw.Data.Unit:dimension/1,factor/1,kind/1,symbol/2,convert/3,minor_units/1.Macaw.Data.Quantity:base_unit/1,kind/1,bounds/1,bounds/2,additivity/2,family/2,3,label/1,of_unit/1.Macaw.Data.FFT:fft/1,spectrum/3.
Functions that already answered
{:ok, _}or{:error, _}now answer, rather than raise, for the input they used to raise on:Macaw.Data.Operations.apply/4(a step refusing its input at run time is a fault at its index),Macaw.Data.Table.from/2, the sandbox'scheck/2,compile/2andrun/3({:error, :options, faults}), andMacaw.Data.Delivery.table/2,Macaw.Data.Derived.validate/2andMacaw.Data.Derived.Engine.new/2for an option they do not take. A sum beyond a float's range is:overflowwhere it raisedArithmeticError.Implementers of the behaviors may answer faults. A transform's
run/3andbind/3may answer{:error, {key, reason}}, and an aggregate'sreduce/2andfinish/2may answer{:error, :overflow}. Code that calls these callbacks directly matches those answers.filteracross kinds is a fault (#96, #100, #102, #109, #118). An ordering (lt,lte,gt,gte) compares by the column's type, assortdoes, andeq/neon a time column compare by instant. Avaluethat is no value of the column's type (a number against a time, a string against a number, an unparsable time,nilagainst an ordering) is refused once, before any row is read:apply/4answers{[i, "value"], {:reading, type}}andapply!/4raises it. In 0.9.x these answered silently by term order, so a time ordering kept the wrong rows; an expectation written from those rows changes.filter(test: :eq, value: nil)on a time column becomestest: :absent.A derived source's output is a table (#47, #103, #104).
Macaw.Data.Derived.Engine.outputs/1andMacaw.Data.Derived.Engine.source/2,3's delivery carry aMacaw.Data.Table, soMacaw.Data.Delivery.rows/1of an output reads as any table's rows: every field present,nilwhere the function left one out, keys as strings, and a category's atom as its string (%{"state" => "ok"}, not:ok). Match an output's rows on those readings. Change is told byMacaw.Data.Table.equal?/2, so1→1.0now marks readers, while an atom and its string, or an absent field andnil, do not.Macaw.Data.Derived.Engine.push/4tells a changed delivery by value identity (#106, #108).1.0re-delivered where1was is a change and recomputes; a table and the same rows as a list are two forms, so also a change. One table chunked two ways is not.Macaw.Data.Derived.window/3answers a table for a table input under a request window (window,box,region,latest), where it answered the table's rows (#105, #125). CallMacaw.Data.Table.rows/1on it where you want rows; a list input still answers rows.Columnar derived sources: the form is version 2, and the kernels' interface is 7 (#107, #110–#113, #115).
input: :rows | :columnsis new onMacaw.Data.Derived,:rowsby default, so an existing source runs as before.to_map/1andto_json/1stamp every derived source document version 2. A 0.10 reader reads a version 1 document, but a 0.9.x reader refuses every 0.10 document by name ({:newer_schema, 2, 1}). Upgrade every reader before a writer.- The native kernels' interface version is 7. A build with Zig recompiles them with the package; a library built for 0.9.x is refused at start and every step runs its Elixir path (with
native: :required, the start fails), so rebuild the dependency (mix deps.compile macaw_data --force) if a stale build lingers. - A checker fault naming an allowed module's unknown function whose name is an accessor's names it by atom:
{:restricted, {Macaw.Data.Ops, :mean, 1}}.Macaw.Data.Derived.Sandbox.Allowlist.modules/0lists nine modules.
Changes
Changed:
Macaw.Data.Source.Finish.finish/4finishes a table delivery's:box,:region,:orderand:budgeton its columns too (#105, #126, spec/01 §2.4). No step reads a table's rows, and each keeps exactly the rows it kept before. An order whose head the budget keeps gathers that head alone. On a 50k-row table, reductions before and after:- a region keeping half: 3,697,471 to 718,109;
- an order's top 1,000: 7,851,916 to 1,569,977;
- a budget of 1,000: 10,263,925 to 5,685,119;
- a full order: 6,092,786 to 5,316,784.
Added:
Macaw.Data.Source.Order.sort_with/4sorts any items by an order under a schema, each field read by a function.Macaw.Data.Source.Order.sort/3is it over rows (#126).Changed:
Macaw.Data.Source.Order.sort_by/2sorts one key with:lists.keysort/2, stably in either direction, sosortsteps and finishing order cost less (#126).Breaking:
Macaw.Data.Derived.window/3answers a table for a table input under a request window (window,box,region,latest), where it answered the table's rows (#105, #125, spec/02 §2.2). A list of rows still answers rows. The derived engine makes an input's rows only at a function's call.Changed:
Macaw.Data.Source.Finish.finish/4finishes a table delivery's:windowand:lateston its columns (#105, #125, spec/01 §2.4). It keeps rows of the table's chunks and makes no row, and keeps exactly the rows it kept before. On a 50k-row table, a window keeping half its rows costs 319,799 reductions (5,349,486 before). With the table sorted by its window field, a binary search costs 5,252. Latest costs 609,412 (5,163,695 before).Breaking: a quantity's facts answer
{:ok, fact}or{:error, reason}where a map read raised (#114, #121, spec/03 §9–§11). The functions areMacaw.Data.Quantity.base_unit/1,Macaw.Data.Quantity.kind/1,Macaw.Data.Quantity.bounds/1,Macaw.Data.Quantity.additivity/2,Macaw.Data.Quantity.family/2andMacaw.Data.Quantity.family/3, and the reasons are:{:unknown_quantity, quantity}for a quantity not inall/0;{:unknown_metric_kind, kind};{:unknown_system, system};Macaw.Data.Unit.parse/1's reason for a unit that is no code.
Each has a
!twin with the old return (Macaw.Data.Quantity.base_unit!/1,Macaw.Data.Quantity.kind!/1,Macaw.Data.Quantity.bounds!/1,Macaw.Data.Quantity.additivity!/2,Macaw.Data.Quantity.family!/2,Macaw.Data.Quantity.family!/3).Breaking:
Macaw.Data.Source.Push.new/3answers{:ok, push},Macaw.Data.Derived.Engine.source/2,3{:ok, output}andMacaw.Data.Source.Finish.ceiling/2{:ok, {budget, clamped?}}, or{:error, faults}(#121, spec/00 §5):- an option a function does not take is
{[key], :unknown_key}; - a missing
host:is{[:host], :required}; - a path the engine does not define is
{[:path], :unknown_path}.
Macaw.Data.Source.Push.new!/3,Macaw.Data.Derived.Engine.source!/3andMacaw.Data.Source.Finish.ceiling!/2keep the old returns.- an option a function does not take is
Changed:
Macaw.Data.Delivery.table/2,Macaw.Data.Derived.validate/2andMacaw.Data.Derived.Engine.new/2answer an option they do not take as a fault,{[key], :unknown_key}(the engine's{[], [key], :unknown_key}), whereKeyword.validate!/2raised (#121).Changed: the raise guard's allowlist holds only bang helpers, compile-time code and named invariants. The
:convertclass is refused (#121, spec/00 §5).Breaking: the functions of a unit code answer
{:ok, value}or{:error, reason}instead of raising (#114, #120, spec/03 §2, §5, §6):Macaw.Data.Unit.dimension/1,Macaw.Data.Unit.factor/1,Macaw.Data.Unit.kind/1andMacaw.Data.Unit.symbol/2:parse/1's reason for a string that is not a code, and{:syntax, text}for a term that is no string;Macaw.Data.Unit.convert/3:conversion/2's reason, or:not_a_number;Macaw.Data.Unit.minor_units/1:{:unknown_currency, code}.
Each has a
!twin (Macaw.Data.Unit.dimension!/1,Macaw.Data.Unit.factor!/1,Macaw.Data.Unit.kind!/1,Macaw.Data.Unit.symbol!/2,Macaw.Data.Unit.convert!/3,Macaw.Data.Unit.minor_units!/1) with the old return.Breaking:
Macaw.Data.Quantity.label/1,Macaw.Data.Quantity.of_unit/1andMacaw.Data.Quantity.bounds/2answer{:ok, value}or{:error, reason}(#120, spec/03 §9, §14).Macaw.Data.Quantity.label!/1,Macaw.Data.Quantity.of_unit!/1andMacaw.Data.Quantity.bounds!/2keep the old returns.Macaw.Data.Quantity.bounds/1is unchanged.Breaking:
Macaw.Data.FFT.fft/1answers{:ok, terms}andMacaw.Data.FFT.spectrum/3{:ok, lines}, or{:error, reason}:{:samples, n},{:window, kind},{:every, every}or{:values, :number}, worded byMacaw.Data.FFT.format_error/1(#120, spec/04 §4.11).Macaw.Data.FFT.fft!/1andMacaw.Data.FFT.spectrum!/3keep the old returns.Breaking: a sum beyond a float's range is the fault
:overflowwhere it raisedArithmeticError(#120, spec/04 §5):sumandmeananswer{:error, :overflow}, whichMacaw.Data.Aggregatenow allows fromreduce/2andfinish/2;Macaw.Data.Operations.aggregate/4answers{["values"], :overflow};- a grouping step answers
{[i, "field"], :overflow}, andbinanswers{[i, "fn"], :overflow}; Macaw.Data.Derived.Columns.sum/2andMacaw.Data.Derived.Columns.mean/2answer{:overflow, name}.
Breaking: the sandbox's options are faults, not raises (#120, spec/02 §6.4).
Macaw.Data.Derived.Sandbox.check/2andMacaw.Data.Derived.Sandbox.compile/2answer{:error, [{nil, {:option, key}}]}and{nil, {:bound, key, ceiling}}, and a compile without a registry{nil, {:operations, :registry}}.Macaw.Data.Derived.Sandbox.run/3answers{:error, :options, faults}.Macaw.Data.Derived.Engine.new/2answers a bound above its ceiling as{[], [key], {:bound, key, ceiling}}.Breaking:
Macaw.Data.Derived.Functions.registry/1answers{:ok, registry}or every fault:{[module], :not_a_function_module}and{[name], {:defined_twice, module, other}}(#120, spec/02 §7).Macaw.Data.Derived.Functions.registry!/1keeps the old return.Changed: the conformance kit's
Macaw.Data.Source.Conformance.check/3answers its own options' faults as findings:{check, {:unknown_check, checks}},{check, {:option, key}},{check, {:request, ask}}and{check, {:viewers, :two}}. The test it writes fails with their words, wherecheck/3raised before (#120, spec/01 §9).Breaking:
Macaw.Data.Source.Finish.finish/4answers{:ok, delivery}or{:error, faults}instead of a delivery or a raise (#114, #119, spec/01 §2.4, spec/04 §7). Every fault is answered before any step runs:- a request the schema cannot answer:
Macaw.Data.Window.check/2's faults; {[:reduce], {:unknown, :reducer, name}};{[key], :unknown_key}for an option it does not take;- tuple rows without names:
{[:names], :required}; - a number field's reading the reducer cannot place:
{[:rows, field], {:reads, :number}}.
Macaw.Data.Source.Finish.reduce/5answers the same way.Macaw.Data.Source.Finish.finish!/4andMacaw.Data.Source.Finish.reduce!/5keep the old returns.- a request the schema cannot answer:
Breaking:
Macaw.Data.Source.Order.sort/3answers{:ok, rows}, or{:error, faults}(Macaw.Data.Source.Order.faults/2's: an unknown field, a field of a type with no order, an order of the wrong shape) instead of raising (#119, spec/01 §2.3).Macaw.Data.Source.Order.sort!/3keeps the old return.Added:
Macaw.Data.Table.source/1checks a source's shape::ok, or every fault (#119, spec/01 §1.1):{[], {:type, :rows}}: a list mixing row maps and other terms;{[name], {:length, n}},{[name], {:type, :column}}: a column map's unequal or non-list column;{[], {:rank, rank}}: a tensor of rank other than 1 or 2;{[], {:type, :source}}: any other term.
Macaw.Data.Table.from/2answers those faults where it raised.Macaw.Data.Delivery.new/2answers a column map's or a tensor's shape faults under[:rows], andMacaw.Data.Operations.apply/4answers them under"source".Macaw.Data.Table.rows/1reads a source thatsource/1accepts.Breaking:
Macaw.Data.Operations.apply/4keeps its own contract: it answers{:ok, table}or{:error, faults}and never raises (#114, #118, spec/04 §3.1). A step that refuses its input at run time stops the run with a fault at its index, where it raisedArgumentErrorbefore:{[i, key], {:reads, kind}}: a column holds a reading of the wrong kind (:number,:time,:positive,:comparable,:point);{[i, "value"], {:reading, type}}: filter's cross-kind value (#100, #109);{[i, "latest"], {:needs, :schema | :identity}}: window'slatest;{[i, "op"], {:context, :now, :time}}and{[i, "op"], {:unknown, :scale, scale}}: the context;{[i, "op"], :not_a_table}: a transform answering something that is not a table;{["source"], :not_a_table}, orMacaw.Data.Table.from/2's faults under"source": a source that is not a table.
Macaw.Data.Operations.apply!/4raises them inMacaw.Data.Operations.format_error/1's words. A typed column is read by its type alone, so a run over typed columns does no extra work: the frame loop's full recompute of its five chains costs 1,936,195 reductions on the Elixir path (1,935,794 to 1,936,190 before) and 179,223 with the kernels (179,160 to 179,193), the same milliseconds within noise.Breaking:
Macaw.Data.Operations.registry/1answers{:ok, registry}or{:error, faults}:{[module], :not_a_package},{[package, member], :no_behavior},{[kind, name], {:named_twice, module, other}}and{[:aggregate, name], {:step_key, key}}(#118, spec/04 §2).Macaw.Data.Operations.registry!/1keeps the old return.Macaw.Data.Operations.registry/0is unchanged.Macaw.Data.Derived.Engine.new/2answers a registry fault as{[], [:packages | path], reason}.Breaking:
Macaw.Data.Operations.step_digest/2answers{:ok, digest}, or{:error, {["op"], {:unknown, :transform, op}}}and{:error, {["op"], :required}}.Macaw.Data.Operations.step_digest!/2keeps the old return (#118).Breaking:
Macaw.Data.Operations.Step.run/4answers{:ok, table}andMacaw.Data.Operations.Step.parts/4{:ok, parts}, or the step's fault{:error, {key, reason}}.Macaw.Data.Operations.Cache.run/5answers{:ok, table, cache}or the fault (#118, spec/04 §1.4, §1.5).Macaw.Data.Operations.Step.run!/4,Macaw.Data.Operations.Step.parts!/4andMacaw.Data.Operations.Cache.run!/5keep the old returns.Breaking: a transform's
run/3(Macaw.Data.Transform) andbind/3may answer{:error, {key, reason}}, the step's fault over its input. (#118).Added: an aggregate may declare
reads/0, what its readings are read as::number,:comparable,:pointor:any, the default (Macaw.Data.Aggregate.reads/1).Macaw.Data.Operations.aggregate/4checks the values against it and answers{["values"], {:reads, kind}}(#118, spec/04 §1.2, §5).Breaking:
Macaw.Data.Ops.aggregate/2,3answers{:ok, value}andMacaw.Data.Ops.transform/2,3{:ok, rows}, or{:error, faults}, instead of raising (#118, spec/04 §6). In a stored function a fault's atoms are spelled as the function's literals are, so the function matches it with its own words.Macaw.Data.Ops.format_error/1words it.Macaw.Data.Ops.aggregate!/2,3andMacaw.Data.Ops.transform!/2,3keep the old returns, and all five are on the sandbox's allowlist. A stored function that wants the bare value moves to the!twin. Options that name an op are now{[0, "op"], {:invalid_for, name}}.Changed: spectrum's
asis checked as one name or a list of two or three ({:list, :name, 2, 3}, a new option type), so a bad list is a validation fault before the step runs (#118, spec/04 §1.3, §4.11).Fixed: grouping
bya column the rows lack no longer crashes. Every row's key isnil, as a missing column reads. The property thatMacaw.Data.Operations.apply/4never raises found it (#118).Breaking:
Macaw.Data.Delivery.new/2answers{:ok, delivery}or{:error, faults}instead of a delivery or a raise (#114, #117, spec/01 §3). Each fault is{path, reason}, and all of them are answered at once, in the order of the fields:{[:rows], {:type, :rows}},{[:rows], :delivery},{[key], :unknown_key},{[:served, fact], :unknown_key},{[:served, fact], {:type, kind}}or{:enum, values}, and the{:type, kind}ofnames,types,schema,origin,retractedandreplace, with an index for a bad retracted row or replacement.Macaw.Data.Delivery.format_error/1words them.Macaw.Data.Delivery.new!/2keeps the old return and raisesArgumentErrorin those words. Upgrade: callnew!/2where the input is known to be good, or match the result.Breaking:
Macaw.Data.Delivery.named_rows/1answers{:ok, rows}or{:error, [{[:names], :required}]}for tuple rows without names, or{:error, [{[:names], {:width, n, width}}]}for rows that don't fit their names, instead of raising (#117).Macaw.Data.Delivery.named_rows!/1keeps the old return.Measured:
Macaw.Data.Delivery.new/2costs 64 reductions with no options (69 before) and 121 with every option (115 before). Both take 0.6 to 3 µs, the same as before within the noise.Added: errors are values (#114, #116, spec/00 §5). A public function answers bad input with
{:error, reason}; aname!twin raises its reason, a macro's misuse is aCompileError, and a raise remains only for a named internal invariant. The CI jobraises(ci/raises.exs) fails on araise,reraiseorthrowinlib/outside a bang twin andci/raises_allowlist.exs. Its 46:convertentries, the 59 sites still to convert, go as this release's work items land.Added: a
:columnssource may answer a table or a column map (#107, #113, spec/02 §1.2, §8.2).Macaw.Data.Derived.conform/2holds the answer to the schema column by column and answers the output table,{:ok, table};Macaw.Data.Derived.Enginepacks it without making a row.- A whole column's fault has index
nil. The new reasons are:duplicate,:not_a_columnand{:length, n}. A table column of another type is{:type, type}. - Any other fault is at the lowest index, with an identity gap before a value fault. A single fault is the one the answer's rows would give.
- A table already in the output's layout is the output as it is.
- Under
per, an entity may answer a table, which is read as its rows. - A
:rowssource still answers rows alone.
- A whole column's fault has index
Faster: answering 5,000 rows of two fields as a column map costs the engine 164,585 reductions to conform and pack, where the same rows cost 976,673.
Added: a derived source may read its inputs as columns:
input: :rows | :columnsonMacaw.Data.Derived(#107, #112, spec/02 §1.1, §2.2, §7, §8.2). It is:rowsby default, and a:rowssource runs exactly as before.- Under
:columns,Macaw.Data.Derived.Enginehands the function each input as aMacaw.Data.Table, read withMacaw.Data.Derived.Columns, and makes no row. - A table delivery is handed over as windowed. A list delivery, or a request window's finished rows, is packed under the input's schema. Rows that don't pack are the fault
:nonconforming, with detail{:input, key, faults}and no value. - A stored function's tables are detached (no origin, no chunk identity, no fetch closure), so they hold only their values.
- Under
per, each entity gets a table of its rows, gathered from the input's chunks. - The option applies to stored and host functions alike: the source that names a host function says how it is called.
new/1refuses any other value with{[:input], {:enum, [:rows, :columns]}}.
- Under
Changed: the derived source's map and JSON form is version 2 (#112, spec/02 §1.5). It writes
"input": {"$atom": "columns"}for a columnar source and nothing for:rows. A version 1 document reads as a rows source. A reader of version 1 refuses a version 2 document by name ({:newer_schema, 2, 1}).Faster: one recompute of a source summing one field of a 50,000-row, four-column table input, counting the engine's process and the function:
Source Reductions ms, default heap ms, 4M-word heap host :rows2,406,074 225.3 81.8 host :columns, kernels2,375 0.9 0.4 host :columns, Elixir path152,407 n/a n/a stored :rows11,757,116 487.7 233.3 stored :columns, kernels3,684 0.5 0.5 stored :columns, Elixir path153,716 n/a n/a The
:rowsrows are unchanged from before. On the Elixir path, only the function's reductions were measured without the kernels.Added: a stored function may read columns:
Macaw.Data.Derived.Columnsis on the sandbox's allowlist with every accessor, its!variant andformat_error/1(#107, #111, #115, spec/02 §6.1–§6.3). It is written by its full name, asMacaw.Data.Opsis.- The checker holds a literal column name, a string or an atom, to
fields:as it holdsrow.field, soColumns.sum(t, "w")over inputs withoutwis{:unknown_field, "w"}at its line. - Every accessor compiles to a twin in the sandbox's runtime that spells its names, values and reasons by the function's literals' rule. So a stored function's
:okmatches a category's"ok", as it does in rows, and{:error, {:unknown_column, _}}matches the reason a missing column answers. - A stored function that matches
{:ok, v} =on a bad call, or calls a!variant, raises; the runner answers{:error, :raised, MatchError}orArgumentError. - A map read of a table (
t.chunks) raisesArgumentError, so a table's fields are read through the accessors alone. Macaw.Data.Derived.Sandbox.Allowlist.modules/0now lists nine modules.
- The checker holds a literal column name, a string or an atom, to
Changed: a checker fault that names an unknown function of an allowed module whose name is an accessor's (
Macaw.Data.Ops.mean(rows)) names it by the atom ({:restricted, {Macaw.Data.Ops, :mean, 1}}), as it names every function the checker knows; it named it by its string. A fault that names a function no allowed module has is unchanged.Added:
Macaw.Data.Derived.Columns, a derived function's input read as columns (#107, #110, spec/02 §2.2). It holds accessors over a windowed input table that read its packed buffers and never make a row:names/1,type/2andsize/1;- the positional reads
column/2,numbers/2,instants/2(integer Unix microseconds onMacaw.Data.Source.Order.instant/1's scale, read from the buffer with no time made) andvalid/2, eachnil(orfalse) at a gap; - the reductions
count/2,sum/2(an exact sum, as thesumaggregate's),mean/2,min/2andmax/2, which mean what the aggregates of the same names mean, the first on a tie and a time compared by its instant; slice/3,take/2androws/1.
Every accessor answers
{:ok, value}or{:error, reason}and raises on no input (#115). The reasons are{:unknown_column, name},{:wrong_type, name, expected, actual},{:type, :name},{:type, :table},{:type, :integer}and{:type, :non_neg_integer}. Each accessor has a!variant that answers the value alone or raisesArgumentErrorinformat_error/1's words. A column is named by either spelling. With the native kernels loaded,sum,countandmeanread the exact-sum kernel (group_sum), ungrouped, andminandmaxread a new kernel,extremes, which answers a view's first least and first greatest slot of an int64, float64 or int32 buffer. The kernels' interface version is now 7. The Elixir path reads the same buffers and answers the same, term for term. The engine's opt-in,input: :columns, comes in #112.Fixed:
filter'seqandnecompare a time column by its instant, as its orderings do (#102, #109, spec/04 §4.1). They compared each reading withvalueby struct equality, so one instant written in another zone or at another precision (~U[2026-01-01 00:00:00Z]and the same instant at+01:00, or at microsecond precision 0 and 6) passedlteandgteyet was noteq, and a naive or date column was nevereqaDateTime. Nowvalueis read at the edge as the orderings read it (aDateTime,NaiveDateTime,Dateor ISO 8601 string) and compared byMacaw.Data.Source.Order.instant/1, solteandgteboth hold exactly wheneqdoes.nestayseq's negation: anilreading passesne. The rule for every column is stated in spec/04 §4.1: equal on the type's scale, so on a number column1equals1.0and0.0equals-0.0(unchanged). This is comparison, not identity:1→1.0is still a change to a derived source's answer and to a pushed delivery (spec/01 §1.1.2, spec/02 §8.2).Breaking (behavior):
eqandneon a time column against a value with no instant (a number, an unparsable string,nil) raiseArgumentErroronce over the input, before any row is read, with the orderings' message (#102, #109). They used to keep no row (ne: every row).filter(test: :eq, value: nil)on a time column istest: :absent. On any other column a value of another kind is still answered, not refused. Since #118,Macaw.Data.Operations.apply/4answers this as the fault{[i, "value"], {:reading, type}}andapply!/4raises it.Faster:
eqandneon a packed time column run in the native kernel's int64 mode on the:utc,:zonedand:naiveforms, and without the kernels scan the same buffer (and a date column's days), building noDateTime. Measured over one 8,192-row chunk (apply!/3, the least of 7 runs; the box at load 20, so ms are noisy):column, test native, before native, now Elixir path, before Elixir path, now UTC time, eq99,665 r, 2.0 ms 1,387 r, 0.08 ms 99,666 r, 1.9 ms 9,586 r, 0.99 ms UTC time, ne117,086 r, 2.5 ms 3,618 r, 0.29 ms 125,029 r, 2.5 ms 25,029 r, 1.1 ms UTC time, a tenth missing, eq131,527 r, 3.1 ms 1,361 r, 0.05 ms 127,922 r, 2.4 ms 11,635 r, 0.14 ms naive time, eq116,765 r, 2.2 ms 1,219 r, 0.05 ms 116,706 r, 2.0 ms 9,418 r, 1.0 ms date, eq82,892 r, 1.6 ms 25,594 r, 0.12 ms 82,893 r, 1.5 ms 25,595 r, 0.12 ms Breaking (behavior):
Macaw.Data.Derived.Engine.push/4tells a changed delivery by value identity, as an answer's change is told (#106, spec/02 §8.2). It compared the whole delivery with==, which takes1for1.0: a source that re-delivered1.0where it delivered1marked no reader, and its derived sources kept an answer computed from the old value. Now the rows of two tables compare byMacaw.Data.Table.equal?/2(so one table chunked two ways is no change, where==of the structs called it one), any other rows by===, and everything else a delivery carries (served facts, schema, origin, retractions, replacements) by===. A table and the same rows as a list are two forms, so a change. The answer's watermark is compared exactly too: acomplete_untilof1then1.0is a changed answer.Breaking (behavior): a derived source's output is a table, packed once under its output schema, and its change is told by the table's identity (#47, #104, spec/02 §8.2, §8.4, spec/01 §1.1.2).
Engine.outputs/1andEngine.source/2's delivery carry aMacaw.Data.Table, one column a field, named by its string, in name order. SoDelivery.rows/1of an output reads as any table's rows: every field present,nilwhere the function left one out, and a category's atom as its string (%{"state" => "ok"}where it was:ok). A reader that wants columns no longer packs the rows again:Delivery.table/2of a 5,000-row output cost 381,292 reductions per read, and the engine now packs it once, for 245,278. A downstream derived source windows it on its columns (#103). The change check isMacaw.Data.Table.equal?/2, not!=over the conformed rows:1and1.0are now a change, which marks the readers. A category's atom and its string, and an absent field andnil, are not a change, where they were. Where you match an output's rows, match the table's reading.Faster:
Macaw.Data.Table.equal?/2of two tables of whole hashed chunks laid out alike compares their chunks' digests and decodes no column. Two 50,000-row canonical tables that differ in one value are told apart in under 1,000 reductions, where comparing their columns decoded both. One recompute of an expression keeping 5,000 of 50,000 input rows: 5,701,833 → 5,944,162 reductions (the packing), 694.6 → 696.4 ms at the default heap.Faster: the derived engine windows a table input as a table (#47, #103, spec/02 §2.2, §8.2). It used to transpose every input with
Delivery.rows/1, then sort and group row maps for%{last: n}. Nowlast: nruns on the columns. It reads the window field as keys (a packed time column's instants straight from its buffer) and the identity fields as values, and skips the sort for a table already sorted by its window field. It keeps the rows as views of the input's chunks, or gathers them by their slots (Chunk.gather/5). A request window still finishes the table's rows, transposed once, as before: finishing a table as a table repacks what it keeps, which costs 37% more (#105 proposes columnar finishing). The rows a function reads are made once per input, at the call, from the windowed table. A row input is handed over as given, as before. One recompute of alast: 20source over a 50,000-row table input of 50 hosts: 7,923,737 → 915,785 reductions, 322–353 → 16 ms at the default heap, and 146–151 → 9 ms atmin_heap_size4M words.Derived.window/3takes a table and answers one.Fixed:
last: norders a time by its instant (Source.Order.instant/1), whatever its kind, to the microsecond (#47, #103). It usedWindow.position/1, which keyed aDateTimein Unix microseconds, aNaiveDateTimein Gregorian seconds as a float and aDatein Gregorian days. So a window field mixing kinds sorted by scale: aDatein 2027 stood before aDateTimein 2026 and was dropped as oldest. Naive times a few microseconds apart also tied, near 6.4e10 seconds, and kept their arrival order. The row window and the columnar one share the key, so they cannot drift. A window field that is neither:timenor:numberkeeps its rows' order, as a:pointfield did. So does a value with no reading for its type, such as a number in a:timefield, which used to sort as a number.Fixed:
filter's orderings compare by the column's type, assortorders (#96, #100, spec/04 §4.1).lt,lte,gtandgtecompared a reading withvalueby term order, so a time column compared each%DateTime{}as a map, by day of month before year.lt 2026-01-01kept none of2026-10-02, 2026-09-30, 2026-10-01, 2025-12-31, andgtkept all four. A naive or date time column against aDateTimekept nothing. The reading and the value are now keyed byMacaw.Data.Source.Order.key/2, the comparator ofsortand a request's order, so the three cannot drift: a time by its instant, a number by its value, text and a category by spelling. The value is read as the column reads it at the edge, so against a time column aDateTime, aNaiveDateTime, aDateand an ISO 8601 string are each an instant. Anilreading, and one that is no value of its column's type, never passes.Breaking (behavior): an ordering whose
valueis no value of the column's type raisesArgumentErroronce over the input, before any row is read. The message names the step,value, the test, the field and its type. That covers a number against a time column, a string against a number column, an unparsable string against a time column, andnilagainst any column. Under term order these answered silently: every number stood below every string and atom. A column the rows lack still keeps no row and raises nothing. Where a node compared across kinds on purpose, give it a value of the column's type. Since #118,Macaw.Data.Operations.apply/4answers this as the fault{[i, "value"], {:reading, type}}andapply!/4raises it.Changed:
Order.key(:time)keys every kind of time on one scale (Order.instant/1, new): Unix microseconds, aNaiveDateTimeby its wall clock, aDateat midnight. It wasWindow.position/1, which keys the three on three scales, so a naive, date or mixed time column sorted wrongly besideDateTimes. A number in a time field is no instant and sorts last.Faster: a time ordering runs in the native kernel on the
:utc,:zonedand:naiveforms, comparing the column's int64 instants with the value's. No kernel changed: this is the number mode. Without the kernels, the Elixir path scans the same buffer (and a date column's days), gaps or none, and builds noDateTime. Measured over one 8,192-row chunk (apply!/3,gt, the least of 7 runs in reductions and of 21 in ms, on a host at load 25 to 30):column native, 0.9.0 native, now Elixir path, 0.9.0 Elixir path, now UTC time 120,068 r, 3.7 ms 3,502 r, 0.17 ms 124,987 r, 3.2 ms 23,191 r, 0.77 ms UTC time, a tenth missing 148,710 r, 4.3 ms 3,390 r, 0.21 ms 177,926 r, 4.6 ms 47,898 r, 0.39 ms naive time 129,004 r, 1.7 ms (wrong rows) 3,502 r, 0.17 ms 129,008 r, 2.2 ms (wrong rows) 23,191 r, 0.77 ms date 90,802 r, 0.87 ms (wrong rows) 33,280 r, 0.26 ms 90,806 r, 0.79 ms (wrong rows) 38,339 r, 0.24 ms Number and text orderings cost what they did: within 3% in reductions, and in ms within the host's noise.
v0.9.0 (2026-11-01)
On macaw_layers 0.4.0, and a request carries a row order (#92, #94). A capped or budgeted delivery now holds the first rows in the order a reader asked for, where it used to hold a sorted sample. The schema's JSON document carries a version stamp and is read through the kernel's gates (#95). A derived source can hold no URL. sort orders a time column by its instant (#93). A raise in a stored function or a conformance callback is answered without logging an [error] (#83). This release is breaking for an adopter that pins macaw_layers ~> 0.3, matches the fault shapes below, or reads Schema.to_json/1's output itself. The guide below lists what to change, and the entries after it give the detail.
Upgrading from 0.8.x
- Move to macaw_layers
~> 0.4.0(#92). macaw_data 0.9.0 requires it, so an application that pins{:macaw_layers, "~> 0.3.0"}(or~> 0.3) widens the pin to"~> 0.4.0"and follows macaw_layers 0.4.0's own upgrade notes:Form.read/3takesurls:and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither change touches what it reads. - Match the new fault shapes.
- A derived source document's
$atomoutside the form's atoms is{:unknown_atom, spelling}at its path. It was{:enum, [:identity]}. - A URL anywhere in a derived source document (
{"$url": …}) is{path, {:url, reason}}at the source's path. A schema document refuses one by type. Schema.from_json/2's malformed JSON is{[], {:json, words}}, where it was{:json, inspect(reason)}. A newer stamp is{[], {:newer_schema, needs, 1}}, a stamp that is no positive integer is{["$version"], {:type, :pos_integer}}, a key beside"$version"and"fields"is{[key], :unknown_key}, and no"fields"is{["fields"], :required}.- A request's order that is malformed or that the schema can't answer is
{[:order, i], reason}fromRequest.new/1andWindow.check/2, and{:error, {:order, faults}}fromSource.fetch/3. Derived.format_error/1andSchema.format_error/1word the kernel's reasons in the kernel's words (Macaw.Layers.Fault). A test that matches the oldinspect/1text needs its expected words updated.
- A derived source document's
Schema.to_json/1writes an object (#95):{"$version": 1, "fields": [[name, description], …]}. Where you decode its output yourself, match on the object, not a list. A document written by 0.8.x or earlier still reads:Schema.from_json/2lifts its list root to version 1.to_map/1andfrom_map/1are unchanged, still the bare pairs, so a derived source's embedded"schema"is unchanged too.Request.orderis new and defaults to[](#94). A request with no order behaves as before. A source that declares the:ordercapability must serve its firstbudgetrows in the order and sayserved.ordered: :source. Otherwise finishing sorts what the source served.Finish.reduce/5of an ordered delivery keeps the head where it would have reduced by LTTB or M4.Description.capabilities/0now lists:order, so a test that pins that list needs updating.sortputs anillast in either direction (#93). It came first for a time, text or category column ascending, and first for a number column descending. A time column now sorts by its instant, not by struct term order, so an expectation written from the old order changes.
Changes
- Breaking: the schema's JSON document has a version stamp, read through the kernel's gates (#84, #95, spec/01 §1.11).
Schema.to_json/1writes an object,{"$version": 1, "fields": [[name, description], …]}, so the first change to the form can tell an old document from a new one.to_map/1andfrom_map/1stay the bare pairs, for a document that embeds a schema (a derived source's"schema", unchanged).Schema.from_json/2reads throughMacaw.Layers.Form.gates/3, with no limit or scan code of its own: a newer stamp is refused by name before anything is read, malformed JSON is the kernel's{:json, words}(was{:json, inspect(reason)}), and a key beside"$version"and"fields"is{[key], :unknown_key}. A document written by 0.8.x or earlier still reads: its root is a list, which no version of the object has, so it is lifted to version 1 once its bytes and nesting pass the gates. Match on the object, not a list, where you readto_json/1's output yourself.Schema.format_error/1words the kernel's reasons where it raised for them. New: a request carries a row order, so a capped delivery holds the first rows in it (#80, #94, spec/01 §2.3–§2.4, §3, §9).
Request.orderis[{field, :asc | :desc}, …],[]by default. A "slowest first" table under a budget showed a sorted sample, since a source caps before the runtime sees a row (macaw#196).Window.check/2holds the order's fields to the schema, as{[:order, i], {:unknown_field, name}}or{:invalid_for, type}for:pointand:color.Source.fetch/3refuses such an order before calling the source,{:error, {:order, faults}}, as it refuses parameters. Rows compare by their field's type (Macaw.Data.Source.Order): a time by its instant, a number by value, text by string, a category by its declaredorder, a missing reading last in either direction, stably.- New: the
:ordercapability andserved.ordered. A source naming:orderserves its firstbudgetrows in the order and saysordered: :source. Otherwise finishing sorts what it served, in a new step before the budget:ordered: :runtime, every row ordered, the head kept; orordered: :sample, where the source had already cut its rows (reduced_by: :sourceorcapped), so the reader is told they need not be the first. The conformance kit takesorderinrequests:and checks a source naming:order. A source stating:runtimeor:sampleis a finding. - Changed (behavior):
Finish.reduce/5of an ordered delivery (served.orderedset) keeps its firstbudgetrows, whatever reducer is named, where it reduced by LTTB or M4. A pooled ordered fetch reduces to each reader's own head. Deliveries with no order are reduced as before.Description.capabilities/0lists eight,:orderbefore:budget. - Breaking: on macaw_layers
~> 0.4.0(was~> 0.3.0) (#91, #92). An adopter on macaw_layers 0.3.x moves to 0.4.0 with it, following its upgrade notes:Form.read/3takesurls:and refuses every URL without it, and a theme refuses untrusted text. macaw_data builds no theme and its documents hold no URL, so neither touches what it reads (spec/00 §3). - Breaking (faults): a derived source document's
$atomoutside the form's atoms is{:unknown_atom, spelling}at its path, where it was{:enum, [:identity]}(macawlayers 0.3.1, spec/02 §1.5). Match on `{:unknown_atom, }fromDerived.from_map/2andfrom_json/2`. - Fixed: a derived source holds no URL (spec/02 §1.5). macaw_layers 0.4.0 makes
$urllive in every tag table, and a derived source's reader composes the kernel's gates withnew/1rather thanForm.read/3, so a{"$url": …}at a window's bound read into a%Macaw.Layers.Url{}thatnew/1accepted, under no host's policy. The reader now holds the source it reads to the kernel's closed policy: a URL, whatever its scheme, is{path, {:url, reason}}at the source's path, and its words never quote it. A schema's reader reads no$url, so one is refused by type (spec/01 §1.11). - Changed:
Derived.format_error/1words the kernel's own reasons (an unknown tag or atom, a limit, a newer version, a URL) in the kernel's words (Macaw.Layers.Fault), where it showed them withinspect/1. - Fixed:
sortorders a column by its type, not by term order (#93, #97, spec/04 §4.6). A time column was sorted by%DateTime{}'s struct term order, so by day of month before year:2026-10-01, 2026-10-02, 2026-09-30, 2025-12-31ascending. A time now sorts by its instant, asDateTime.compare/2orders it, a number by its value, and text or a category by its spelling.:color,:pointand:termkeep term order. The step sharesMacaw.Data.Source.Order's comparison (key/2,sort_by/2, new) with a request's row order, so the two order a field alike. Changed (behavior): anilreading sorts last in either direction. Before, it came first for a time, text or category column ascending, and first for a number column descending.filter's orderings still compare by term order (#96). - Fixed: a raise answered by the bounded runner logs no error (#83, spec/02 §6.4, spec/01 §9).
Macaw.Data.Boundedspawned its process with:erlang.spawn_opt/2, so a raise in a stored function, in a compile the compiler refused, or in a conformance callback was also reported by the runtime, as an[error]logged a few milliseconds after the caller had its answer,{:error, :raised, module}. A host running its users' functions logged an error for every user mistake. The runner now spawns through:proc_lib.spawn_opt/2: the answer and the exit reasons are unchanged, and nothing is rescued. - Changed (internal): the schema's JSON adapter lives in its own hidden module, Macaw.Data.Schema.Form (#85, spec/01 §1.11).
Schema.to_map/1,to_json/1,from_map/1andfrom_json/2stay onMacaw.Data.Schemawith their docs, so the public API is unchanged. - Tests: the dev-only
surfexdependency is~> 0.11.0(#90), andWorkTestboundsmaps:from_list/1's randomly seeded sort charge (#87). Neither changes the package. - A value entry's
{"$ramp": [name, t]}reads for any name in a theme's grammar (macaw_layers 0.3.1'sColor.Form), as$colordoes, so a schema's ramp color round-trips through JSON.
v0.8.1 (2026-10-31)
An application without Zig starts with no error logged (#76). Every app on macaw_data without Zigler, the default, logged an [error] crash report at each start, though nothing was wrong. Upgrading from 0.8.0 needs no change; from 0.7.0, follow 0.8.0's guide below.
Changes
- Fixed: an application without the native kernels starts with nothing logged at error level (#76, #77, spec/04 §1.7).
Macaw.Data.Nativeasked the kernels their interface version at the application's start even when the build had only their stub, and the stub's:not_builtraise, in the process that asked, logged a crash report.Macaw.Data.Native.loaded?/0was false, as it should be. Now only built kernels are asked: a build without them is known at compile time and starts quietly. With Zig,native: :optionaland:requiredbehave as before. - Fixed: the Hex package never carries Zigler's copy of the kernels' source (#77). Zigler writes
.Elixir.<Module>.zigbeside a module it compiles, so a package built from a checkout that had compiled the kernels would carry it. The package excludes such files. CI now builds the package, boots a minimal app from it without Zigler, and fails on any[error]line, aloaded?/0that isn't false, or such a file. - Changed: each build stages the native kernels in its own directory (#72, #73, spec/04 §1.7). Zigler stages a module in
/tmp/<module>by default, one directory for every checkout, worktree and build path on the host, so two builds at once wrote each other'sbuild.zigandmodule.zig. The kernels now stage under the build's path,<build path>/zigler/, unlessZIGLER_STAGING_ROOTnames another root. - Tests: suite verdicts are independent of load and build state (#73, #74, #75, #81): a
--warnings-as-errorsabort on a warning a lazily loaded test module raised is fixed,LadderNativeTest's property is bounded by its work in reductions rather than ExUnit's clock, and the suite's reduction counter is exact but formaps:from_list/1, whose residue it bounds.
v0.8.0 (2026-10-31)
The source contract is a time-varying relation, and a table is chunked (macaw_data#10, #9), with optional native kernels (#38). A push is a numbered changelog of increments, retractions and range replacements, with settled ranges and derived watermarks. A table is a list of immutable chunks with stable ids, and every step runs a chunk at a time, through a caching contract a caller holds. Where Zig is in the build, the hot steps run natively, bit for bit as their Elixir path, and a 50k-row live frame meets visualize's ~17 ms budget. The suite's verdicts no longer depend on how loaded the host is. On macaw_layers ~> 0.3.0, as 0.7.0 was. This release is breaking: the guide below is what an adopter of 0.7.0 changes, and the entries after it are the detail.
Upgrading from 0.7.0
- A table is chunked, opaque, and equal by its contents (#17). Compare tables with
Macaw.Data.Table.equal?/2, never==, since the same rows chunked two ways are one value. Read a table through its functions (rows/1,get/2,column/2,chunks/1), never the struct's fields.Table.t()is opaque. Keep functions, PIDs, references and ports out of tables:new/1refuses them as{[name, i], :not_data}. - Number a push's increments (#12). A source sends
Push.increment_message/3, numbered from 1 per subscription, and may sendPush.snapshot_message/2withserved.position.Push.delivery_message/2is gone. Rows are no longer deduplicated: an identical row pushed again is a second row. Guard redelivery by the number, not by the row. Push.held/1's rows are aMacaw.Data.Table(#19). Read held rows by name withMacaw.Data.Table.get/2, compare a category as a string, and expect a time column's values asDateTimes. Rows with no window value sort last.sumandmeanare exact (#21). Integers stay an integer. With a float among them, the exact sum is rounded once, ties to even, so a float sum may differ from a sequential sum in the last place:[1.0e16, 1.0, -1.0e16]sums to1.0, where it was0.0. A golden written from a sequential float sum may need regenerating.lttbis MinMaxLTTB, andm4andlttbreduce on one bucket ladder (#22).lttbkeeps at mostnrows within its stated tolerance, the same under every chunking. The exact algorithm stays asMacaw.Data.Reduce.lttb/3.m4is exact over pixel columns whose edges move to a bucket edge, at most a quarter of a pixel. A time column reduces under the:timescale, in microseconds, by default; passscale: :linearfor the former seconds.- The row helpers
Transforms.answer/4,point_reader/2andcolumn/2are gone (#23). No table operation reads rows any more: a transform reads columns, or declares itself a column-native step (#20). - Don't reuse a cache built by an earlier version. An
Operations.Cache'sm4/lttbpartials changed shape twice (#40, #50), andaggregate,sum,binandspectrumstep keys no longer carry a chunk's place (#24). Drop a held cache on upgrade and let it rebuild. - Native kernels are optional; nothing is needed to keep the Elixir path. To build them, list
{:zigler, "~> 0.16.0", runtime: false}in your deps and have Zig 0.16.0 in the build (#48). Zigler'szig_docpinsex_docto 0.39.1, so a project that listsex_docitself may needoverride: trueon it. Without Zig the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path, with the same results. Setconfig :macaw_data, native: :requiredto fail the build, and the start, without the kernels.
Changes
- Spec: the source contract is one model, a relation that varies over time (#11, #10): a delivery is a snapshot, a push the changelog of Z-set increments,
Push.held/1their integral,:as_ofa past snapshot, a derived source a map from relations to relations (spec/01 §2.5). The deferred §2.6 and §4.1–§4.4, and spec/02 §8.6, specify what #12–#15 and macaw_data#9 build: numbered increments and the snapshot's handoff, retractions and range replacements, one resolution per subscription, settled ranges, and derived watermarks. - Breaking: push increments are numbered, and a snapshot hands off to the stream by position (#12, spec/01 §4.1). A source sends
{:increment, seq, delivery}(Push.increment_message/3, numbered from 1 per subscription) and may send{:snapshot, delivery}(Push.snapshot_message/2) whoseserved.position(new) is the last increment it reflects;Push.delivery_message/2is gone.Pushdrops a repeated or older increment whole (counted inignored, a count by reason), applies increments in the order of their numbers, holds back those past a gap, records the gap and answers it fromPush.refetch/1; a snapshot, received or fetched with the request's newsubscription:and handed over byPush.snapshot/2, replaces what is held and drops the increments it already reflects. Rows are no longer deduplicated: an identical row pushed again is a second row (tally taps), so redelivery is guarded by the number, not the row. The conformance kit's:pushcheck holds increments to their numbering. Upgrade: number each subscription's increments from 1 and send them withincrement_message/3. - New: pushed increments carry corrections (#13, spec/01 §4.2–§4.3). A delivery's
retractedrows are withdrawn, each matched by identity, the window value it was held at and the fields it names, one copy per unit of weight, never a guessed row: one not held, held only at another window value, or that can't tell rows apart is{:retract, reason, row}inerrors, and against acappedor reduced holding an unheld retraction is counted inignored(:unheld). Itsreplaceranges,%{from, to, identities, rows}, make the relation over[from, to), limited to identities, exactly the rows given; a change inside a replaced range, a replacement's row outside it and an overlapping replacement are recorded and not applied. Row retractions apply first, then range retractions, then insertions. An increment is merged only into a relation held at itsserved.resolution; a mismatch is recorded and answered byPush.refetch/1. - New: settled ranges (#14, spec/01 §4.4). A delivery's
served.complete_untilis the source's watermark, a time or a number below which it expects no further change: settled, not final.Pushcarries the latest claim forward on what it holds and records a smaller one as{:complete_until, :backwards, held, given}, keeping the held claim; a new subscription starts afresh. A description may declarelateness, a hard bound (aDurationover a time window field, a number over a number one); given toPush.new/3aslateness:, it refuses a change older than the claim minus the bound ({:late, row, bound}) and a replacement reaching past it whole. Without it a change of any age is accepted. - New: derived watermarks (#15, spec/02 §8.6). A derived source's output states
served.complete_until, the minimum of its inputs' claims, each capped by its request window's upper bound; an input with no claim leaves none, and a change of the watermark alone marks the readers. A description may declareidle_after(milliseconds), which the owner passes toEngine.new/2by name (idle_after:): a source that has delivered nothing for that long, judged against thenowthe engine is given, is left out of the minimum until its next delivery, unless every input is idle. - Breaking: a table is chunked, and its identity is its contents (#17, macaw_data#9, spec/01 §1.1.1–§1.1.2; an amendment to dco-tek/macaw/macaw#188 D5). A table is an ordered list of immutable chunks of
Macaw.Data.Table.chunk_rows/0(8,192) rows, each column packed per chunk, so a column's form and a dictionary are a chunk's.append/2,trim/2andslice/3keep whole chunks and offsets into them, copying nothing, andchunks/1lists every chunk a table references.Table.t()is opaque.equal?/2compares contents strictly anddigest/1is the content digest, the kernel's digest of the canonical term{"macaw.table", [{name, type}, …], [values, …]}. The same rows chunked two ways are one value, andcompact/1rewrites any table into canonical chunks. Each chunk carries its own digest (Table.Chunk.digest/1), computed once when it is built.new/1refuses a value that is not data, a function, a PID, a reference, a port or an improper list, as{[name, i], :not_data}, since it has no digest. Upgrade: compare tables withMacaw.Data.Table.equal?/2, not==; read storage throughchunks/1, never the struct's fields; keep PIDs, references and functions out of tables. - New: chunk identity (#18, macaw_data#9, spec/01 §1.1.3). A chunk has a logical id, where it sits, and a version (
Chunk.id/1,Chunk.version/1). A table built from rows names its chunks by their content digests. A source's table, built byMacaw.Data.Table.new/2withorigin: %{source, resolution, window, span}, lays its chunks on fixed slots and names them{source, resolution, slot}: a span's start in event time, else a row offset that is a multiple ofchunk_rows/0. So ids hold across a rebuild and a restart, and two resolutions never share one.append/2keeps a source's alignment, writing rows of the last slot into its chunk again, andcompact/1keeps its slots. A description may declarechunk_span, andDescription.chunk_span/2derives one from the resolution otherwise, never from the data. A step's output chunk (Chunk.derive/3,Chunk.combine/3, gathered byMacaw.Data.Table.from_chunks/3) is named after its inputs and step key. It is versioned by a derivation digest and its contents are not hashed, so a chain of steps hashes nothing. - Breaking:
Push.held/1's rows are a chunked table (#19, macaw_data#9, spec/01 §1.1.4, §2.6). The rows are aMacaw.Data.Tableof the source's origin, its columns the schema's fields, packed asDelivery.table/2packs them, so a category's atom reads back as its string and a time column's values asDateTimes. Every change reaches the table through the newMacaw.Data.Table.apply/2. It writes an open chunk's new contents and puts a settled chunk's changes (wholly beforecomplete_until) in its delta: a deleted set, predicate deletes for replacements, and a side chunk of insertions, all applied on read. Lazy compaction at a quarter of the base keeps the chunk's id and digest and emits nothing. A cold chunk (Table.Chunk.cold/5) is never fetched for a predicate delete; its digest stays pending untilMacaw.Data.Table.warm/1.Push.new/3takeschunk_span:. Rows with no window value now sort last in a source's table, as a push holds them. Rows that don't pack are{:table, faults}inerrors. Upgrade: read held rows by name withMacaw.Data.Table.get/2, and compare a category as a string. - New: steps run a chunk at a time (#20, macaw_data#9 §5, spec/04 §1.4). A transform may declare itself a column-native step: its class (
class/1), the context it reads (reads/0), the context bound over the whole input (bind/3), the context as it resolves for a chunk (resolve/3) and the chunk's part (chunk/3);Macaw.Data.Operations.Stepruns it, andStep.parts/4lists each chunk's view and step key, the step's identity, its context as resolved for the chunk and the run it reads, for a caller's cache.filter,takeandwindow(withoutlatest) are per-row steps: each chunk is passed through, dropped, cut to a run without copying, or gathered (Table.Chunk.gather/5, every column's slots copied, its id{input id, step key}), and the merge is concatenation. None of them transposes: at 50k rowstake n: 100is about 0.01 ms (was 81 ms),filterabout 8 ms (was 93 ms).windowkeys a chunk wholly inside it withoutnow, drops one outside, and binary-searches a sorted window field (Macaw.Data.Table.sorted/1, a source's window field, kept by every per-row step) to cut only its edge chunks.Macaw.Data.Table.from_chunks/3takes views andsorted:;Chunk.values/4andChunk.at/3read a step's column.Macaw.Data.Table.from/2names its columns without merging a row map. The row helpers moved to an internal module, Macaw.Data.Table.Edge, and are still called throughMacaw.Data.Table. - New: the associative combines (#21, macaw_data#9 §5, spec/04 §1.4, §1.2, §5). A step of class
:combinedeclarescombine/3, two runs' partials merged (associative, in chunk order), andfinish/3, its table;Step.merge/5folds a table's partials andStep.output/3names a combine's output chunk over its inputs.sum,aggregateandbin(counting, or withcount,sum,mean,min,maxorextent) andwindowwithlatestare combines: any chunking and any grouping of their partials give equal tables. An aggregate may give the combining form,merge/3. Breaking (values):sumandmeansum exactly: integers stay an integer, and with a float among them the exact sum is rounded once to the nearest float (ties to even), so any order of the readings gives the same bits; a float sum may differ from the sequential sum in the last place (e.g.[1.0e16, 1.0, -1.0e16]sums to1.0, was0.0).latest's ties are broken by chunk order, then row order, as before. Aggregates without the combining form (median,quantile,distinct,variance,deviation,slope,delta) still read every reading at once; a quantile stays exact, never a sketch. - Breaking:
lttbruns MinMaxLTTB, andm4andlttbreduce on one absolute bucket ladder (#22, macaw_data#9 §5, spec/04 §4.4, §4.5, §4.5.1). Buckets are2^kwide from origin 0 in the x scale's space, named by the new context keysscale(:linear,:timein microseconds,:logaslog2 x;:timeby default over a time column) andlevel(the previous level, for a hysteresis of a factor of 2). A chunk's partial depends only on its rows, the scale's kind and the level, so a zoom within a level reuses every partial, a zoom out folds finer partials (Ladder.coarsen/2), buckets straddling a chunk edge and gaps on one merge, and the count option leaves the step key (the new optional callbackkeyed/1).lttb's tolerance: at mostnrows without gaps, the first and last kept, deterministic, the same under every chunking; its mean interpolation error, over the y extent, is at most twice the exact algorithm's plus 0.005 (measured worst 1.27× sines, 1.37× walks, 1.56× sparse spikes). The exact algorithm stays asMacaw.Data.Reduce.lttb/3.m4's tolerance: exact M4 over pixel columns whose boundaries move to a bucket edge, at most a quarter of a pixel away (half under hysteresis);Reduce.m4/3's answer where bucket and pixel edges meet. A budget's reducer (spec/04 §7) picks as the steps do. A time column reduced under the default:timescale buckets in microseconds; passscale: :linearfor the former seconds. - New: the global steps and the reshapes (#23, macaw_data#9 §5, spec/04 §1.4). A step of class
:globalgets every chunk's part in itsfinish/3:sort,stack,spectrum, andaggregateandbinwith anfnthat has no combining form (median,quantile,distinct,variance,deviation,slope,delta). A global output is one chunk named by its step key over its inputs' ids and versions, so a step after it still caches; rows kept across chunks are gathered byStep.gather/5, codes remapped.foldis a per-row reshape ({:reshape, indexes, added},Table.Chunk.reshape/7: carried columns gathered with repeated indexes, the key and value added), andstackgathers across chunks with repeated indexes.Stack.series/2stacks data as given. No table operation callsMacaw.Data.Table.rows/1orOperations.table/2any more; the row helpersTransforms.answer/4,point_reader/2andcolumn/2are gone. - New: the caching contract (#24, macaw_data#9 §6, spec/04 §1.5, spec/01 §1.1.4).
Macaw.Data.Operations.Cacheis the example caller: a plain value the caller holds, keyed by{Chunk.id/1, Chunk.version/1, step key}.Cache.run/5reuses a held part, updates a changed chunk's part by inverse from its base's part where exact (Step.update/4, a step's new optionalupdate/4and an aggregate'sretract/3: counts,bincounts and integer sums), computes any other, and merges; its result equalsStep.run/4's.Cache.prune/2evicts by logical id. A lookup hashes nothing. A chunk with a delta keeps its base's version (Chunk.base_version/1) and a read view built once per state, so reads through a delta (Chunk.values/4,Chunk.at/3, a gather,window's binary search) no longer apply the delta on every read, and compaction makes the view the base without repacking. Fixed:aggregate,sum,binandspectrumno longer put the chunk's place (:at,:rows) in their step keys, so a chunk's key holds while other chunks come and go. - New: parallel chunks on request (#25, macaw_data#9 §7, spec/04 §1.6, §3.2).
parallel: truein the context asksStep.run/4(so every table operation throughOperations.apply/4) andCache.run/5to work a step's chunks across the schedulers, one task a part, when the input holds at least two chunks andStep.parallel_rows/0(16,384) rows; otherwise, and by default, they run inline. The parts reach the merge in chunk order whatever worked them, so a parallel run is bit-identical to the inline one, floating-point sums included.parallelis no step's declared context, so it never enters a step key. - Measurement: a synthetic frame loop at 50k rows (#26, macaw_data#9 item 7, spec/04 §1.5): a live window with a chunk's worth of new readings a frame, running
take,filter,window→m4,window→lttbandsumby series, recomputed in full and per chunk throughOperations.Cache. The suite states the per-chunk cost in reductions (its changed chunks at most a full run over one chunk's readings, its frame at most half the full recompute);mix test --only benchmarkprints the wall clock. Measured then, before #39–#54: about 400 ms a frame in full, 190 ms per chunk, of which about 95 ms is them4andlttbmerges. Results and the native-kernel decision are on macaw_data#9; #54 below gives the release's figures. - Faster:
m4andlttbmerge their partials in work proportional to the chunks, not the buckets (#39, #40, spec/04 §4.5.1). A partial holds a piece per view, each run's buckets ascending, soLadder.combine/2concatenates and reads no bucket; the chunk edges are joined once when the merged partial is read (Ladder.runs/1), andfinish/3folds the buckets into pixel columns (m4) or takes its candidates in input order (lttb) in one pass. Results are unchanged under every chunking and grouping. In #26's frame loop the merges alone went from about 100 ms to about 40 ms a frame, and the per-chunk frame from about 215 ms to about 125 ms. A partial's term changed shape (Ladder.partial/0), so anOperations.Cachebuilt by an earlier version is not to be reused. - Faster: reading a column costs its slots, not its bits (#41, #42, spec/01 §1.1.5). An all-valid column decodes without expanding its validity bitmap; a slice cuts a bitmap as one little-endian integer; a gather takes an all-set bitmap without reading a bit; and
Chunk.gather/5andreshape/7gather a run from the chunk's storage at its offset instead of slicing every column first. Bitmaps live in an internal module, Macaw.Data.Table.Bitmap. Results and digests unchanged. - Faster: a chunk's
m4/lttbpartial is one pass with nothing allocated per row but new extremes (#41, #43, spec/04 §4.5.1). The bucket width is read once a partial, an integer x buckets by a shift (s >>> k,s <<< -k), a number x on a linear or time scale is read without a call, and the open bucket's first, last, least and greatest ride as loop arguments. About half the reductions a row; partials unchanged. - Faster: grouping, exact sums and
filterwithout per-row overhead (#41, #44, spec/04 §4.1, §4.10, §5). A chunk's groups are found in one pass, one map update a row, and grouping by one category column reads its dictionary codes and names each group once.Exactsums floats' signed mantissas by exponent and counts them in units of 2^-1074 once, when the value is read.filterscans a packed number column's buffer for an ordering against a number; its passing rows, when they are one run, are now cut from the chunk without copying rather than gathered (spec/04 §4.1), so such an output chunk is the input's, read from a row. Values unchanged. - Measurement: the frame loop's pure-Elixir cost is pinned (#41, #45, spec/04 §1.5): a per-chunk frame at most 1,100,000 reductions (1,982,948 before #41) and its merges alone 600,000. spec/04 §1.5 states that a frame's garbage collection is the caller's to size: at the default heap a frame's wall time is mostly collecting the cache's partials, and the benchmark now also reports frames in a process whose minimum heap holds the cache (#46 takes up keeping partials off the heap).
New: optional native kernels, built and loaded (#48, #38, macaw_data#9 §8, spec/04 §1.7).
Macaw.Data.Nativecompiles Zig kernels through Zigler 0.16.0 when the build has Zig 0.16.0 (.tool-versions), found throughzig env; Zigler is an optional dependency, so an adopter that wants the kernels lists{:zigler, "~> 0.16.0", runtime: false}. Without the toolchain the build prints a warning, not a compiler diagnostic, and every step runs its Elixir path.Native.loaded?/0is decided once at the application's start (the new application module, Macaw.Data.Application) and held in:persistent_term.config :macaw_data, native: :optional | :required, default:optional::requiredfails the build without the toolchain, and the start when the kernels don't load. A call over at mostNative.sync_rows/0rows (8,192, #49) runs on a normal scheduler, a larger one on a dirty CPU scheduler. Kernels are built ReleaseSafe, so a panic is never silent. CI tests with Zig under:required, without Zig, and checks that:requiredwithout Zig fails the build.ex_docis now a dev and test dependency, pinned by Zigler'szig_docto 0.39.1. #48 itself gives no step a kernel; #49–#53 below do. A kernel never replaces its step's Elixir path, which stays the reference, the path for every form a kernel hands back, and the whole of a build without Zig.- Faster: a chunk's
m4/lttbladder partial runs in a native kernel where the kernels are loaded (#49, #38, spec/04 §4.5.2).Ladder.partial/4hands the view to it when x is int64, float64, a UTC, zoned or naive time or a date, and y is int64 or float64, on a linear or time scale; the kernel reads the packed buffers and validity bitmaps in place and answers the Elixir walk's piece term for term (Ladder.partial_elixir/4, the reference). Where the walk would answer past 64 bits or raise, it hands the view back.Native.sync_rows/0is 8,192, one canonical chunk: a call's time follows its answer's terms, at worst one bucket a row. Reductions count no kernel's work, soNative.elixir/1runs a function on the Elixir path (Native.active?/0false inside it), and the frame loop's reduction statements and pins are measured there. - Faster: a ladder partial's piece is one packed binary, off the process heap (#50, #38, #46, spec/04 §4.5.3). The Elixir walk packs its segments (
Ladder.Packed: a header with the points' number types, a fixed-width head a segment, 88 bytes a bucket; the external term format for a piece of mixed or past-int64 values), and the native kernel writes the same bytes rather than building a term a bucket. The frame loop's cache went from about 824,000 words on the caller's heap to about 11,000, so a collection no longer copies it; a test holds it under 50,000.combine/2still only concatenates;runs/1reads each piece once. Results unchanged. A partial's shape changed again (Ladder.piece/0is{view, binary}), so anOperations.Cachebuilt by an earlier version is not to be reused. - Faster:
m4's andlttb's finishes run in native kernels where the kernels are loaded (#51, #38, spec/04 §4.5.4).m4_finishandlttb_finishread the merged partial's packed pieces in place, join them asLadder.runs/1does, and answer the refs to gather:m4's pixel columns, andlttb's candidates throughReduce.lttb/3. Their arithmetic is Erlang's, operation for operation (exact integers, an integer converted to the nearest float where it meets one, strict floats), so the refs are the Elixir path's (M4.refs/3,Lttb.refs/2, held equal by a property test). A piece kept as a term, a value past int64 or anything Erlang would raise on is handed back to the Elixir path. A finish over at mostNative.finish_buckets/0(2,048) buckets runs on a normal scheduler, a larger one dirty. - Faster: the exact sum by group runs in a native kernel where the kernels are loaded (#52, #38, spec/04 §5.1).
sum, andaggregatewithsum,meanorcount, over an int64 or float64 column grouped by one dictionary column or not at all, hand a chunk's view togroup_sum, which reads the values, the codes and both bitmaps in place and answers each group's code, first row, rows, readings and exact-sum state (the internal Macaw.Data.Exact) (integers exact in 128 bits, past int64 as a bignum; floats' mantissas summed by biased exponent), so the part is the Elixir fold's term for term (held equal by a property test). A code past the dictionary or a non-finite float is handed back. A call answering more thanNative.sync_entries/0(2,048) groups and exponent sums answers:dirtybefore building a term and runs on a dirty scheduler. Sum by series over an 8,192-row chunk: 127,254 → 173 reductions, ~2.3 ms → ~80 µs. kernels.zig is split by kernel family (buffers, ladder, finish, group). - Faster:
filter's test and the gather of the rows it keeps run in native kernels where the kernels are loaded (#53, #38, spec/04 §4.1.1).filter_maskreads the tested column in place and answers the selection itself:present/absentfrom any column's validity bitmap, an int64 or float64 column against a number compared exactly as Erlang compares (an integer against a float past 2^53 too), a dictionary of strings by each string's answer decided once with the Elixir path's own test.Column.gather/2(every gather and reshape) copies fixed-width slots (take) and bitmap bits (take_bits) natively, the same bytes. Held equal to the Elixir path by property tests over every form, test and kind of value.v > 0over an 8,192-row chunk with rows apart: the mask 62,275 → 30 reductions (~470 → ~75 µs), the whole step 112,003 → 6,507 reductions (~1.2 → ~0.5 ms). - Measurement: #26's frame loop with every native kernel (#54, #38, #9, spec/04 §1.5.1). Per chunk 958,680 → ~181,000 reductions and 36–47 → 7–10 ms at the default heap (13–16 → 6–8 ms with the frame heap sized); recomputed in full 1,931,845 → ~186,000 reductions, 55–58 → 9–11 ms. With the kernels the frame meets visualize's ~17 ms budget at the default heap, per chunk and in full. A new default-suite pin holds the frame with the kernels to at most 250,000 reductions per chunk, merges alone and in full.
- New:
Macaw.Data.Native.normal_schedulers/1(#60, #61, spec/04 §1.7). It runs a function with every kernel call in its process on a normal scheduler, whatever the call's size. A call on a dirty scheduler is charged reductions by its time, about four a microsecond, so a count with one in it moves with the host's load. A normal-scheduler call is charged a fixed few. Use it for a statement of work in reductions with the kernels, such as the frame's (spec/04 §1.5.1), never for serving: a large call then holds its scheduler past the 1 ms rule.scheduler/1,finish_scheduler/1and the newentry_limit/0answer for it. - Fixed: a bounded call is hung only when it is blocked (#33, #62, #67, spec/02 §6.4, spec/01 §9). The conformance kit's runner and the derived engine's sandbox called any call still running at
timeouthung, so on a loaded host a slow compile or a capped CPU-bound call was misjudged{:hung, …}. Now a call with a reductions cap is judged by its count while it works. It is hung only once it has been waiting inreceive, its reductions unchanged, fortimeoutms on end, and each piece of work gives it the whole bound again. A call with no cap still has only the clock. A wait on another process is a block all the same, so give a stored function a generoustimeout:, as spec/02 §6.4 asks. - Fixed: the first stored function no longer waits on the code server (#69, spec/02 §6.3). In interactive mode a node's first compile loads Elixir's own modules on first use, and a busy code server could hold a bounded compile past its timeout. The application now compiles and runs one trivial function through the sandbox as it starts, so those modules are loaded before any bounded compile. Starting the application costs that one compile.
- Docs: spec/00 names the dependencies as they are, with no pinned version:
macaw_layersrequired, andtable,nxandzigleroptional (#34, #70). Every doc reference resolves, and CI builds the docs with--warnings-as-errors. - Tests: the suite is warnings-clean and stable under load (#37, #55, #56, #58, #63, #64, #65): call traces run in sessions of their own, test files compile with
--warnings-as-errors, and no test's verdict depends on the host's load or on what other tests run beside it. Fifty loaded full-suite runs, with and without Zig, confirm it.
v0.7.0 (2026-10-30)
Terms are the form (dco-tek/macaw/macaw#188), macaw_data's part. A value's identity and validity are the BEAM term's own; JSON and Arrow are adapters at the edge. On macaw_layers 0.3.0. This release is breaking throughout; the guide below is what an adopter of 0.6.1 changes, and the entries after it are the detail.
Upgrading from 0.6.1
- Depend on macaw_layers
~> 0.3.0(#27); its own upgrade guide applies to any kernel call you make directly. - Every schema version changes (D1). A stored version, cache key or golden written by 0.6.1 differs; regenerate it. A schema's names, states and roles are now strings in the struct (
Schema.new/1normalizes them), so code that pattern-matches:latencyinside a schema matches"latency"; every function that takes a name still takes either spelling. - A data color is a color term (D2): a
%Macaw.Layers.Color{}literal or a typed color, never a string. Parse CSS text at your input edge withMacaw.Layers.Color.parse!/1, or write~RGB"#3b6fa8"in code. A role spelled like a CSS name ("orange") is now a valid status role. - A derived source is built from its value (D3). Pass
Derived.new/1a keyword list or a map of terms; read a stored form withDerived.from_map/2orfrom_json/2.to_map/1andto_json/1now answer{:ok, form}or faults. - The operations registry's digest changes (D4); a compile cache keyed by it recompiles once.
Operations.step_digest/2is new: a checked step's canonical identity. - Category values read back as strings (D5): a
:categorycolumn holding:okanswers"ok"fromrows/1,get/2andcolumn/2; compare with strings, or withOperations.same?/2.nil,trueandfalseare kept. - Packed columns are faithful and Arrow-compatible (D5). A mixed integer/float column and a zoned time read back exactly as given; anything that held packed buffers (stored binaries) changes with the bitmap order.
Changes
- Breaking: on macaw_layers
~> 0.3.0(#27): the kernel's digest is over the value, so every pinned version moved before D1 changed what it digests. - Breaking: every schema version changes (D1, #28, dco-tek/macaw/macaw#188).
Schema.versionis the kernel's digest of the schema's term, not of its JSON form, which is now only an adapter at the edge; a value entry's color no longer goes through the JSON form whenSchema.new/1versions the schema, so a typed color there can't crash it. The struct holds every field name, state and role as its string, normalized once bynew/1, sofields,identity/1,window_field/1,compatibility/2's faults andidentity/2's maps name fields by their strings; every function that takes a name still takes either spelling. A state named in both spellings in one description is:duplicate. - Breaking: data colors are color terms, and a string is never a color (D2, #29, dco-tek/macaw/macaw#187, dco-tek/macaw/macaw#188). A value entry's
color, a:colorfield's row values (a derived source's output), a:colorparameter andSchema.color?/1take a literal%Macaw.Layers.Color{}or a theme-relative typed color ({:color, "good"},{:alpha, {:color, "bad"}, 0.74}), its names in a theme's name grammar; a string there,"#2e7d32"or"red", is{:type, :color}.new/1holds a typed color's names as strings, so{:color, :good}and{:color, "good"}are one version. The schema's JSON form writes and reads an entry's color in macaw_layers' color form (Macaw.Layers.Color.Form:{"$rgb": "#2e7d32"},{"$color": "good"}), making no atom.Macaw.Data.Table.column_types/1infers:colorfor a column of color terms; a column of strings stays:text. A status role spelled like a CSS name (:orange) is now a role. Upgrade: read CSS text into a color at your input edge withMacaw.Layers.Color.parse!/1, or write~RGB"#2e7d32"in source (import Macaw.Layers.Color, only: [sigil_RGB: 2]); a stored schema whose entry colors are strings reads back refused, so convert its colors withparse!/1before reading it. - Breaking: a derived source is built from its value, and its forms are an adapter (D3, #30, dco-tek/macaw/macaw#188).
Derived.new/1takes the value only, a keyword list or a map of its fields as terms; it no longer guesses whether it was given a map form, and a map form's string keys are each:unknown_key. The map and JSON forms are written and read by macaw_layers' JSON adapter (Macaw.Layers.Form), with the kernel's tags and the source's own in one tag table:to_map/1andto_json/1answer{:ok, form}or every value without a form, by path;from_map/2andfrom_json/2read through the gates (the limits, the version, the tag table, the keys, making no atom) and then judge the value withnew/1. The output schema is used as built, a%Schema{}as given and fields bySchema.new/1, never written to its form and read back; in the form it isSchema.to_map/1's.Derived.Form's private tag scheme is gone. A color, literal or typed, in a schema's value entry reaches a stored function's context as the term it is; before, its tags were respelled as strings. Upgrade: the form changed: an input is{"$input": [name, window]}, a source's name{"$source": name},per{"$atom": "identity"}or{"$identity": input}, a region an object keyed by field names, andto_json/1stamps"$version": 1. Nothing in this library stored the old form of pair lists, so no version step reads it: write a stored source again withto_map/1from its value. Read a map form withfrom_map/2, notnew/1. - Breaking: the operations registry's
digest/1is the kernel's term digest of its names and modules, not a SHA-256 ofterm_to_binary(D4, #31, dco-tek/macaw/macaw#188), so every registry digest changes and a compile cache keyed by one recompiles once. New:Operations.step_digest/2, a checked step's canonical identity, for caches of a step's results. - Breaking: a table's packed columns are faithful and laid out as Arrow lays them out (D5, #16, dco-tek/macaw/macaw#188), spec/01 §1.1. Packing never changes a value. A category column of strings and atoms is one Arrow dictionary of strings, so
rows/1,get/2andcolumn/2read a category's atom back as the string that spells it (:okand"ok"are one code, read as"ok"), in every form the column takes;nil,trueandfalsestay themselves, beside the names in side bitmaps, so[:ok, true, "warn"]reads back["ok", true, "warn"]. Validity bitmaps are least-significant-bit first, Arrow's order, not Elixir's<<b::1>>order, so every stored packed binary changes. ADateTimekeeps its zone (name, abbreviation and both offsets, in a per-column dictionary, with no time-zone database) and its precision, so a time reads back as given rather than in UTC at microsecond precision. A text column is an offsets buffer and one data buffer, Arrowutf8when every value is valid UTF-8 and Arrowbinarywhen one is not; a category column holding a string that is not valid UTF-8 is kept as its list.new/1takesNaiveDateTimes andDates in a time column as themselves, and refuses a value in a:colorcolumn that is not a color term (Schema.color?/1),{[name, i], {:type, :color}}. Upgrade: compare a category value withOperations.same?/2, or as a string. - Fixed: a number column mixing integers and floats reads back each as given, through an integral bitmap, or a dense union of int64 and float64 when an integer is beyond ±2^53; an integer beyond int64 keeps its column a list rather than becoming a float. A time column keeps each value's zone and precision, so
~U[2026-01-01 00:00:00Z]and~U[2026-01-01 00:00:00.000000Z]stay distinct (#16, dco-tek/macaw/macaw#188).
v0.6.1 (2026-10-29)
- Fixed: compiling macaw_data in an app without the optional
tablepackage warned that Table.Reader.impl_for/1 is undefined; CI now compiles it without its optional dependencies (#7).
v0.6.0 (2026-10-29)
- What a source steps its window by:
Window.before/4,region/2(visualize'sbox/2: a window as the region it asks of a schema),range/2, andcheck/2of the axes alone as well as of a request (dco-tek/macaw/macaw#185, #5). - Packing a table and reading it back do about half the work: types inferred only where not given, values checked in one pass, column names merged once, rows built a column step at a time, and time columns converted by the date's fields, a fifth of
DateTime's own conversion (dco-tek/macaw/macaw#185, #4). - Breaking: a delivery keeps the rows its source gave (dco-tek/macaw/macaw#185).
%Delivery{rows, names, types, served, schema, origin}: nothing is packed or rewritten when it is made;Delivery.rows/1andMacaw.Data.Table.rows/1of a delivery read the rows as they are (a row list uncopied, tuple rows unchanged);Delivery.table/2andtable!/2pack them for a reader that wants columns;named_rows/1names tuple rows and tensors bynames:;empty?/1. Thetablefield is gone: readDelivery.table!/1. Finishing and push keep the rows' form.Macaw.Data.Table.unnamed?/1(#3).
v0.5.4 (2026-10-28)
- The macaw logo in the docs and the README (macaw#175).
- Tests check "makes no atom" on the input's own names, never the node's atom count, and the atom gate forbids the count in
test/(#1).
v0.5.3 (2026-10-28)
- On Hex: depends on
macaw_layers ~> 0.2.2from Hex instead of its git tag, and the README installs from Hex (macaw#155). - The README opens with the early-release notice (macaw#174).
v0.5.2 (2026-10-28)
- Dependencies named by their
dco-tek/macawgroup paths (macaw#152). - Package metadata for Hex: the public home, the changelog link, the files shipped, and docs with
ex_doc(macaw_compose#155).
v0.5.1 (2026-10-28)
- macaw_data v0.5.1: on macaw_layers v0.2.0 (macaw_compose#145)
v0.5.0 (2026-10-27)
- macaw_data v0.5.0: transforms, table operations and aggregates (macaw_compose#83): the behaviors, the registry, twelve table operations, thirteen statistics, Macaw.Data.Ops in the sandbox, budget reducers by name
v0.4.0 (2026-10-27)
- macaw_data v0.4.0: derived sources (macaw_compose#82): the model, the sandbox and its gates, host functions and the engine
v0.3.0 (2026-10-27)
- macaw_data v0.3.0: the source contract (macaw_compose#58): the behavior, description, request and parameters; windows and finishing; push; the conformance kit
v0.2.0 (2026-10-27)
- macaw_data v0.2.0: the data model (macaw_compose#57): units and quantities, the columnar table, the schema, the delivery
v0.1.0 (2026-10-27)
- macaw_data 0.1.0: Macaw.Data.Event