Builds what mix temper.push sends from history lines: batches for a
Temper.Push.Adapter.
Pure: the task reads the files and gives the lines here. Lines are
trimmed, blank ones dropped and byte-identical copies (overlapping
cache restores) sent once. Lines of runs the destination already accepted
are left out, unless all: true. Corrupt lines (unreadable JSON such as
truncated cache tails, or JSON that is not an object) are dropped and
counted; objects without a run_id (a future kind) are kept.
With scrub_messages: true, the message of every failed test line's
failure is removed before anything leaves CI. The failure kind and
hash stay, so failure modes still group. Only those lines are
re-encoded; every other line goes out byte for byte.
The lines are split into batches of whole runs, each at most
max_batch_bytes uncompressed (32 MB by default), and each batch is
gzip compressed. A run is never split (a receiver applies a run once,
so a second half would be dropped): a run larger than the limit is not
sent, and is listed in :oversized.
A line that cannot be read is dropped, and the rest of its run is sent without it. History files are append-only, so the line cannot come back; holding the run back would lose all of it.
Summary
Types
@type t() :: %{ batches: [Temper.Push.Adapter.batch()], lines: non_neg_integer(), runs: non_neg_integer(), acknowledged: non_neg_integer(), corrupt: non_neg_integer(), oversized: [String.t()], history_run_ids: MapSet.t(String.t()) }
Functions
Builds the batches for lines, leaving out the runs in acknowledged
(a MapSet of run ids).
Options: :all (send acknowledged runs too), :scrub_messages,
:max_batch_bytes. :acknowledged and :corrupt count the lines left
out for each reason; :lines and :runs describe what is in the
batches; :oversized names the runs too large to send, and
:history_run_ids every run in lines, sent or not.