Batteries-included text handling for pipeline persistence: scrubbing text so Postgres will accept it, whitespace normalization, and head+tail truncation for oversized artifact bodies.
scrub/1 / scrub_strings/1 are the single implementation
These were duplicated in the host's core app for Phases 1–6
(that app could not yet name ALLM.Pipeline.*), guarded against
drift by a host-side parity test. Phase 7.2 (2026-08-24) converged them:
the host's core app took its own :allm_pipeline dependency (7.1 — an
in-umbrella dep then; the Phase 8 path dep now),
the host copy and the parity test were deleted, and every scrub call
site re-points here. This module is now the only copy,
and no code path calls back into a host module.
See steering/2026-08-10_ALLM_PIPELINE_EXTRACTION.md §3.9 (the copy→converge
sequence) and steering/2026-08-24_ALLM_PIPELINE_PHASE_7.md §7.2.
Summary
Types
Head+tail truncation window, in bytes. Text longer than twice this is reduced
to the first and last window bytes with an elision marker between.
Functions
The default head+tail truncation window, in bytes.
Collapses internal whitespace runs to a single space and trims both ends.
Remove NUL bytes and invalid UTF-8 from text.
Scrub every binary value in a (shallow) map, leaving non-binary values (integers, atoms, nested structs, etc.) untouched.
Head+tail truncate text to roughly 2 * window bytes, elided in the middle.
Types
@type window() :: pos_integer()
Head+tail truncation window, in bytes. Text longer than twice this is reduced
to the first and last window bytes with an elision marker between.
Functions
@spec default_window() :: window()
The default head+tail truncation window, in bytes.
Collapses internal whitespace runs to a single space and trims both ends.
The canonical cleanup for HTML-extracted text, which arrives with the source document's line breaks and indentation embedded.
iex> ALLM.Pipeline.Text.normalize(" Board Meeting\n ")
"Board Meeting"Remove NUL bytes and invalid UTF-8 from text.
Document extraction and external transcript feeds occasionally carry bytes
that PostgreSQL's text/jsonb columns reject, surfacing as
ERROR 22P05 (untranslatable_character) on insert. Two distinct problems are
handled here:
NUL bytes (
U+0000). PostgreSQL cannot store a NUL in a text column, yet<<0>>is a perfectly valid UTF-8 codepoint, soString.valid?/1reportstrueand won't catch it. These are dropped explicitly.Invalid UTF-8 byte sequences. Mojibake / truncated multibyte sequences from scanned-PDF OCR. These bytes are dropped, preserving every well-formed codepoint around them.
Passes nil through unchanged so callers can sanitize optional fields without
a guard. The common case (already-valid text with no NUL bytes) returns the
original binary without allocating a new one.
Scrub every binary value in a (shallow) map, leaving non-binary values (integers, atoms, nested structs, etc.) untouched.
Useful for sanitizing a changeset attrs map before insert without naming each string field individually.
Head+tail truncate text to roughly 2 * window bytes, elided in the middle.
Used when an LLM prompt or response is too large to store whole in an artifact body. Both ends are kept because the interesting parts of an oversized prompt are its instructions (head) and its most recent context (tail); a plain prefix truncation discards the latter.
Text at or under 2 * window bytes is returned unchanged, as is nil, so
callers can pipe optional fields through without a guard. window counts
BYTES, not graphemes — the point is fitting a storage limit — so a cut can
land mid-codepoint and the result is not guaranteed to be valid UTF-8. Run it
through scrub/1 if the result is bound for a Postgres text column.
iex> ALLM.Pipeline.Text.truncate("abcdefghij", 2)
"ab\n…[truncated]…\nij"
iex> ALLM.Pipeline.Text.truncate("abcd", 2)
"abcd"