Foresight.Text.Sanitize (Foresight v0.2.1)

Copy Markdown View Source

Scrubs model output of characters that break downstream systems.

Removes ASCII control characters other than tab, newline and carriage return (they break JSON decoding and Postgres's UTF-8 encoding), the DEL character, and any bytes that are not valid UTF-8. Applied to every text the retain extractor and the consolidation merge take from a model before it is stored. Port of Hindsight's sanitize_llm_output (engine/llm_wrapper.py).

Summary

Functions

llm_output(text)

@spec llm_output(String.t() | nil) :: String.t() | nil