Retain-time atomic fact extraction — the keystone that gives reflect something structured to aggregate over.
Faithful port of Hindsight's engine/retain/fact_extraction.py: each incoming
retain item (a chat session, document, or paragraph) is decomposed by an LLM
into a LIST of atomic facts, one input to N discrete memory units. Whole-session
blobs cannot be aggregated across sessions (the buried "took course X" / "spent
€40" facts never individuate, so cross-session counts and temporal ordering are
unanswerable); atomizing them restores the material that recall ranks,
consolidation merges into observations, and reflect sums.
Each produced fact-item INHERITS the parent's provenance and — critically — its
mentioned_at (the session date), so every atomic fact carries the timestamp
reflect's per-entry temporal arithmetic depends on. Extraction OVERRIDES only
content (the assembled fact text), type (world/experience), and the
occurred_* dates for datable events.
Extraction is gated: it runs only when a mode is configured AND an extraction-capable LLM is available. It degrades to passthrough (the original item, i.e. today's blob behaviour) on any LLM error, timeout, or empty result — extraction never fails a retain. This mirrors the LlmCore isolation the rest of the pipeline relies on.
Summary
Types
Per-call accounting of how extraction degraded. fallback_items counts items stored
as their raw text because extraction ERRORED (LLM error, timeout, or a response that
was not a facts list); empty_items counts items stored raw because extraction ran
and found no facts. Both are 0 when extraction is disabled.
Functions
True when mode and llm would drive real extraction (used by callers to log/branch).
Like maybe_extract/2, returning {items, stats}. Retain surfaces the stats on its
result so a caller can tell a degraded retain from a clean one without attaching to
the [:foresight, :retain, :extraction_fallback] telemetry event.
Expand items into atomic fact-items, or return them unchanged (passthrough).
Types
@type stats() :: %{ enabled: boolean(), mode: String.t() | nil, fallback_items: non_neg_integer(), empty_items: non_neg_integer() }
Per-call accounting of how extraction degraded. fallback_items counts items stored
as their raw text because extraction ERRORED (LLM error, timeout, or a response that
was not a facts list); empty_items counts items stored raw because extraction ran
and found no facts. Both are 0 when extraction is disabled.
Functions
True when mode and llm would drive real extraction (used by callers to log/branch).
Like maybe_extract/2, returning {items, stats}. Retain surfaces the stats on its
result so a caller can tell a degraded retain from a clean one without attaching to
the [:foresight, :retain, :extraction_fallback] telemetry event.
Expand items into atomic fact-items, or return them unchanged (passthrough).
Opts:
:mode— "concise" | "verbatim" | "custom" | nil/"none" (nil/"none" disables):llm— theForesight.LLMimplementation module (needscomplete_json_schema/4):mission— optional extraction mission steering what is worth keeping:custom_instructions— guidelines body for"custom"mode:provider/:provider_opts/:sleep_ms— passed through to the LLM