Retain-time atomic fact extraction — the keystone that gives reflect something structured to aggregate over.
Faithful port of Hindsight's engine/retain/fact_extraction.py: each incoming
retain item (a chat session, document, or paragraph) is decomposed by an LLM
into a LIST of atomic facts, one input to N discrete memory units. Whole-session
blobs cannot be aggregated across sessions (the buried "took course X" / "spent
€40" facts never individuate, so cross-session counts and temporal ordering are
unanswerable); atomizing them restores the material that recall ranks,
consolidation merges into observations, and reflect sums.
Each produced fact-item INHERITS the parent's provenance and — critically — its
mentioned_at (the session date), so every atomic fact carries the timestamp
reflect's per-entry temporal arithmetic depends on. Extraction OVERRIDES only
content (the assembled fact text), type (world/experience), and the
occurred_* dates for datable events.
Extraction is gated: it runs only when a mode is configured AND an extraction-capable LLM is available. It degrades to passthrough (the original item, i.e. today's blob behaviour) on any LLM error, timeout, or empty result — extraction never fails a retain. This mirrors the LlmCore isolation the rest of the pipeline relies on.
Summary
Types
Per-call accounting of how extraction degraded. fallback_items counts items stored
as their raw text because extraction ERRORED (LLM error, timeout, or a response that
was not a facts list); empty_items counts items stored raw because extraction ran
and found no facts. Both are 0 when extraction is disabled.
Functions
True when mode and llm would drive real extraction (used by callers to log/branch).
Like maybe_extract/2, returning {items, stats}. Retain surfaces the stats on its
result so a caller can tell a degraded retain from a clean one without attaching to
the [:foresight, :retain, :extraction_fallback] telemetry event.
Expand items into atomic fact-items, or return them unchanged (passthrough).
Types
@type stats() :: %{ enabled: boolean(), mode: String.t() | nil, fallback_items: non_neg_integer(), empty_items: non_neg_integer() }
Per-call accounting of how extraction degraded. fallback_items counts items stored
as their raw text because extraction ERRORED (LLM error, timeout, or a response that
was not a facts list); empty_items counts items stored raw because extraction ran
and found no facts. Both are 0 when extraction is disabled.
Functions
True when mode and llm would drive real extraction (used by callers to log/branch).
Like maybe_extract/2, returning {items, stats}. Retain surfaces the stats on its
result so a caller can tell a degraded retain from a clean one without attaching to
the [:foresight, :retain, :extraction_fallback] telemetry event.
Expand items into atomic fact-items, or return them unchanged (passthrough).
Opts:
:mode— "concise" | "verbatim" | "custom" | nil/"none" (nil/"none" disables):llm— theForesight.LLMimplementation module (needscomplete_json_schema/4):mission— optional extraction mission steering what is worth keeping:custom_instructions— guidelines body for"custom"mode:chunk_size— characters per extraction chunk (default 3000):deadline_ms— wall-clock budget for the whole call. Once it has elapsed, every item (and every chunk) not yet started is stored as-is and counted infallback_items, with the fallback telemetry reason:deadline. A chunk already in flight finishes (or hits the LLM timeout). nil (default) is no deadline.:max_concurrency— chunks of one item extracted concurrently, in order (default 1, i.e. sequential). Wall time then scales with the slowest chunk rather than the sum. Usage recorded by the LLM inside the concurrent calls is folded back into the caller's ledger.:provider/:provider_opts/:sleep_ms— passed through to the LLM