Consolidation persistence: an Elixir port of Hindsight's
engine/consolidation/consolidator.py. Turns raw world/experience facts
into consolidated observation memory units via an LLM creates/updates/
deletes merge contract (Foresight.Prompts.Consolidation.V1).
The LLM call and embedding happen OUTSIDE the DB transaction (see
prepare_scope/7); only the resulting writes run inside it (apply_prepared!/4)
— mirroring the retain path's txn discipline and avoiding the reflect
txn-across-LLM pool-timeout class of bug.
Summary
Functions
Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's
source facts consolidated_at, and return the observations written
(creates + updates) for downstream mental-model refresh.
Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL
AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending
filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the
bisection path is never retried forever; without the failed-at guard the stub
reprocessed every fact on every run).
Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).
How many times to retry a failing LLM merge for a batch BEFORE the orchestration
bisects it. Mirror of Hindsight consolidation_max_attempts (default 3,
consolidator.py:1286). Bank override wins, else engine config, else 3.
Pool the existing observations that belong to scope_key so the LLM can merge
against them. Strict scope match (exact tags + observation_scopes) prevents
cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.
Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.
Resolve the source facts backing a set of pooled observations so their text +
dates can be serialized into the merge prompt (B3 / Hindsight
_build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.
Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's
_build_observations_for_llm (consolidator.py:1190-1226): each observation
carries id, text, proof_count, its temporal range, mentioned_at, and a
source_memories array (the supporting facts' text + dates, resolved from
source_memory_ids via source_facts_by_id). Temporal fields and
source_memories are only emitted when present so the prompt stays lean.
Stamp consolidated_at = now on the given source facts so they are not reprocessed.
Quarantine facts that a size-1 LLM merge could not process. Stamps
consolidation_failed_at = now, excluding them from the pending query
thereafter (Hindsight consolidator.py:509-513). Only reached after bisection
has isolated the poison fact, so the rest of a batch still consolidates.
Functions
Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's
source facts consolidated_at, and return the observations written
(creates + updates) for downstream mental-model refresh.
Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL
AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending
filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the
bisection path is never retried forever; without the failed-at guard the stub
reprocessed every fact on every run).
Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).
How many times to retry a failing LLM merge for a batch BEFORE the orchestration
bisects it. Mirror of Hindsight consolidation_max_attempts (default 3,
consolidator.py:1286). Bank override wins, else engine config, else 3.
Pool the existing observations that belong to scope_key so the LLM can merge
against them. Strict scope match (exact tags + observation_scopes) prevents
cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.
Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.
{:ok, prepared}— the LLM call succeeded (even with empty ops); source facts for this scope will be stampedconsolidated_at.{:skip, reason}— the LLM/provider was unavailable; the scope is left untouched and its facts stay pending (retryable). Degrade-safe: never raises.
Resolve the source facts backing a set of pooled observations so their text +
dates can be serialized into the merge prompt (B3 / Hindsight
_build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.
Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's
_build_observations_for_llm (consolidator.py:1190-1226): each observation
carries id, text, proof_count, its temporal range, mentioned_at, and a
source_memories array (the supporting facts' text + dates, resolved from
source_memory_ids via source_facts_by_id). Temporal fields and
source_memories are only emitted when present so the prompt stays lean.
Stamp consolidated_at = now on the given source facts so they are not reprocessed.
Quarantine facts that a size-1 LLM merge could not process. Stamps
consolidation_failed_at = now, excluding them from the pending query
thereafter (Hindsight consolidator.py:509-513). Only reached after bisection
has isolated the poison fact, so the rest of a batch still consolidates.