Consolidation persistence: an Elixir port of Hindsight's
engine/consolidation/consolidator.py. Turns raw world/experience facts
into consolidated observation memory units via an LLM creates/updates/
deletes merge contract (Foresight.Prompts.Consolidation.V1).
The LLM call and embedding happen OUTSIDE the DB transaction (see
prepare_scope/7); only the resulting writes run inside it (apply_prepared!/4)
— mirroring the retain path's txn discipline and avoiding the reflect
txn-across-LLM pool-timeout class of bug.
Summary
Functions
Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's
source facts consolidated_at, and return the observations written
(creates + updates) for downstream mental-model refresh.
Hindsight's observation capacity note (consolidator.py:1520-1536): when a scope is
at or near max_observations_per_scope, tell the merge model so it prefers UPDATE
over CREATE. nil when no cap is configured or the scope has ample room.
Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL
AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending
filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the
bisection path is never retried forever; without the failed-at guard the stub
reprocessed every fact on every run).
Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).
Serialise every writer of one consolidation scope (sync and Oban runs alike).
The transaction-scoped advisory lock is released at commit or rollback, so an
overlapping run waits, then sees committed state. Hindsight holds a per-scope
lock around the same window (consolidator.py _resolve_write_scopes).
How many times to retry a failing LLM merge for a batch BEFORE the orchestration
bisects it. Mirror of Hindsight consolidation_max_attempts (default 3,
consolidator.py:1286). Bank override wins, else engine config, else 3.
Identity + version of the pooled observations a merge plan was built against.
The subset of unit_ids still pending consolidation, locked FOR UPDATE for the
rest of the transaction. A chunk whose facts a concurrent run already stamped
must not be applied again.
Pool the existing observations that belong to scope_key so the LLM can merge
against them. Strict scope match (exact tags + observation_scopes) prevents
cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.
Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.
Resolve the source facts backing a set of pooled observations so their text +
dates can be serialized into the merge prompt (B3 / Hindsight
_build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.
Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's
_build_observations_for_llm (consolidator.py:1190-1226): each observation
carries id, text, proof_count, its temporal range, mentioned_at, and a
source_memories array (the supporting facts' text + dates, resolved from
source_memory_ids via source_facts_by_id). Temporal fields and
source_memories are only emitted when present so the prompt stays lean.
Stamp consolidated_at = now on the given source facts so they are not reprocessed.
Quarantine facts that a size-1 LLM merge could not process. Stamps
consolidation_failed_at = now, excluding them from the pending query
thereafter (Hindsight consolidator.py:509-513). Only reached after bisection
has isolated the poison fact, so the rest of a batch still consolidates.
Functions
Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's
source facts consolidated_at, and return the observations written
(creates + updates) for downstream mental-model refresh.
Hindsight's observation capacity note (consolidator.py:1520-1536): when a scope is
at or near max_observations_per_scope, tell the merge model so it prefers UPDATE
over CREATE. nil when no cap is configured or the scope has ample room.
Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL
AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending
filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the
bisection path is never retried forever; without the failed-at guard the stub
reprocessed every fact on every run).
Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).
Serialise every writer of one consolidation scope (sync and Oban runs alike).
The transaction-scoped advisory lock is released at commit or rollback, so an
overlapping run waits, then sees committed state. Hindsight holds a per-scope
lock around the same window (consolidator.py _resolve_write_scopes).
How many times to retry a failing LLM merge for a batch BEFORE the orchestration
bisects it. Mirror of Hindsight consolidation_max_attempts (default 3,
consolidator.py:1286). Bank override wins, else engine config, else 3.
Identity + version of the pooled observations a merge plan was built against.
The subset of unit_ids still pending consolidation, locked FOR UPDATE for the
rest of the transaction. A chunk whose facts a concurrent run already stamped
must not be applied again.
Pool the existing observations that belong to scope_key so the LLM can merge
against them. Strict scope match (exact tags + observation_scopes) prevents
cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.
Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.
{:ok, prepared}— the LLM call succeeded (even with empty ops); source facts for this scope will be stampedconsolidated_at.{:skip, reason}— the LLM/provider was unavailable; the scope is left untouched and its facts stay pending (retryable). Degrade-safe: never raises.
Resolve the source facts backing a set of pooled observations so their text +
dates can be serialized into the merge prompt (B3 / Hindsight
_build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.
Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's
_build_observations_for_llm (consolidator.py:1190-1226): each observation
carries id, text, proof_count, its temporal range, mentioned_at, and a
source_memories array (the supporting facts' text + dates, resolved from
source_memory_ids via source_facts_by_id). Temporal fields and
source_memories are only emitted when present so the prompt stays lean.
Stamp consolidated_at = now on the given source facts so they are not reprocessed.
Quarantine facts that a size-1 LLM merge could not process. Stamps
consolidation_failed_at = now, excluding them from the pending query
thereafter (Hindsight consolidator.py:509-513). Only reached after bisection
has isolated the poison fact, so the rest of a batch still consolidates.