Foresight.Storage.ConsolidationPersistence (Foresight v0.3.0)

Copy Markdown View Source

Consolidation persistence: an Elixir port of Hindsight's engine/consolidation/consolidator.py. Turns raw world/experience facts into consolidated observation memory units via an LLM creates/updates/ deletes merge contract (Foresight.Prompts.Consolidation.V1).

The LLM call and embedding happen OUTSIDE the DB transaction (see prepare_scope/7); only the resulting writes run inside it (apply_prepared!/4) — mirroring the retain path's txn discipline and avoiding the reflect txn-across-LLM pool-timeout class of bug.

Summary

Functions

Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's source facts consolidated_at, and return the observations written (creates + updates) for downstream mental-model refresh.

Hindsight's observation capacity note (consolidator.py:1520-1536): when a scope is at or near max_observations_per_scope, tell the merge model so it prefers UPDATE over CREATE. nil when no cap is configured or the scope has ample room.

Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the bisection path is never retried forever; without the failed-at guard the stub reprocessed every fact on every run).

Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).

Serialise every writer of one consolidation scope (sync and Oban runs alike). The transaction-scoped advisory lock is released at commit or rollback, so an overlapping run waits, then sees committed state. Hindsight holds a per-scope lock around the same window (consolidator.py _resolve_write_scopes).

How many times to retry a failing LLM merge for a batch BEFORE the orchestration bisects it. Mirror of Hindsight consolidation_max_attempts (default 3, consolidator.py:1286). Bank override wins, else engine config, else 3.

Identity + version of the pooled observations a merge plan was built against.

The subset of unit_ids still pending consolidation, locked FOR UPDATE for the rest of the transaction. A chunk whose facts a concurrent run already stamped must not be applied again.

Pool the existing observations that belong to scope_key so the LLM can merge against them. Strict scope match (exact tags + observation_scopes) prevents cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.

Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.

Resolve the source facts backing a set of pooled observations so their text + dates can be serialized into the merge prompt (B3 / Hindsight _build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.

Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's _build_observations_for_llm (consolidator.py:1190-1226): each observation carries id, text, proof_count, its temporal range, mentioned_at, and a source_memories array (the supporting facts' text + dates, resolved from source_memory_ids via source_facts_by_id). Temporal fields and source_memories are only emitted when present so the prompt stays lean.

Stamp consolidated_at = now on the given source facts so they are not reprocessed.

Quarantine facts that a size-1 LLM merge could not process. Stamps consolidation_failed_at = now, excluding them from the pending query thereafter (Hindsight consolidator.py:509-513). Only reached after bisection has isolated the poison fact, so the rest of a batch still consolidates.

Functions

apply_prepared!(repo, bank, prepared, repo_opts)

Apply a prepared scope plan (DELETE → UPDATE → CREATE), stamp the scope's source facts consolidated_at, and return the observations written (creates + updates) for downstream mental-model refresh.

capacity_note(remaining, cap, fact_count)

Hindsight's observation capacity note (consolidator.py:1520-1536): when a scope is at or near max_observations_per_scope, tell the merge model so it prefers UPDATE over CREATE. nil when no cap is configured or the scope has ample room.

consolidation_groups(repo, bank_id, payload, repo_opts \\ [])

Group unconsolidated facts by scope. Only facts with consolidated_at IS NULL AND consolidation_failed_at IS NULL are eligible (mirrors Hindsight's pending filter — consolidator.py:290-293,339-341 — so a poison fact quarantined by the bisection path is never retried forever; without the failed-at guard the stub reprocessed every fact on every run).

llm_batch_size(bank)

Resolve the per-scope LLM batch size (facts per merge call). Bank config override wins, else the engine default, else Hindsight's default of 8 (consolidator.py:257, config.py:604). Foresight previously sent an entire scope in one call — this bounds each merge call for JSON reliability (B1).

lock_scope(repo, bank_id, scope_key, repo_opts)

Serialise every writer of one consolidation scope (sync and Oban runs alike). The transaction-scoped advisory lock is released at commit or rollback, so an overlapping run waits, then sees committed state. Hindsight holds a per-scope lock around the same window (consolidator.py _resolve_write_scopes).

max_attempts(bank)

How many times to retry a failing LLM merge for a batch BEFORE the orchestration bisects it. Mirror of Hindsight consolidation_max_attempts (default 3, consolidator.py:1286). Bank override wins, else engine config, else 3.

normalize_observation_scopes(item)

observation_fingerprint(observations)

Identity + version of the pooled observations a merge plan was built against.

pending_unit_ids(repo, unit_ids, repo_opts)

The subset of unit_ids still pending consolidation, locked FOR UPDATE for the rest of the transaction. A chunk whose facts a concurrent run already stamped must not be applied again.

pool_observations(repo, bank_id, scope_key, repo_opts \\ [])

Pool the existing observations that belong to scope_key so the LLM can merge against them. Strict scope match (exact tags + observation_scopes) prevents cross-scope contamination — the Hindsight tags_match="all_strict" guarantee.

prepare_scope(llm, embedder, bank, scope_key, units, existing_obs, opts, source_facts \\ %{})

Run the LLM merge for one scope and pre-compute embeddings for every resulting create/update. Returns a prepared plan applied later inside the transaction.

  • {:ok, prepared} — the LLM call succeeded (even with empty ops); source facts for this scope will be stamped consolidated_at.
  • {:skip, reason} — the LLM/provider was unavailable; the scope is left untouched and its facts stay pending (retryable). Degrade-safe: never raises.

refresh_mental_model!(repo, bank, scope_key, scope_observations, units, repo_opts, bank_document_record_fun)

requested_scope_keys(payload)

resolve_source_facts(repo, bank_id, source_ids, repo_opts)

Resolve the source facts backing a set of pooled observations so their text + dates can be serialized into the merge prompt (B3 / Hindsight _build_observations_for_llm). Returns a %{id_string => MemoryUnit} map.

scope_key_for_item(item)

serialize_observations(observations, source_facts_by_id \\ %{})

Serialize pooled observations for the LLM merge prompt. Mirrors Hindsight's _build_observations_for_llm (consolidator.py:1190-1226): each observation carries id, text, proof_count, its temporal range, mentioned_at, and a source_memories array (the supporting facts' text + dates, resolved from source_memory_ids via source_facts_by_id). Temporal fields and source_memories are only emitted when present so the prompt stays lean.

stamp_consolidated!(repo, unit_ids, repo_opts)

Stamp consolidated_at = now on the given source facts so they are not reprocessed.

stamp_consolidation_failed!(repo, unit_ids, repo_opts)

Quarantine facts that a size-1 LLM merge could not process. Stamps consolidation_failed_at = now, excluding them from the pending query thereafter (Hindsight consolidator.py:509-513). Only reached after bisection has isolated the poison fact, so the rest of a batch still consolidates.