Foresight.Storage.Consolidation (Foresight v0.1.0)

Copy Markdown View Source

Consolidation orchestration. Phase A (pool existing observations, run the LLM merge, embed results) runs OUTSIDE the DB transaction; Phase B applies the prepared writes, stamps sources, and refreshes mental models INSIDE one transaction. This keeps slow LLM/embedding IO off the tenant DB connection (the reflect txn-across-LLM pool-timeout class of bug).

Faithful port of Hindsight's consolidator.py batching (calibration Cluster B):

  • B1 — each scope's pending facts are split into consolidation_llm_batch_size chunks and merged one chunk per LLM call, instead of cramming the whole scope into a single oversized prompt. Chunks are processed sequentially and each commits before the next, so a create from an earlier chunk is re-pooled as an existing observation for later chunks (consolidator.py:360-364).
  • B4 — a drain loop keeps fetching and processing pending groups until none remain (consolidator.py:324 while True), so tail facts beyond the first fetch slice still become observations.
  • B5 — on an LLM merge failure a sub-batch is bisected down to size 1; a single fact that still fails is quarantined via consolidation_failed_at (consolidator.py:382-513), so the rest of a batch still consolidates and a poison fact is not retried forever.

The Phase-A-outside-txn / Phase-B-inside-txn discipline is preserved at CHUNK granularity: each chunk's LLM/embed work happens outside the txn, and only that chunk's writes run inside a (per-chunk) transaction — mirroring Hindsight's per-batch commits for crash recovery.

Summary

Functions

consolidate(ctx, payload, opts)

consolidate(ctx, payload, opts, callbacks)