Changelog

Copy Markdown View Source

All notable changes to this project are documented here. This project adheres to Semantic Versioning.

[0.1.5] — 2026-10-12

Fixed

  • update_mode: "replace" no longer rebuilds every entity in the bank. Any replace (and delete_document, update_document, clear_observations, clear_memory_observations) used to delete all entities and links in the bank and re-derive them from stored text with the regex extractor: O(bank) per call, and LLM-extracted entities and their name embeddings were lost for good. These paths now retract only the deleted units' mentions: mention counts and co-occurrence weights are decremented, exhausted links dropped, and entities nobody mentions any more pruned. Entities the surviving units mention keep their identity, type, and embedding. replace is still opt-in, as in Hindsight, but it is no longer expensive or lossy.
  • Concurrent retains of one document_id are safe. The document upsert is a single INSERT ... ON CONFLICT DO UPDATE; the former read-then-write pair raised Ecto.ConstraintError (HTTP 500) for the loser of a concurrent first insert. Replace mode takes a transaction-scoped advisory lock per document, so overlapping replaces serialise and leave exactly one set of memory units. delete_document and update_document take the same lock before touching the document, and a batch upserts its document rows in a stable order, so no pair of these writers can deadlock with each other. Post-commit link construction can still cross a concurrent replace or delete of the units it links to; Postgres then aborts one side, and the retain or document write is retried instead of failing with deadlock_detected. delete_document and update_document re-read the document under the lock, so a row deleted by a concurrent writer yields :not_found rather than Ecto.StaleEntryError. Document.changeset/2 declares the unique constraint.
  • Co-occurrence link weights start at 1. A fresh EntityLink struct's schema default made the first co-occurrence of a pair count as 2.
  • A failed recall normalisation returned the timing wrapper. Rejected payloads (an unsupported field, a bad min_score) came back as {{:error, _}, ms} instead of {:error, %Foresight.Error{}}.
  • Retain extraction fallback is fully reported and no longer leaks. A model response that was valid JSON but not a facts list fell back to storing the raw text with no log line and no telemetry; it now reports like every other fallback. The warning names only the failure kind: the raw reason, which for a structured-output failure carries facts the model derived from the user's text, is no longer written to the log. Telemetry handlers still receive it, plus a reason_kind.
  • The regex entity extractor no longer reads When and Involving as entities. Stored facts are assembled as what | When: x | Involving: y, and the capitalised labels matched the entity pattern on every fact.
  • Cross-encoder scores are no longer NaN on CPU. transformers 5.x leaves CPU weights backed by the memory-mapped checkpoint; on macOS/arm64 a forward pass over those tensors bus-faults or returns NaN for every pair (reproduced with torch 2.10 and 2.13, transformers 5.8 and 5.13) while MPS is correct. The Python worker now copies CPU-resident weights into ordinary tensors after loading, and refuses to return non-finite scores or vectors (a clean error instead of the erlport protocol failure a NaN used to cause). The Elixir reranker fails closed on a non-numeric score.

Added

  • score on every recall result, and an optional min_score cutoff. Both are Foresight extensions to the Hindsight recall contract. score is the reranked, boosted score the results are ordered by; min_score (a non-negative number) drops results below it after boosting and before the token trim. after, before, limit, and offset stay rejected.
  • extraction on the retain result: %{"enabled", "mode", "fallback_items", "empty_items"}. fallback_items counts items stored as raw text because extraction errored; empty_items counts items stored raw because extraction found no facts. The HTTP retain response carries it. It describes the synchronous write: an async retain answers with operation ids, and the report is on the completed operation's result. The [:foresight, :retain, :extraction_fallback] telemetry event fires once per failed chunk, so its count can exceed fallback_items, which counts items. The MCP recall tool schema declares min_score.
  • Foresight.Retain.FactExtraction.extract/2 returns {items, stats}; maybe_extract/2 is unchanged.

[0.1.4] — 2026-10-05

Added

  • Text-body documents in retain_files. A file given as %{"text" => body} (or a bare binary) is retained as-is by a dedicated text converter, bypassing the configured converter and never reading a "path". This is the shape a Hindsight migration carries by construction. Without a "document_id" or "filename" the document gets a content-addressed id (text:<sha256>), so single-file calls cannot collide on file-0. Pass "request" => %{"async" => false} for a synchronous write. Request options are read with string keys; atom keys are ignored.
  • Blank text bodies are rejected, not silently dropped: retain discards items with no alphanumeric content, which would make a blank document vanish from a copy without trace.
  • update_mode passthrough on retain_files (per file or in "request"). Re-retaining a document_id upserts the document but, by default, appends a second set of memory units. "update_mode" => "replace" deletes the prior set but also rebuilds every entity in the bank on each call, so it is opt-in only; bulk copies should rely on their own once-only ledger. Large text bodies are retained as a single unchunked memory unit: pre-chunk them.

Documentation

  • config :foresight, backend: is now documented in the getting-started guide, whose "working configuration" omitted it and so left new consumers on the Noop backend. Use backend: Foresight.Backend.Store; storage: Foresight.Storage.Ecto belongs under engine: and lacks retain_files/3 as a backend.

Fixed

  • update_mode: "replace" raised on any document that already had memory units (Postgrex expected a binary of 16 bytes in affected_observation_ids): the source-unit id array is now bound with an explicit {:array, Ecto.UUID} type. The path had no Postgres coverage before.
  • update_mode: "replace" now releases sibling units for reconsolidation. Units that fed an observation deleted by the replace were left marked consolidated; they are reset as delete_documents already did.

[0.1.3] — 2026-09-28

Changed

  • Trademark acknowledgment in the README. Upstream Hindsight now carries a claimed trademark mark alongside a commercial cloud offering. Foresight is an independent port and is not affiliated with, endorsed by, or sponsored by Vectorize AI, Inc.; references to Hindsight serve attribution (the MIT notice in LICENSE, unchanged since 0.1.0) and interoperability description only. No code changes.

[0.1.2] — 2026-09-28

Fixed

  • Foresight.Migrations.install/2 no longer collides across BEAMs. The scratch directory used to materialize control-plane migrations was named with System.unique_integer/1, which is unique only within a single BEAM. Two VMs sharing a TMPDIR — a release beside a migration task, concurrent cold boots, CI jobs on a shared volume — could land on the same path and silently load each other's partial link trees. Scratch directories are now allocated under cryptographically random names with an exclusive mkdir (retried on the astronomically unlikely collision), partial link trees are removed on failure, and stale directories left by another BEAM are ignored.

[0.1.1] — 2026-09-27

Added

  • Foresight.Migrations.install/2, an idempotent in-BEAM installer for packaged control-plane migrations using the collision-safe foresight_schema_migrations ledger.
  • Foresight.Tenancy.provision_mode_c_schema/4, the supported role-neutral consumer seam for installing the control plane and provisioning a tenant schema without creating roles or tenant keys.
  • An exhaustive packaged-migration classification manifest.
  • Automatic adoption of structurally verified 0.1.0 control-plane migrations into the dedicated Foresight ledger.

Fixed

  • Explicit per-call rls_enabled: false now reaches every Ecto tenant scope, parallel recall strategy, nested reflect helper, and applicable async worker instead of being replaced by the application default.
  • Tenant webhook discovery and delivery creation now execute inside the same RLS scope as the operation that emitted them.

[0.1.0] — 2026-09-07

Initial public release, published as foresight_memory.

Foresight is an Elixir port of Hindsight (MIT, Vectorize AI, Inc.). The ported architecture is theirs and is attributed in the README (Provenance) and the LICENSE (third-party notice); divergence from it is documented where introduced. Everything below predates the first publish and is part of the 0.1.0 story.

Changed

  • Recall fusion now defaults to :normalized / :zscore (was :rrf / :min_max). Measured over 134 graded-relevance queries: +0.1009 nDCG@10, 95% CI [0.0597, 0.1476], and MRR 0.776 → 0.903. :rrf merges on rank position and discards the arm scores; normalizing keeps them. Existing banks with an explicit recall_fusion override are unaffected.
  • Default reranker is now Rerankers.Passthrough (was Rerankers.Bumblebee). The old default made a bare install demand the Nx/Bumblebee/EXLA stack and fail its capability preflight, so Foresight could not boot without ML dependencies nobody had asked for.

Fixed

  • Foresight is installable. :plug and :llm_core were declared optional: true while being referenced unconditionally at compile time, so {:foresight, "~> 0.1"} on its own failed mix compile. :plug and the new direct :llm_toolkit are now required; :llm_core remains optional and its adapter is guarded, with a stub that names the missing dependency.
  • Recall trace reports the fusion algorithm actually in use. It was hardcoded to "rrf" and reported the wrong algorithm on every request once the default changed.
  • MCP: async_processing accepted as an alias for async; fabricated lenient sessions are closed at the end of the request that created them; undeclared tool arguments are named in the log instead of being dropped.
  • Enqueueing without Oban returns a structured :unavailable error instead of raising.
  • Scrivener ingestion: path traversal via an unvalidated bundle UUID, escape via symlinked path components, and unbounded reads from non-regular files are all closed.

Security

  • Mode C schema-per-tenant isolation verified under a genuine least-privilege role (rolsuper = false, RLS forced): five forged-header cross-tenant attacks, 5/5 blocked, verified by canary rather than status code.

[0.1.0]

Initial development version.