Foresight readiness

Last updated: 2026-08-14

This document exists for the first consumer. It states plainly what Foresight is ready for, what it is not, and which claims are measured versus assumed. Where a number appears, the method that produced it appears with it.

Read this before the README. The README describes how to run the thing; this describes whether you should.

Headline

Foresight is at statistical parity with Python Hindsight, and is ready to be depended on. It does not measurably exceed Hindsight, and no claim in this repo should be read as saying it does.

Parity is the shipping bar we chose deliberately. Work that might push past it exists, is scoped, and is deferred.

What "parity" means here, exactly

The claim rests on a matched-reader benchmark, both arms atomizing, n=30×3 on LongMemEval-S:

Δ +0.089 (SD 0.069). McNemar p = 0.227 / 1.000 / 0.774 across three replications — not significant in any of them, and the mean delta sits inside the ±0.1 nondeterminism band for n=30. The point estimate favours Foresight; the test does not support calling that a lead. Treat the two as equivalent.

Measured 2026-08-14, gpt-4o-mini both arms, gpt-4o judge, seed 42, n=30, three replicates, with the :normalized/:zscore fusion default in place. Foresight's own level rose from 0.708 in July while Hindsight moved 0.686 → 0.711 (inside its own noise), so the library improved and the scoreboard still reads parity.

The single most important number here is the spread: the delta ranged +0.034 to +0.166 across identical configuration. Any single run of this benchmark can be made to say almost anything.

This matters because it did not start there. The same benchmark once showed Foresight at 0.467 against Hindsight's 0.733 — a real, significant deficit. That deficit was closed by one cluster of fixes in retain extraction (chunking, the When: clause, temporal offsets), which is why the attribution is quoted rather than a general "we improved things".

What is NOT measured

• No exceed result exists. The three-replicate head-to-head above is the strongest evidence available and it says parity. Per-bank learned fusion weights were also re-measured properly at n=150 with query-level splits and came out at +0.0153 nDCG@10 (SD 0.0275, negative in 3 of 12 splits) — not significant, not shipped.
• What DID measurably improve is recall ranking, not answer accuracy. The fusion default moved from :rrf to :normalized/:zscore on +0.1009 nDCG@10 (95% CI [0.0597, 0.1476], 12/12 splits positive, 134 labelled queries). That is a ranking metric on one corpus with machine-anchored labels — it did not translate into a significant answer-accuracy lead, and this document does not claim it did.
• Consolidation and recall improvements are unmeasured, not unproven. The n=30 answer-judge lane hit its sensitivity ceiling: a consolidation change that produced +52% observations moved the score 0.711 → 0.708, i.e. nothing. Those changes shipped as correctness wins with no eval-lift claim attached.

Label provenance — read this before trusting any eval number

Gold labels in this repo are agent-anchored, not human-graded. This is a standing posture, adopted deliberately because human grading capacity was not available, and it is stated here rather than buried.

Practically:

• Labels carry source: :agent — a provenance distinct from :human on purpose, so nothing can later mistake one for the other.
• LLM-judged labels carry source: :llm and stamp judge_model as "<model>@<rubric-version>", derived from the rubric actually sent. There is deliberately no caller override for that field: one existed, and it allowed a label to claim a rubric version the judge never saw.
• Agreement is gated on Cohen's quadratic-weighted kappa ≥ 0.6 before labels are accepted.

If you need human-anchored ground truth for a regulatory or publication claim, this repo does not have it, and the trust gate does not manufacture it.

Which surface to use

Three surfaces exist. They are not equal, and the ordering is a recommendation, not a formality.

1. In-BEAM library — recommended

Take foresight as a dependency and call it directly. No serialization, no session lifecycle, no transport error class, and errors arrive as Foresight.Error structs you can pattern-match. Everything below is a wrapper over this.

Two of the "optional" dependencies are not optional at compile time. This was verified by building a throwaway consumer project against this repo, not reasoned about. mix.exs declares :plug and :llm_core as optional: true, but 19 modules import Plug.Conn unconditionally and storage/ecto.ex references %LlmToolkit.Tool.Call{} unconditionally — so a consumer that omits them gets a hard CompileError, not a gracefully absent capability, even with HTTP and MCP switched off. Foresight.MCP.Server guards itself properly with Code.ensure_loaded?; the HTTP modules do not. Tracked, not yet fixed.

Until it is, this is the recipe that works — verified end to end, compile and boot:

# mix.exs
{:foresight, github: "fosferon/foresight"},
{:plug, "~> 1.0"},        # required to COMPILE, even with http: [enabled: false]
{:llm_core, ">= 0.5.0"},  # required to COMPILE, even with no LLM configured

# config/config.exs — minimal in-BEAM consumer, no HTTP/MCP/Oban/local ML
config :foresight,
  engine: [
    embedder: Foresight.Embedders.Noop,
    reranker: Foresight.Rerankers.Passthrough
  ],
  http: [enabled: false],
  mcp: [enabled: false],
  oban: [enabled: false],
  repo: [enabled: false],
  file_ingestion: [enabled: false]

Note the nesting: embedder and reranker live under engine:. Setting them at the top level silently does nothing, and the failure surfaces later as a preflight error about :nx rather than as a config error.

The capability preflight itself is good — it refuses to boot with a specific, actionable message (optional dependency :nx is not available) rather than failing deep in a request. Expect to meet it while narrowing the config down.

One inherited surprise: llm_core starts a Hindsight discovery monitor that probes an HTTP endpoint at boot and retries on connection refused. That is llm_core's behaviour, not Foresight's, but you will see it in your logs because Foresight cannot currently compile without it.

2. REST — 67 routes, recommended if you need a network boundary

A full surface (/v1/:tenant/banks/...) covering banks, memories, recall, reflect, observations, mental models, operations, directives, import/export. Well-trodden; this is what the benchmark harnesses drive.

3. MCP — 11 tools, works, but understand what you are choosing

Every defect found in the July/August spike is fixed: the server boots in the right order, a bad tool call can no longer take the process down, list_banks (the usual health probe) answers its own valid call, retain no longer silently requires Oban, and the handshake gap is bridged.

Two structural properties survive the fixes and are worth knowing:

• One Hermes.Server.Base GenServer owns every live session. The reachable crash paths are closed and tested, but a raise originating outside the tool path would still affect all sessions. That is a property of the architecture, not of any remaining bug.
• lenient_session: true is opt-in and default OFF. Consumers ported from Python Hindsight that post bare tools/call with no MCP handshake need it. With it off they get Server not initialized. Fabricated sessions are torn down per request rather than held for the idle timeout, so this no longer accumulates session processes under load.

Argument-name compatibility with Python Hindsight is handled: async_processing is accepted as an alias for async, and an explicit async wins over the alias rather than being overwritten by it. Undeclared arguments are named in the log instead of vanishing silently.

Security posture

Mode C (schema-per-tenant) has been run under a genuine least-privilege role and proven, not assumed.

• The application role is rolsuper = false. This matters: RLS is not enforced against a superuser, so any isolation claim made while connected as one is vacuous.
• Tenant tables carry both ROW LEVEL SECURITY and FORCE ROW LEVEL SECURITY.
• Five forged-header cross-tenant attacks were run over real HTTP against a live instance. 5/5 blocked, verified by grepping for the victim's canary string in the attacker's responses rather than by trusting status codes.
• The proof script exits 2 (INCONCLUSIVE) rather than 0 if it cannot confirm its own self-check — a green run means the test could have failed.

Setup is genuinely fiddly and gets it wrong in five distinct ways. MODE_C_LEAST_PRIVILEGE_RUNBOOK.md documents the requirements in the order they actually failed, with the real error strings.

The posture monitor compares against the connection's own login role rather than a hardcoded postgres, which previously made the least-privilege posture structurally unreportable.

Known gaps

Things a consumer could reasonably expect that are absent or deferred:

None of these block the in-BEAM or REST surfaces.

Verifying this yourself

mix ci

Runs format check, compile --warnings-as-errors, three architectural fitness gates (namespace purity, tenancy boundary, isolation coverage), and the suite: 957 tests, 0 failures.

The fitness gates fail the build rather than warn. isolation_coverage specifically fails if the tenant-isolation tests are tagged out or if the suite is empty — an excluded isolation suite is indistinguishable from a passing one otherwise, which is exactly how this class of test dies quietly.

For the security claim specifically:

VICTIM_API_KEY=... elixir scripts/mode_c_isolation_proof.exs

Exit 0 = attacks blocked and the self-check confirmed. Exit 2 = inconclusive, do not read it as a pass.