Foresight.Evals.LongMemEval (Foresight v0.1.0)

Copy Markdown View Source

Adapter for LongMemEval (ICLR 2025), an MIT-licensed benchmark for long-term agent memory. It maps almost one-to-one onto Foresight's differentiators: temporal reasoning, knowledge updates (supersession), multi-session reasoning, and abstention.

Native format (per instance): question, answer, question_type, question_date, haystack_session_ids + haystack_dates + haystack_sessions (each session a list of {role, content} turns, evidence turns flagged has_answer), and answer_session_ids (the gold evidence sessions ⊆ haystack). Abstention instances have a _abs question_id.

Retrieval unit is the session: each haystack session becomes one retain-ready memory keyed by its session id, and relevance labels a session 1 iff it is an answer session. Session granularity matches the gold answer_session_ids; turn-level has_answer flags are preserved in metadata for a finer labelling pass later.

Data is NOT vendored (we do not redistribute the corpus). Fetch a split with mix foresight.evals.fetch into priv/evals/data/ (gitignored), then load/1 it.

Summary

Functions

load(path, opts \\ [])

Load and normalise a LongMemEval split from a JSON file path into Dataset.Instances.

load!(path, opts \\ [])

Same as load/2 but raises on error.

to_instance(raw)

Normalise a single raw LongMemEval instance (a decoded map) into a Dataset.Instance. Public so the loader can be unit-tested against a synthetic instance without shipping any of the copyrighted corpus.

Functions

load(path, opts \\ [])

@spec load(
  String.t(),
  keyword()
) :: {:ok, [Foresight.Evals.Dataset.t()]} | {:error, term()}

Load and normalise a LongMemEval split from a JSON file path into Dataset.Instances.

Options:

  • :limit — take at most N instances (after optional sampling).
  • :types — keep only these question_types.

load!(path, opts \\ [])

@spec load!(
  String.t(),
  keyword()
) :: [Foresight.Evals.Dataset.t()]

Same as load/2 but raises on error.

to_instance(raw)

@spec to_instance(map()) :: Foresight.Evals.Dataset.t()

Normalise a single raw LongMemEval instance (a decoded map) into a Dataset.Instance. Public so the loader can be unit-tested against a synthetic instance without shipping any of the copyrighted corpus.