Dataset sources and local-data policy

Copy Markdown View Source

Plexus does not vendor the external corpora used by the examples. Fetch scripts acquire them from their standard upstream source into .plexus-data/, which is ignored by Git. Users remain responsible for complying with each upstream dataset's current terms.

SWE-bench Verified

  • Upstream: SWE-bench/SWE-bench_Verified on Hugging Face.
  • Shape used: repository, issue/problem statement, base commit, gold patch, hints, version and benchmark metadata.
  • Plexus use: the patch is used only after inference to derive a fixed post-hoc fix-shape label (source_code, tests, documentation, configuration_build, or mixed). Patch content is never included in semantic state or used to construct the answer vocabulary.
  • Fetch path: Hugging Face datasets-server rows API.

NYC 311 Service Requests from 2020 to Present

  • Upstream dataset identifier: erm2-nwe9 on NYC Open Data.
  • Shape used: request id/time, agency, problem, problem detail, location type, borough, latitude/longitude, status.
  • Plexus use: large report population + spatiotemporal cluster actors; semantic work occurs at selected cluster boundaries.
  • Optional SOCRATA_APP_TOKEN can be supplied for API identification/rate handling.

GAIA / MicroSS

  • Upstream: CloudWise-OpenSource/GAIA-DataSet.
  • Shape used: MicroSS trace, business-log and run/fault-injection records.
  • Plexus use: raw trace and business-log rows are replayed while persistent service/hypothesis actors are alive. Trace parentage changes the :calls graph over time; hypotheses repeatedly revise, challenge peers, react to provenance invalidation, receive local branch credit, and may be pruned while the replay continues. Run/fault-injection records remain hidden until post-run scoring.
  • Acquisition: the official release-v1.0 Git ref via Git + Git LFS, followed by extraction of the standard MicroSS split archives with 7-Zip. The corpus is intentionally not copied into the package.
  • Terms: the upstream repository has conflicting license signals across its README/GitHub metadata; the fetcher downloads from upstream rather than redistributing GAIA data. Review the upstream LICENSE and dataset documentation before redistribution.

deps.dev

  • Upstream: Google Open Source Insights, stable v3 JSON API.
  • Shape used: source and target resolved dependency graphs plus selected target-version metadata/advisories.
  • Plexus use: every target graph node is an actor; topology follows the resolved graph; exact (system, name, version) target keys absent from the source graph consume semantic measurement budget. Package-level collapsing happens only when constructing human-readable migration-order search steps.

SciFact

  • Upstream: AllenAI SciFact release, https://scifact.s3-us-west-2.amazonaws.com/release/latest/data.tar.gz.
  • Shape used: claims_dev.jsonl and corpus.jsonl.
  • Plexus use: claim/document pair actors create predicted support/contradiction/insufficient-evidence edges and are evaluated against public dev labels.
  • Upstream license documentation states claims/evidence annotations are CC BY 4.0 and corpus abstracts are from S2ORC under ODC-By 1.0.

NOAA Storm Events

  • Upstream: NOAA/NCEI Storm Events bulk CSV directory.
  • Shape used: yearly StormEvents_details files, including event identity/type/time/state, casualties/damage and event narrative.
  • Plexus use: every selected historical event becomes a dormant actor subscribed to its day. State/day actors aggregate awakened cohorts; only the highest-impact groups invoke TypeSafe/Jev.

What is committed

Only code, documentation and tiny test fixtures are committed. These paths are ignored:

.plexus-data/
examples/**/data/
examples/**/.data/
examples/**/cache/
examples/**/downloads/

test/fixtures/examples/ is intentionally tracked; those records test local parsers and never stand in for the public example datasets.