Data Acquisition

Sidereon.GNSS.Data fetches GNSS archive products and DTED terrain tiles into a verified local cache. It is cache-first, supports offline reads, records provenance next to each data file, and derives product names, URLs, terrain paths, and DTED bytes through the core NIF.

Terrain Quick Start

Fetch the tile first, then pass the terrain cache root to the terrain reader:

{:ok, tile_path} = Sidereon.GNSS.Data.fetch_dted(36.75, -106.25)
terrain_root = tile_path |> Path.dirname() |> Path.dirname()

{:ok, terrain} = Sidereon.Terrain.dted(terrain_root)
{:ok, height_m} = Sidereon.Terrain.height(terrain, -106.25, 36.75)

For fully ocean or no-data tiles, fetch_dted/3 returns:

{:ok, {:no_coverage, tile_id}}

The terrain reader treats absent tiles as sea level, so callers can decide whether to prefetch no-coverage markers or simply rely on the reader fallback.

Bulk Terrain Cache

Populate a region while online:

terrain_root = "/tmp/sidereon-terrain"

{:ok, report} =
  Sidereon.GNSS.Data.prefetch_dted_bbox({36.0, -107.0, 37.0, -106.0},
    cache_dir: terrain_root
  )

report.fetched
report.cached
report.no_coverage
report.errors

Tile lists accept core Skadi ids or {lat_index, lon_index} pairs:

{:ok, report} =
  Sidereon.GNSS.Data.prefetch_dted_tiles(["N36W107", {36, -106}],
    cache_dir: terrain_root
  )

Later, run without network:

{:ok, tile_path} =
  Sidereon.GNSS.Data.fetch_dted(36.75, -106.25,
    cache_dir: terrain_root,
    offline: true
  )

GNSS Products

Build products through the catalog wrappers and fetch them to the GNSS cache:

{:ok, sp3_product} = Sidereon.GNSS.Data.mgex_sp3(:esa, ~D[2020-06-24])
{:ok, sp3_path} = Sidereon.GNSS.Data.fetch(sp3_product)
{:ok, sp3} = Sidereon.GNSS.SP3.load(sp3_path)

{:ok, ionex_path} = Sidereon.GNSS.Data.fetch_ionex(:cod_rap, ~D[2026-06-13])
{:ok, ionex} = Sidereon.GNSS.Ionosphere.load_ionex(ionex_path)

{:ok, nav_product} = Sidereon.GNSS.Data.mgex_nav(:igs, ~D[2020-06-25])
{:ok, nav_path} = Sidereon.GNSS.Data.fetch(nav_product)

{:ok, clk_product} = Sidereon.GNSS.Data.mgex_clk(:gfz, ~D[2020-06-24])
{:ok, clk_path} = Sidereon.GNSS.Data.fetch(clk_product)

Exact Product and Explicit Source Selection

The exact-acquisition API treats product identity and distribution as separate public concepts. First build the catalog product, then list only the acceptable sources in priority order:

alias Sidereon.GNSS.Data
alias Sidereon.GNSS.Distribution

{:ok, product} = Data.mgex_sp3(:cod, ~D[2020-06-25])

{:ok, request} =
  Data.request(product, [
    Distribution.nasa_cddis(),
    Distribution.direct()
  ])

{:ok, result} =
  Data.acquire(request,
    earthdata_auth:
      Distribution.EarthdataAuth.bearer(System.fetch_env!("EARTHDATA_TOKEN"))
  )

result.path
result.provenance.distribution_source
result.provenance.sha256

Only the listed sources are tried. If CDDIS returns 404, the example may try the direct archive, but it retains the same publisher, analysis-center product line, solution class, issue, date, cadence, family, format, and official filename. Earlier failures appear in result.provenance.attempts.

When several exact products are required, declare the complete identity set and gate dependent processing on the resolved acquisition identities:

expected = [request_a.identity, request_b.identity]

available = [
  result_a.provenance.resolved_identity,
  result_b.provenance.resolved_identity
]

:ok = Data.validate_exact_product_set(expected, available)

The gate rejects empty declarations, duplicates, missing products, undeclared products, and same-filename identities with different prediction metadata. For SP3 observed/predicted timing, use Sidereon.GNSS.SP3.prediction_summary/1; issue times and catalog fields are not substitutes for the product's record flags.

Caller-provided inputs use the same validation path:

Distribution.local_file("/data/exact-product.sp3.gz")
Distribution.in_memory(product_bytes, compression: :none)

These sources do not grant permission to relabel their contents. SP3 and IONEX bytes must parse and match the requested start date/time and cadence.

CDDIS and Earthdata Login

SP3 CDDIS URLs use the GNSS-products GPS-week directory. Current long-name IONEX URLs use ionex/<year>/<day-of-year>. Both keep the identity's exact official filename and use gzip as transport packaging.

Credentials are supplied by the caller. Besides bearer tokens, Earthdata's documented netrc mechanism is available:

machine urs.earthdata.nasa.gov login EARTHDATA_USERNAME password EARTHDATA_PASSWORD

auth = Distribution.EarthdataAuth.netrc()
{:ok, result} = Data.acquire(request, earthdata_auth: auth)

The bearer token, netrc path and password, cookies, authorization headers, and URL queries are excluded from errors and provenance. Redirects are restricted to approved HTTPS hosts. Cookies retain their documented host/domain, path, and secure restrictions.

The path and authentication decisions follow these official public sources:

• NASA CDDIS archive access
• Earthdata curl and wget access
• Earthdata bearer-token example
• NASA precise orbit products
• NASA GNSS atmospheric products
• IGS long product filename guidelines

Exact Acquisition Provenance and Cache

Distribution.Provenance records requested and parsed/resolved identity, publisher, source, official filename, sanitized original and final URLs, retrieval time, decompressed and archive lengths and SHA-256 hashes, compression, ETag/Last-Modified when present, cache-hit state, and earlier source failures.

The cache key includes the source and every exact identity discriminator. Each accepted entry is one immutable transaction containing the decompressed product, original archive bytes, and JSON provenance. A SHA-256-bound commit record names that transaction and is atomically replaced only after the files and directories are synchronized. Hits follow only that record, then recheck both hashes and lengths, full requested and resolved identities, source, caller checksum, and a fresh SP3/IONEX semantic parse.

On Linux and macOS, acquisition delegates to the shared Rust transaction implementation. Threads, BEAM instances, and other cooperating processes use its per-entry advisory lock across cache validation, acquisition, and commit. Waiters therefore reuse the completed entry instead of downloading it again. :cache_lock_timeout_ms bounds the wait (30,000 milliseconds by default); a timeout or cache write failure is terminal and never authorizes trying another distributor. The operating system releases a dead owner's lock, so abandoned transactions are removed only after a new owner holds it. Low-level callers can use Sidereon.GNSS.ExactCache directly while retaining responsibility for transport and product-format validation.

Publication relies on same-filesystem atomic rename, synchronized regular files, and synchronized entry, entries, and commit-record directories. A process death or power loss at a boundary leaves the prior complete commit or no acceptable commit. Fully valid cache triples from 0.29.0-0.29.2 are revalidated and migrated without another acquisition. result.path keeps the official filename inside the immutable transaction directory. These guarantees apply to local filesystems on Linux and macOS that honor POSIX advisory locks, atomic same-directory rename, and directory synchronization. The legacy Data.fetch/2 API and cache layout remain unchanged.

Failures remain typed tagged tuples, including authentication required/failed, authorization denied, product not published, retired endpoint, redirect policy, malformed URL, transport timeout/connection, HTML or invalid content, content length, download size, decompression, checksum, semantic validation, and cache read/write errors. Multiple failed sources return {:all_distributors_failed, failures}. Retries are bounded to meaningful transport errors, HTTP 408/429, and server errors; callers should retain the verified cache and keep public-service concurrency modest.

Merged SP3 acquisition fetches each contributing center, parses the products, and uses the existing SP3 merge implementation:

{:ok, merged_sp3, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3(~D[2024-09-03], [:igs_ult, :gfz_ult])

Ultra-rapid acquisition probes each center's current primary filename pattern, then known duration/sampling alternates and documented latest-product aliases when an archive returns 404. report.contributors records the pattern that succeeded. Merge policy options are forwarded unchanged, including guarded cell precedence:

{:ok, merged_sp3, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3(
    ~N[2026-07-12 18:30:00],
    [:igs_ult, :cod_ult, :esa_ult, :gfz_ult],
    combine: :precedence,
    min_agree: 1,
    precedence_scope: :cell,
    outlier_reject: [position_m: 0.5, clock_ns: 5.0]
  )

prediction = Sidereon.GNSS.SP3.prediction_summary(merged_sp3)
prediction.observed_through

The default merge timeline uses the finest commensurate input cadence, with lower-precedence centers filling missing cells. No interpolation is introduced. Even one successful contributor passes through the same merge path so system filters and cadence validation remain consistent.

Each successful contributor now has two deliberately separate records: artifact_identity contains the requested and content-resolved product identities, explicit distributor, official filename, product and archive SHA-256 digests and lengths, and compression; acquisition contains cache-hit status, retrieval time, sanitized public URLs, HTTP metadata, and earlier attempts. Only the first record enters report.stable_input_identity.

That versioned identity canonically binds the complete contributor set and merge policy. It is unchanged by a warm-cache hit, process restart, map order, or non-semantic contributor enumeration order, and changes when an artifact, resolved identity, contributor set, or merge option changes. Precedence source order is bound explicitly because reversing it can change merged bytes. Persist a secret-free map without inspecting cache directories:

persisted = Sidereon.GNSS.Data.merge_report_to_map(report)
json = Jason.encode!(persisted)
:ok = json |> Jason.decode!() |> Sidereon.GNSS.Data.verify_merge_report()

The compatibility file helper still returns only the written path. Call fetch_merged_sp3_file_with_report/4 when the report must be retained:

{:ok, path, report} =
  Sidereon.GNSS.Data.fetch_merged_sp3_file_with_report(
    ~D[2024-09-03],
    [:igs_ult, :gfz_ult],
    "merged.SP3"
  )

Merged-SP3 acquisition accepts only complete, verified exact-acquisition records. It does not scan cache directories or infer a contributor identity from a filename, timestamp, or local path.

Every fetch returns either {:ok, value} or {:error, reason}. Offline misses, checksum failures, redirects, archive 404s, no-coverage terrain, cache failures, and catalog validation all use typed tagged reasons.