Gitility.ODB.PackFetch (Gitility v0.4.0)

Copy Markdown View Source

Eagerly hydrates an immutable pack manifest into a local gix object store.

start_link/1 uses the same callback supervision and request-resource protocol as Gitility.ODB.start_link/1, but the backend implements Gitility.ODB.RangeBackend. Startup deliberately blocks until the initial manifest has been fetched, every pack/index pair has been verified and atomically published, and the resulting objects directory is open. A handle is never exposed in a half-hydrated state.

{:ok, supervisor} =
  Gitility.ODB.PackFetch.start_link(
    backend: {MyApp.PackStore, backend_options},
    into: {:dir, "/var/cache/gitility"},
    concurrency: 8,
    verify: :always
  )

{:ok, odb} = Gitility.ODB.handle(supervisor)

into: {:dir, path} writes only beneath the explicit path, using objects/pack/pack-<checksum>.{pack,idx}. Existing valid pairs are reused; corrupt pairs are replaced. Removed manifest entries are retained in 0.2 — refresh never deletes packs during the publisher's grace period. If a later pack fails, earlier verified pairs remain for the next attempt; no unverified file is left under a final name. Within one store lifetime, refresh size-checks already verified pairs and is O(new packs). A restart re-hashes the whole reused volume once because trust is never persisted to disk.

into: :memory is available only on Linux. It uses a caller-invisible directory below /dev/shm, which is a RAM-backed tmpfs, enforces max_bytes, and removes the directory during orderly provider shutdown. Native resource destruction never performs filesystem work on a BEAM scheduler. An abnormal death can leave this bounded tmpfs directory; the next start_link/1 with the same :name sweeps it before starting. This is not a bytes-in-Rust gix store: stock gix-pack is path-only and mmap-based (design finding F7). macOS and other platforms return :unsupported_operation; use an explicit directory there.

into: {:bundle, path} serves from a caller-invisible private scratch directory below System.tmp_dir!/0 while maintaining path as the durable artifact. A valid existing bundle is checksum-extracted into that scratch directory before hydration, so warm starts reuse its verified pairs without remote reads; corrupt sections are omitted and fetched again. The scratch directory is always removed during orderly provider shutdown, while the bundle is never cleaned up by the provider. An abnormal death can leave the scratch directory, including full pack copies, on real disk; the next start_link/1 for the same expanded bundle path sweeps it before starting.

Hydration bundles contain the current manifest's pack/index pairs and zero reference rows: they are ODB-only snapshots. Use Gitility.Bundle.write/2 for a full repository bundle with refs. The file is atomically replaced only when the would-be snapshot changes. It records the manifest from the last completed start_link/1 hydration. Gitility.ODB.refresh/1 serves new packs from the scratch store without rewriting the bundle; restarting re-publishes the latest manifest. Packs removed from that manifest remain in scratch for the process lifetime, but the rewritten bundle omits them, so the grace period does not survive a bundle-destination restart. Exactly one store may own a bundle path at a time; concurrent writers to one path are outside the contract. Correct concurrent refresh publication requires a future single-owner publisher design. A non-bundle file at path is never clobbered; remove it explicitly before starting PackFetch.

Options

  • :backend (required) — {module, init_arg} implementing Gitility.ODB.RangeBackend.
  • :into{:dir, path}, {:bundle, path}, or :memory (default :memory).
  • :name — supervisor registered name; omitted starts privately.
  • :hash:sha1 (default) or :sha256; must match the manifest.
  • :concurrency — maximum outstanding read_ranges callbacks per pack (default 8).
  • :chunk_bytes — fixed pack range size (default 8 MiB).
  • :request_timeout — per-callback timeout in milliseconds (default 15_000).
  • :verify — only :always is accepted.
  • :runtime — query runtime (default shared).
  • :max_hydration_bytes — bytes the hydration plan may actually fetch (default 4 GiB). This one-time bulk-load ceiling is distinct from query-time Gitility.Limits.max_provider_bytes (default 256 MiB). Existing pairs are verified before planning, so a warm volume is charged only for manifest metadata plus missing or corrupt pairs. Local pre-extraction from a bundle is never charged to this backend-read budget.
  • :limits — the other hydration-job ceilings and timeout. Hydration always replaces this struct's max_provider_bytes with max_hydration_bytes; query-time limits are unchanged.
  • :max_bytes — RAM destination ceiling (default 256 MiB); used only by into: :memory.
  • :bundle_source_identity — deterministic source_identity metadata for into: {:bundle, path}. Defaults to "packfetch:generation:" <> manifest.generation.

Summary

Functions

Returns whether into: :memory is supported on this platform.

Starts a PackFetch store over a range backend.

Functions

memory_supported?()

@spec memory_supported?() :: boolean()

Returns whether into: :memory is supported on this platform.

start_link(opts)

@spec start_link(keyword()) :: Supervisor.on_start()

Starts a PackFetch store over a range backend.

:backend ({module, init_arg}) and :limits are required; :into selects the hydration destination (:memory by default, {:dir, path}, or {:bundle, path}). See the moduledoc for the full option set and the semantics of each destination.