Gitility.ODB (Gitility v0.3.0)

Copy Markdown View Source

Object databases: where Git objects come from.

An ODB is content-addressed and immutable-by-ID — it has no notion of branches or names (that is Gitility.RefDB). A caller that already knows a commit ID needs only an ODB; Gitility.Snapshot.open/2 takes one directly.

All stores yield the same opaque t/0 handle, and every store answers the same query API identically — storage never leaks into query semantics. Stores:

Layers are queried in order and must share a runtime and hash algorithm. A successful remote read populates earlier writable cache layers. Disk caching is never implicit — there is no disk anywhere in this module.

Process-backed stores have a two-shape API: start_link/1 starts and returns the provider supervisor pid, and handle/1 obtains the opaque ODB handle used by queries. Value stores such as from_objects/2 start no process and return their handle directly.

Summary

Types

A cache layer descriptor produced by cache/1.

t()

An opaque handle to an object store.

Functions

A writable in-memory cache layer for layer/1: stores verified, inflated object payloads under byte, entry, and per-object caps. Bytes enter only after a lower store's verify: :always path. Because process memory is not a new trust boundary, release-build hits are served without re-hashing. Debug builds verify on insertion and serving as a tripwire, not as the cache's trust guarantee. Never disk.

Builds a static in-memory ODB from an enumerable of Gitility.Object structs — a fixed store for tests, small generated repositories, and callers that already hold object data.

Returns the opaque ODB handle owned by a running provider supervisor.

Reads one object's header (type and size) without its payload.

Composes stores into one read-through ODB. Layers are queried in order; a hit in a later layer populates earlier writable cache layers when allowed. All layers must share a hash algorithm (:hash_mismatch) and runtime (:runtime_mismatch).

Reads one object, bounded. max_bytes: caps the inflated payload; exceeding it returns :object_too_large rather than a partial object. If limits.max_object_bytes is lower, that hard limit becomes the effective cap and an oversized object returns :object_too_large naming :max_object_bytes in the error details.

Reads a batch of objects, bounded by max_total_bytes:. Returns a map of OID to object or :not_found — per-object misses are results, not errors. The batch cap (max_total_bytes:) returns :result_too_large; exhaustion of the overall Limits.max_total_object_bytes budget returns :budget_exceeded.

Asks a provider backend to refresh availability state, bounded by its request_timeout, then clears the native negative cache only after the callback succeeds. A layered handle fans this out to every provider it contains, then refreshes every native layer. Verified positive cache entries remain valid because Git object IDs are immutable. A layered refresh returns :ok only when at least one layer accepts refresh; when every layer refuses, it returns :unsupported_operation.

Starts a provider-backed ODB serving objects through a Gitility.ODB.Backend implementation.

Returns the last completed PackFetch hydration or refresh statistics.

Types

cache_spec()

@opaque cache_spec()

A cache layer descriptor produced by cache/1.

The descriptor is opaque by convention, but Elixir cannot enforce the tuple boundary. Raw {:cache, opts} tuples are accepted and validated identically.

t()

@opaque t()

An opaque handle to an object store.

Carries the store's kind, its native or process reference, its hash algorithm, and its runtime affiliation (used to enforce :runtime_mismatch at composition time). Match on it only via this module's functions.

Functions

cache(opts)

@spec cache(keyword()) :: cache_spec()

A writable in-memory cache layer for layer/1: stores verified, inflated object payloads under byte, entry, and per-object caps. Bytes enter only after a lower store's verify: :always path. Because process memory is not a new trust boundary, release-build hits are served without re-hashing. Debug builds verify on insertion and serving as a tripwire, not as the cache's trust guarantee. Never disk.

Options

  • :max_bytes (required) — total payload ceiling.
  • :max_entries — entry-count ceiling (default 100_000).
  • :max_object_bytes — per-object cap (default: max_bytes); larger objects bypass the cache and remain readable from lower layers.

from_objects(objects, opts \\ [])

@spec from_objects(
  Enumerable.t(),
  keyword()
) :: {:ok, t()} | {:error, Gitility.Error.t()}

Builds a static in-memory ODB from an enumerable of Gitility.Object structs — a fixed store for tests, small generated repositories, and callers that already hold object data.

Distinct from cache/1, which is a writable cache layer; the two are deliberately not both called "memory".

Options

  • :hash:sha1 (default) or :sha256.
  • :verify:always (default) verifies every object at load.
  • :runtime — the Gitility.Runtime to attach to (default: shared).

handle(pid_or_name)

@spec handle(pid() | GenServer.name()) :: {:ok, t()} | {:error, Gitility.Error.t()}

Returns the opaque ODB handle owned by a running provider supervisor.

Accepts either the pid returned by start_link/1 or its configured :name. A handle is permanently bound to the current provider process; obtain a new handle after a provider restart.

header(odb, oid, opts \\ [])

@spec header(t(), Gitility.OID.t() | String.t(), keyword()) ::
  {:ok, Gitility.ObjectHeader.t()} | {:error, Gitility.Error.t()}

Reads one object's header (type and size) without its payload.

layer(layers)

@spec layer([t() | cache_spec()]) :: {:ok, t()} | {:error, Gitility.Error.t()}

Composes stores into one read-through ODB. Layers are queried in order; a hit in a later layer populates earlier writable cache layers when allowed. All layers must share a hash algorithm (:hash_mismatch) and runtime (:runtime_mismatch).

Header queries are answered by the cache when the object is resident, otherwise by the first layer that has it, without fetching payloads. Header queries never populate the cache.

A layer error fails the read and carries its zero-based position in error.details.layer; layers are not failover replicas. A hit before a failing layer still succeeds. Compose caches with one authoritative store; use supervision/retry at the store level for availability.

{:ok, odb} =
  Gitility.ODB.layer([
    Gitility.ODB.cache(max_bytes: 128 * 1024 * 1024),
    remote_odb
  ])

read(odb, oid, opts \\ [])

@spec read(t(), Gitility.OID.t() | String.t(), keyword()) ::
  {:ok, Gitility.Object.t()} | {:error, Gitility.Error.t()}

Reads one object, bounded. max_bytes: caps the inflated payload; exceeding it returns :object_too_large rather than a partial object. If limits.max_object_bytes is lower, that hard limit becomes the effective cap and an oversized object returns :object_too_large naming :max_object_bytes in the error details.

read_many(odb, oids, opts \\ [])

@spec read_many(t(), [Gitility.OID.t() | String.t()], keyword()) ::
  {:ok, %{required(Gitility.OID.t()) => Gitility.Object.t() | :not_found}}
  | {:error, Gitility.Error.t()}

Reads a batch of objects, bounded by max_total_bytes:. Returns a map of OID to object or :not_found — per-object misses are results, not errors. The batch cap (max_total_bytes:) returns :result_too_large; exhaustion of the overall Limits.max_total_object_bytes budget returns :budget_exceeded.

refresh(odb)

@spec refresh(t()) :: :ok | {:error, Gitility.Error.t()}

Asks a provider backend to refresh availability state, bounded by its request_timeout, then clears the native negative cache only after the callback succeeds. A layered handle fans this out to every provider it contains, then refreshes every native layer. Verified positive cache entries remain valid because Git object IDs are immutable. A layered refresh returns :ok only when at least one layer accepts refresh; when every layer refuses, it returns :unsupported_operation.

Local and static handles return :unsupported_operation in this milestone.

start_link(opts)

@spec start_link(keyword()) :: Supervisor.on_start()

Starts a provider-backed ODB serving objects through a Gitility.ODB.Backend implementation.

Returns the provider supervisor pid. Obtain the query handle with handle/1. The provider is a valid child — use {Gitility.ODB, opts} in a supervision tree. Pass a stable name: when supervised so other processes can call handle(name); direct starts without a name receive an internal generated name. Gitility monitors provider exit, fails pending requests with :provider_down, and cancels jobs that cannot progress.

A provider handle is permanently bound to the exact provider process that created it. If that process dies, pending and future reads through the old handle fail with retryable :provider_down; a restarted provider owns a new native handle, so callers must obtain a fresh ODB by calling start_link/1 again.

Options

  • :backend (required) — {module, init_arg}.
  • :name — provider-supervisor registered name (supports via tuples). When omitted, Gitility generates a private global name.
  • :hash:sha1 (default) or :sha256.
  • :verify:always (default): recompute and check every object ID.
  • :concurrency — max concurrent backend callbacks (default 8).
  • :request_timeout — per-batch deadline in ms (default 15_000). Expiry returns retryable :provider_timeout; the job's overall deadline remains :timeout and wins when both expire together.
  • :runtime — the Gitility.Runtime to attach to (default: shared).
  • :cache — provider-side cache: object_bytes:, header_entries:, negative_ttl: (ms; missing objects may arrive later in shallow or incrementally populated stores, so negatives expire fast).

Example

{:ok, provider} =
  Gitility.ODB.start_link(
    name: MyApp.GitObjects,
    backend: {MyCompany.GitObjectBackend, backend_options},
    concurrency: 8,
    cache: [object_bytes: 128 * 1024 * 1024]
  )

{:ok, odb} = Gitility.ODB.handle(provider)
{:ok, snapshot} = Gitility.Snapshot.open(odb, commit_oid)

In a supervision tree, retrieve the same handle by its stable name:

children = [
  {Gitility.ODB,
   name: MyApp.GitObjects,
   backend: {MyCompany.GitObjectBackend, backend_options}}
]

{:ok, _supervisor} = Supervisor.start_link(children, strategy: :one_for_one)
{:ok, odb} = Gitility.ODB.handle(MyApp.GitObjects)

stats(odb)

@spec stats(t()) :: {:ok, map()} | {:error, Gitility.Error.t()}

Returns the last completed PackFetch hydration or refresh statistics.