Object databases: where Git objects come from.
An ODB is content-addressed and immutable-by-ID — it has no notion of
branches or names (that is Gitility.RefDB). A caller that already knows
a commit ID needs only an ODB; Gitility.Snapshot.open/2 takes one
directly.
All stores yield the same opaque t/0 handle, and every store answers
the same query API identically — storage never leaks into query
semantics. Stores:
- Local — a bare or normal repository directory, opened via
Gitility.Repository.open/2(worktree files are never read). - Static — a fixed in-memory object set (
from_objects/2): tests, generated repos, callers already holding objects. - Provider — objects served by your
Gitility.ODB.Backendimplementation (start_link/1): any storage, no filesystem. - PackFetch — immutable packs eagerly hydrated through a
Gitility.ODB.RangeBackend(Gitility.ODB.PackFetch.start_link/1). - Layered — read-through composition (
layer/1), typically acache/1layer in front of a remote store.
Layers are queried in order and must share a runtime and hash algorithm. A successful remote read populates earlier writable cache layers. Disk caching is never implicit — there is no disk anywhere in this module.
Process-backed stores have a two-shape API: start_link/1 starts and
returns the provider supervisor pid, and handle/1 obtains the opaque ODB
handle used by queries. Value stores such as from_objects/2 start no
process and return their handle directly.
Summary
Functions
A writable in-memory cache layer for layer/1: stores verified, inflated
object payloads under byte, entry, and per-object caps. Bytes enter only
after a lower store's verify: :always path. Because process memory is not
a new trust boundary, release-build hits are served without re-hashing.
Debug builds verify on insertion and serving as a tripwire, not as the
cache's trust guarantee. Never disk.
Builds a static in-memory ODB from an enumerable of Gitility.Object
structs — a fixed store for tests, small generated repositories, and
callers that already hold object data.
Returns the opaque ODB handle owned by a running provider supervisor.
Reads one object's header (type and size) without its payload.
Composes stores into one read-through ODB. Layers are queried in order;
a hit in a later layer populates earlier writable cache layers when
allowed. All layers must share a hash algorithm (:hash_mismatch) and
runtime (:runtime_mismatch).
Reads one object, bounded. max_bytes: caps the inflated payload;
exceeding it returns :object_too_large rather than a partial object. If
limits.max_object_bytes is lower, that hard limit becomes the effective
cap and an oversized object returns :object_too_large naming
:max_object_bytes in the error details.
Reads a batch of objects, bounded by max_total_bytes:. Returns a map of
OID to object or :not_found — per-object misses are results, not
errors. The batch cap (max_total_bytes:) returns :result_too_large;
exhaustion of the overall Limits.max_total_object_bytes budget returns
:budget_exceeded.
Asks a provider backend to refresh availability state, bounded by its
request_timeout, then clears the native negative cache only after the
callback succeeds. A layered handle fans this out to every provider it
contains, then refreshes every native layer. Verified positive cache entries
remain valid because Git object IDs are immutable. A layered refresh returns
:ok only when at least one layer accepts refresh; when every layer refuses,
it returns :unsupported_operation.
Starts a provider-backed ODB serving objects through a
Gitility.ODB.Backend implementation.
Returns the last completed PackFetch hydration or refresh statistics.
Types
@opaque cache_spec()
A cache layer descriptor produced by cache/1.
The descriptor is opaque by convention, but Elixir cannot enforce the
tuple boundary. Raw {:cache, opts} tuples are accepted and validated
identically.
@opaque t()
An opaque handle to an object store.
Carries the store's kind, its native or process reference, its hash
algorithm, and its runtime affiliation (used to enforce
:runtime_mismatch at composition time). Match on it only via this
module's functions.
Functions
@spec cache(keyword()) :: cache_spec()
A writable in-memory cache layer for layer/1: stores verified, inflated
object payloads under byte, entry, and per-object caps. Bytes enter only
after a lower store's verify: :always path. Because process memory is not
a new trust boundary, release-build hits are served without re-hashing.
Debug builds verify on insertion and serving as a tripwire, not as the
cache's trust guarantee. Never disk.
Options
:max_bytes(required) — total payload ceiling.:max_entries— entry-count ceiling (default100_000).:max_object_bytes— per-object cap (default:max_bytes); larger objects bypass the cache and remain readable from lower layers.
@spec from_objects( Enumerable.t(), keyword() ) :: {:ok, t()} | {:error, Gitility.Error.t()}
Builds a static in-memory ODB from an enumerable of Gitility.Object
structs — a fixed store for tests, small generated repositories, and
callers that already hold object data.
Distinct from cache/1, which is a writable cache layer; the two are
deliberately not both called "memory".
Options
:hash—:sha1(default) or:sha256.:verify—:always(default) verifies every object at load.:runtime— theGitility.Runtimeto attach to (default: shared).
@spec handle(pid() | GenServer.name()) :: {:ok, t()} | {:error, Gitility.Error.t()}
Returns the opaque ODB handle owned by a running provider supervisor.
Accepts either the pid returned by start_link/1 or its configured :name.
A handle is permanently bound to the current provider process; obtain a new
handle after a provider restart.
@spec header(t(), Gitility.OID.t() | String.t(), keyword()) :: {:ok, Gitility.ObjectHeader.t()} | {:error, Gitility.Error.t()}
Reads one object's header (type and size) without its payload.
@spec layer([t() | cache_spec()]) :: {:ok, t()} | {:error, Gitility.Error.t()}
Composes stores into one read-through ODB. Layers are queried in order;
a hit in a later layer populates earlier writable cache layers when
allowed. All layers must share a hash algorithm (:hash_mismatch) and
runtime (:runtime_mismatch).
Header queries are answered by the cache when the object is resident, otherwise by the first layer that has it, without fetching payloads. Header queries never populate the cache.
A layer error fails the read and carries its zero-based position in
error.details.layer; layers are not failover replicas. A hit before a
failing layer still succeeds. Compose caches with one authoritative store;
use supervision/retry at the store level for availability.
{:ok, odb} =
Gitility.ODB.layer([
Gitility.ODB.cache(max_bytes: 128 * 1024 * 1024),
remote_odb
])
@spec read(t(), Gitility.OID.t() | String.t(), keyword()) :: {:ok, Gitility.Object.t()} | {:error, Gitility.Error.t()}
Reads one object, bounded. max_bytes: caps the inflated payload;
exceeding it returns :object_too_large rather than a partial object. If
limits.max_object_bytes is lower, that hard limit becomes the effective
cap and an oversized object returns :object_too_large naming
:max_object_bytes in the error details.
@spec read_many(t(), [Gitility.OID.t() | String.t()], keyword()) :: {:ok, %{required(Gitility.OID.t()) => Gitility.Object.t() | :not_found}} | {:error, Gitility.Error.t()}
Reads a batch of objects, bounded by max_total_bytes:. Returns a map of
OID to object or :not_found — per-object misses are results, not
errors. The batch cap (max_total_bytes:) returns :result_too_large;
exhaustion of the overall Limits.max_total_object_bytes budget returns
:budget_exceeded.
@spec refresh(t()) :: :ok | {:error, Gitility.Error.t()}
Asks a provider backend to refresh availability state, bounded by its
request_timeout, then clears the native negative cache only after the
callback succeeds. A layered handle fans this out to every provider it
contains, then refreshes every native layer. Verified positive cache entries
remain valid because Git object IDs are immutable. A layered refresh returns
:ok only when at least one layer accepts refresh; when every layer refuses,
it returns :unsupported_operation.
Local and static handles return :unsupported_operation in this milestone.
@spec start_link(keyword()) :: Supervisor.on_start()
Starts a provider-backed ODB serving objects through a
Gitility.ODB.Backend implementation.
Returns the provider supervisor pid. Obtain the query handle with
handle/1. The provider is a valid child — use {Gitility.ODB, opts} in a
supervision tree. Pass a stable name: when supervised so other processes
can call handle(name); direct starts without a name receive an internal
generated name. Gitility monitors provider exit, fails pending requests
with :provider_down, and cancels jobs that cannot progress.
A provider handle is permanently bound to the exact provider process that
created it. If that process dies, pending and future reads through the old
handle fail with retryable :provider_down; a restarted provider owns a new
native handle, so callers must obtain a fresh ODB by calling start_link/1
again.
Options
:backend(required) —{module, init_arg}.:name— provider-supervisor registered name (supports via tuples). When omitted, Gitility generates a private global name.:hash—:sha1(default) or:sha256.:verify—:always(default): recompute and check every object ID.:concurrency— max concurrent backend callbacks (default8).:request_timeout— per-batch deadline in ms (default15_000). Expiry returns retryable:provider_timeout; the job's overall deadline remains:timeoutand wins when both expire together.:runtime— theGitility.Runtimeto attach to (default: shared).:cache— provider-side cache:object_bytes:,header_entries:,negative_ttl:(ms; missing objects may arrive later in shallow or incrementally populated stores, so negatives expire fast).
Example
{:ok, provider} =
Gitility.ODB.start_link(
name: MyApp.GitObjects,
backend: {MyCompany.GitObjectBackend, backend_options},
concurrency: 8,
cache: [object_bytes: 128 * 1024 * 1024]
)
{:ok, odb} = Gitility.ODB.handle(provider)
{:ok, snapshot} = Gitility.Snapshot.open(odb, commit_oid)In a supervision tree, retrieve the same handle by its stable name:
children = [
{Gitility.ODB,
name: MyApp.GitObjects,
backend: {MyCompany.GitObjectBackend, backend_options}}
]
{:ok, _supervisor} = Supervisor.start_link(children, strategy: :one_for_one)
{:ok, odb} = Gitility.ODB.handle(MyApp.GitObjects)
@spec stats(t()) :: {:ok, map()} | {:error, Gitility.Error.t()}
Returns the last completed PackFetch hydration or refresh statistics.