A content-addressed store on disk, safe to share between OS processes: large values kept by digest, an action cache, verifying traces, and scratch directories, with a garbage collector that never takes what a reader is using.
{:ok, store} = Roux.Blob.open("~/.cache/my_tool/store")
{:ok, digest} = Roux.Blob.put_term(store, big_value)
{:ok, ^big_value} = Roux.Blob.get_term(store, digest)Layout
A store is a directory:
FORMAT— the layout's version; a store of another is refused;cas/<aa>/<digest>— content-addressed entries (put/2,adopt/2,get/2,link/3), named by the SHA-256 of their bytes in lowercase hex,aaits first two digits;ac/<aa>/<key digest>— the action cache (remember/3,recall/2): a term stored under any key;traces/<name digest>/<trace digest>— verifying traces (Roux.Blob.Trace);roots/<owner digest>— the digests an owner keeps alive (retain/3), a manifest's among them;scratch/<os pid>.<token>-<n>/— a directory per use (scratch/2), named after the OS process and a token of its VM's own;tmp/— files being written;trash/— entries being removed, a CAS entry under a name that begins with its digest.
Entries
An entry is immutable. It is written under a name of its own in
tmp/ and hard-linked into place, so a reader finds it whole or not
at all. A CAS entry is never replaced: a writer that finds its name
taken (EEXIST) has found the same bytes — content addressing makes
"already there" always right — and leaves them. A rename that replaces
a name is not atomic everywhere: on APFS a concurrent link(2) or open
finds no name for a moment, so replacing the empty relation file that
every solve links made those links fail ({:error, :enoent}) while
the entry was there all along. CAS entries are read-only on disk: a
hard link to one (link/3) shares its inode, and a program writing
into the link would write into the store.
The names that are replaced on purpose — an action-cache entry, a trace, a root, whose value can change — are renamed into place, and only when their bytes change (an equal write refreshes the file instead). Their readers take a name missing for that moment for a miss, never an error: a recall misses, a trace lookup passes over the trace, and a collection that cannot read a root it listed sweeps nothing. A reader that finds an entry gone, or not holding what its name says, takes it for a miss (and a corrupt one is taken out of its name, so the next write replaces it).
Trust
A store has the manifest's trust model: its files were written by this
tool, for this user, on this machine. Terms are decoded as the
manifest decodes them — without :safe, so a term naming an atom the
VM has not created yet (a function name in a stored finding) decodes
in a fresh VM instead of missing. What makes that trust good is who
can write the files, and open/1 checks exactly that: it refuses a
root, or a FORMAT file, that the current OS user does not own or
that is writable by its group or by everyone (Roux.Blob.TrustError).
A symbolic link as the root is followed, and its target checked the
same way. A root open/1 creates is made 0700. Bytes that do not
decode — a truncated or corrupt file — are still a miss, never a
crash.
Raw I/O, and touches that refresh
Every file operation of the store is raw (Roux.Blob.IO): it goes
straight to the operating system from the calling process, never
through the VM's one file server, so a fan-out of lookups
(Roux.Runtime.parallel/3) runs side by side.
A lookup that finds an entry — a put/2 of bytes already there, a
recall/2 hit, a trace found (Roux.Blob.Trace) — refreshes its
modification time, which is what a collection and a trace prune read
as "in use". It does so only when that time is older than the store's
refresh: interval (an hour by default, open/2): a warm run that
reads the same entries again writes nothing. Recency is therefore
known to within the interval, which is why a collection's grace and
keep periods are never shorter than it (a shorter one is taken as the
interval).
Collection
An entry is swept by renaming it aside — into trash/, under a name
that begins with its digest — then looking at it again: one touched
meanwhile is linked back under its name. For that moment the name is
missing, so a reader of a missing entry looks for its aside copy too
(get/2, link/3): the inode is always under one of the two names,
and a process that found an entry present (a put/2, whose finding
marks it used) can always link it.
gc/2 marks from the live roots — every owner's retained digests
(retain/3), and the digests named by action-cache entries and
traces used recently — and sweeps what is left and older than a grace
period. An entry is renamed aside before it is deleted, and put back
when it was touched meanwhile: a put/2 of bytes already there, or a
recall/2, touches the entry it finds. The grace period must be
longer than the longest run between writing an entry and retaining
it: a run's own entries are young until it retains them.
Summary
Types
An entry's name: the SHA-256 of its bytes, lowercase hex.
Options of gc/2
What a collection did.
A store: its root directory, whether temporary/0 made it, and the
interval within which a hit leaves an entry's modification time alone
(see "Raw I/O, and touches that refresh").
Functions
Moves the file at path into the store, by rename (the file must be on
the store's file system; across file systems it is copied), and
returns its digest. The file is gone from path afterwards.
The value remembered under key, or fun's, remembered — unless it
is :error or {:error, _}: a failure is never kept, so the next
call tries again.
Removes a store and everything in it.
term as put_term/2 stores it, and the digest it is stored under:
equal terms encode to equal bytes, so two digests compare values.
The bytes stored under digest, raising Roux.Blob.MissingError
when there are none.
Collects the store: marks what the live roots name (see the
moduledoc), and removes every other entry older than the grace period,
every action-cache entry and trace unused for longer than keep:, and
scratch directories and writes a dead process left behind. A grace or
keep period shorter than the store's refresh interval is taken as the
interval: an entry in use may look that much older than it is.
The bytes stored under digest, or :miss when there are none — gone,
or not what the digest names (the entry is then taken out of its name,
so the next write replaces it).
The term stored under digest (put_term/2), or :miss — including
for bytes that do not decode as a term. Decoded as the manifest
decodes, atoms and all (see "Trust").
Makes dest name the entry of digest: a hard link, or a copy when
dest is on another file system — never a symbolic link, which a
collection could leave dangling. The linked file is read-only.
Collects the store (gc/2) when the last collection was longer than
every: seconds ago (default a day) and no other process is
collecting it; :skipped otherwise.
Whether digest is stored (without reading it).
Opens the store at root, creating it (mode 0700) when it is not
there. Refuses a directory holding a store of another layout
(FORMAT), and one another user could have written: see "Trust".
Opens the store at root (open/2), raising on failure.
Where the entry of digest lives (whether or not it is there).
Stores data and returns its digest. Bytes already stored are not
written again; their entry is touched, so a collection in progress
keeps it.
Stores encoded under digest, both as encode_term/1 gave them:
for a caller that encoded a term to compare digests and keeps it.
Stores term, encoded deterministically (encode_term/1), and
returns its digest.
The value remembered under key (any term), or :miss. A hit
refreshes the entry (see "Raw I/O, and touches that refresh"): a
collection keeps what recently used entries name.
Forgets what owner kept alive (retain/3).
Remembers value under key (replacing what was there).
Makes digests the entries owner keeps alive, replacing what it
kept before: a collection sweeps none of them. An owner is any term;
an owner that is a string is a path (a manifest's), and is dropped by
the collection once nothing is at that path.
A store of its own in the system's temporary directory (mode 0700),
for a run that keeps nothing: destroy/1 removes it.
Types
@type digest() :: String.t()
An entry's name: the SHA-256 of its bytes, lowercase hex.
@type gc_option() :: {:grace, non_neg_integer()} | {:keep, non_neg_integer()}
Options of gc/2:
:grace— seconds an unmarked entry is kept after it was last written or touched (default a day);:keep— seconds an action-cache entry or trace is kept after it was last used, and counts as a root (default a week).
@type gc_stats() :: %{ removed: non_neg_integer(), bytes: non_neg_integer(), kept: non_neg_integer() }
What a collection did.
@type t() :: %Roux.Blob{ refresh: non_neg_integer(), root: Path.t(), temporary?: boolean() }
A store: its root directory, whether temporary/0 made it, and the
interval within which a hit leaves an entry's modification time alone
(see "Raw I/O, and touches that refresh").
Functions
@spec adopt(t(), Path.t()) :: {:ok, digest()} | {:error, File.posix()}
Moves the file at path into the store, by rename (the file must be on
the store's file system; across file systems it is copied), and
returns its digest. The file is gone from path afterwards.
The value remembered under key, or fun's, remembered — unless it
is :error or {:error, _}: a failure is never kept, so the next
call tries again.
@spec destroy(t()) :: :ok
Removes a store and everything in it.
term as put_term/2 stores it, and the digest it is stored under:
equal terms encode to equal bytes, so two digests compare values.
The bytes stored under digest, raising Roux.Blob.MissingError
when there are none.
Collects the store: marks what the live roots name (see the
moduledoc), and removes every other entry older than the grace period,
every action-cache entry and trace unused for longer than keep:, and
scratch directories and writes a dead process left behind. A grace or
keep period shorter than the store's refresh interval is taken as the
interval: an entry in use may look that much older than it is.
The bytes stored under digest, or :miss when there are none — gone,
or not what the digest names (the entry is then taken out of its name,
so the next write replaces it).
The term stored under digest (put_term/2), or :miss — including
for bytes that do not decode as a term. Decoded as the manifest
decodes, atoms and all (see "Trust").
@spec link(t(), digest(), Path.t()) :: :ok | {:error, :missing | File.posix()}
Makes dest name the entry of digest: a hard link, or a copy when
dest is on another file system — never a symbolic link, which a
collection could leave dangling. The linked file is read-only.
@spec maybe_gc(t(), [gc_option() | {:every, non_neg_integer()}]) :: {:ok, gc_stats()} | :skipped
Collects the store (gc/2) when the last collection was longer than
every: seconds ago (default a day) and no other process is
collecting it; :skipped otherwise.
Whether digest is stored (without reading it).
@spec open( Path.t(), keyword() ) :: {:ok, t()} | {:error, {:format, String.t()} | Roux.Blob.TrustError.t() | File.posix()}
Opens the store at root, creating it (mode 0700) when it is not
there. Refuses a directory holding a store of another layout
(FORMAT), and one another user could have written: see "Trust".
Options
:refresh— seconds within which a hit leaves an entry's modification time alone (default an hour; see "Raw I/O, and touches that refresh").
Opens the store at root (open/2), raising on failure.
Where the entry of digest lives (whether or not it is there).
@spec put(t(), iodata()) :: {:ok, digest()} | {:error, File.posix()}
Stores data and returns its digest. Bytes already stored are not
written again; their entry is touched, so a collection in progress
keeps it.
@spec put_encoded_term(t(), digest(), binary()) :: {:ok, digest()} | {:error, File.posix()}
Stores encoded under digest, both as encode_term/1 gave them:
for a caller that encoded a term to compare digests and keeps it.
@spec put_term(t(), term()) :: {:ok, digest()} | {:error, File.posix()}
Stores term, encoded deterministically (encode_term/1), and
returns its digest.
The value remembered under key (any term), or :miss. A hit
refreshes the entry (see "Raw I/O, and touches that refresh"): a
collection keeps what recently used entries name.
Forgets what owner kept alive (retain/3).
@spec remember(t(), term(), term()) :: :ok | {:error, File.posix()}
Remembers value under key (replacing what was there).
@spec retain(t(), term(), [digest()]) :: :ok | {:error, File.posix()}
Makes digests the entries owner keeps alive, replacing what it
kept before: a collection sweeps none of them. An owner is any term;
an owner that is a string is a path (a manifest's), and is dropped by
the collection once nothing is at that path.
Runs fun with a directory of its own under the store (so on its file
system: what fun writes there can be adopt/2ed, and entries
link/3ed in), removed when fun returns or raises. A directory
whose owner died without removing it is collected after a day.
The directory is new: made exclusively, under a name no other process
of the store uses (Roux.Blob.IO.ospid/0: the OS pid and a token of
the VM's own), and another name tried should it be there anyway. A
directory that was there already would be a dead process's leftover,
which a collection that found it a day old removes, contents and all.
@spec temporary() :: t()
A store of its own in the system's temporary directory (mode 0700),
for a run that keeps nothing: destroy/1 removes it.