Makes each file match its library's storage profile and variant set
(V205, plan §6.3). Storage.Workers.ReconcileJob runs it.
A file records the profile and the variant set (each at a revision) it
was last placed by (placed_profile_uuid / placed_revision,
placed_variant_set_uuid / placed_variant_revision; NULL is the
Default at revision 1). It is stale (stale_query/0) when either
differs from what its library uses now: the library moved to another
profile or set, one of them changed, or an upload made fewer copies or
sizes than wanted (which stamps revision 0).
For a stale file, reconcile_file/1:
- locations (G4, G5, G6) — for each completed instance: an original
upload is an
:original, everything else (sizes, tiles, renders) is:derived(G13; an edit's hidden backup of the unedited original is an original). Its good copies are on the profile's buckets that store that kind and areactiveorread_only. While there are fewer than the profile wants (capped at the buckets it can use), the object is copied to more of its writable buckets and each copy is checked before it counts. Once enough are good, copies on buckets the profile no longer uses for it (not listed,draining, storing the other kind) are unlinked: the location row goes, and the object on that bucket only when nothing else there names the key (G11,Storage.unlink_location/2). - variants (G15, G16) — sizes of the set the file lacks are made,
those whose
spec_hashdiffers from the size's spec now are made again, and those the set no longer has (a size deleted from it; a disabled size is kept) are deleted.
A file that ends up matching both is stamped with what it now matches, at the revisions read when it was started (a change made meanwhile leaves it stale for the next pass). A failure leaves it stale; the next pass tries again. Instances not yet checked by the location backfill are skipped (their buckets are not known), and so is a file still uploading or with an image edit pending.
Summary
Functions
Whether any file is stale.
Makes file match its library's profile and variant set, and stamps it
when it does. Returns :reconciled, :stale (something could not be
done yet; it stays stale) or :skipped (another reconciler holds it).
One reconciler at a time per file, across nodes: a session advisory lock
on the file, held while its bytes are copied.
Reconciles up to limit stale files after cursor (a file uuid; nil
from the start), in uuid order. Returns {:more, last_uuid, totals}
while a full batch was read, {:done, totals} once the walk has passed
the last one; totals counts files :reconciled, :stale (still) and
:skipped.
How many files are stale.
Up to limit stale files for the Health page: the file, its library's
name, and whether its copies (:placement) or its sizes (:variants)
are what is out of date.
The files that are not where, or not what, their library's profile and variant set want. Not a file an upload is still processing, nor one with an image edit pending, nor one tried in the last few minutes.
How many files the reconciler looks after (active and trashed ones).
Functions
@spec pending?() :: boolean()
Whether any file is stale.
@spec reconcile_file(PhoenixKit.Modules.Storage.File.t()) :: :reconciled | :stale | :skipped
Makes file match its library's profile and variant set, and stamps it
when it does. Returns :reconciled, :stale (something could not be
done yet; it stays stale) or :skipped (another reconciler holds it).
One reconciler at a time per file, across nodes: a session advisory lock
on the file, held while its bytes are copied.
@spec run_batch(String.t() | nil, pos_integer(), map()) :: {:more, String.t(), map()} | {:done, map()}
Reconciles up to limit stale files after cursor (a file uuid; nil
from the start), in uuid order. Returns {:more, last_uuid, totals}
while a full batch was read, {:done, totals} once the walk has passed
the last one; totals counts files :reconciled, :stale (still) and
:skipped.
@spec stale_count() :: non_neg_integer()
How many files are stale.
@spec stale_files(pos_integer()) :: [map()]
Up to limit stale files for the Health page: the file, its library's
name, and whether its copies (:placement) or its sizes (:variants)
are what is out of date.
@spec stale_query() :: Ecto.Query.t()
The files that are not where, or not what, their library's profile and variant set want. Not a file an upload is still processing, nor one with an image edit pending, nor one tried in the last few minutes.
@spec total_count() :: non_neg_integer()
How many files the reconciler looks after (active and trashed ones).