Makes each file match its library's storage profile and variant set
(V205, plan §6.3). Storage.Workers.ReconcileJob runs it.
A file records the profile and the variant set (each at a revision) it
was last placed by (placed_profile_uuid / placed_revision,
placed_variant_set_uuid / placed_variant_revision; NULL is the
Default at revision 1). It is stale (stale_query/0) when either
differs from what its library uses now: the library moved to another
profile or set, one of them changed, or an upload made fewer copies or
sizes than wanted (which stamps revision 0).
For a stale file, reconcile_file/1:
- locations (G4, G5, G6) — for each completed instance: an original
upload is an
:original, everything else (sizes, tiles, renders) is:derived(G13; an edit's hidden backup of the unedited original is an original). Its good copies are on the profile's buckets that store that kind and areactiveorread_only. While there are fewer than the profile wants (capped at the buckets it can use), the object is copied to more of its writable buckets and each copy is checked before it counts. Once enough are good, copies on buckets the profile no longer uses for it (not listed,draining, storing the other kind) are unlinked: the location row goes, and the object on that bucket only when nothing else there names the key (G11,Storage.unlink_location/2). - variants (G15, G16) — sizes of the set the file lacks are made,
those whose
spec_hashdiffers from the size's spec now are made again, and those the set no longer has (a size deleted from it; a disabled size is kept) are deleted.
A file that ends up matching both is stamped with what it now matches, at the revisions read when it was started (a change made meanwhile leaves it stale for the next pass). A failure leaves it stale; the next pass tries again. Instances not yet checked by the location backfill are skipped (their buckets are not known), and so is a file still uploading or with an image edit pending.
Summary
Functions
What each library has to do, from one grouped query: for every library with
at least one out-of-date file, %{out_of_date:, eligible:, failing:} —
The libraries that have files the reconciler may take now, as uuid strings.
The files whose placement or sizes are out of date — the revision of their
library's profile or variant set is not the one they were placed by —
whether or not the reconciler may take them now. A file waiting out its
retry window, one with an image edit pending: all are out of date. That is
what "this library is not up to date" means; stale_query/1 is the subset
the reconciler picks next (plan 2026-10-03-job-runs.md, R8).
Whether any file is stale.
Makes file match its library's profile and variant set, and stamps it
when it does. Returns :reconciled, :stale (something could not be
done yet; it stays stale) or :skipped (another reconciler holds it).
One reconciler at a time per file, across nodes: a session advisory lock
on the file, held while its bytes are copied.
Reconciles up to limit stale files after cursor (a file uuid; nil
from the start), in uuid order. Returns {:more, last_uuid, totals}
while a full batch was read, {:done, totals} once the walk has passed
the last one; totals counts files :reconciled, :stale (still) and
:skipped. library_uuid: walks one library's files only.
How many files are stale.
Up to limit stale files for the Health page: the file, its library's
name, and whether its copies (:placement) or its sizes (:variants)
are what is out of date.
The files that are not where, or not what, their library's profile and
variant set want — and that the reconciler may work on now: not a file
an upload is still processing, nor one with an image edit pending, nor one
tried in the last few minutes. This is the work-selection query; what a
library's state counts is out_of_date_query/1.
How many files the reconciler looks after (active and trashed ones).
Functions
What each library has to do, from one grouped query: for every library with
at least one out-of-date file, %{out_of_date:, eligible:, failing:} —
out_of_date— files not at their library's revisions (out_of_date_query/1);eligible— of those, the ones the reconciler may take now (stale_query/1);failing— of those, the ones it tried and could not finish in the last few minutes (they wait out@retry_after_secondsbefore the next try).
Keyed by the library's uuid as a string. A library absent from the map has nothing out of date. An optional uuid list limits the query to the libraries a screen is showing; omitted, it counts every library.
@spec libraries_with_work() :: [String.t()]
The libraries that have files the reconciler may take now, as uuid strings.
@spec out_of_date_query(keyword()) :: Ecto.Query.t()
The files whose placement or sizes are out of date — the revision of their
library's profile or variant set is not the one they were placed by —
whether or not the reconciler may take them now. A file waiting out its
retry window, one with an image edit pending: all are out of date. That is
what "this library is not up to date" means; stale_query/1 is the subset
the reconciler picks next (plan 2026-10-03-job-runs.md, R8).
Takes the same library_uuid: option.
Whether any file is stale.
@spec reconcile_file(PhoenixKit.Modules.Storage.File.t()) :: :reconciled | :stale | :skipped
Makes file match its library's profile and variant set, and stamps it
when it does. Returns :reconciled, :stale (something could not be
done yet; it stays stale) or :skipped (another reconciler holds it).
One reconciler at a time per file, across nodes: a session advisory lock
on the file, held while its bytes are copied.
@spec run_batch(String.t() | nil, pos_integer(), map(), keyword()) :: {:more, String.t(), map()} | {:done, map()}
Reconciles up to limit stale files after cursor (a file uuid; nil
from the start), in uuid order. Returns {:more, last_uuid, totals}
while a full batch was read, {:done, totals} once the walk has passed
the last one; totals counts files :reconciled, :stale (still) and
:skipped. library_uuid: walks one library's files only.
@spec stale_count(keyword()) :: non_neg_integer()
How many files are stale.
@spec stale_files(pos_integer()) :: [map()]
Up to limit stale files for the Health page: the file, its library's
name, and whether its copies (:placement) or its sizes (:variants)
are what is out of date.
@spec stale_query(keyword()) :: Ecto.Query.t()
The files that are not where, or not what, their library's profile and
variant set want — and that the reconciler may work on now: not a file
an upload is still processing, nor one with an image edit pending, nor one
tried in the last few minutes. This is the work-selection query; what a
library's state counts is out_of_date_query/1.
library_uuid: limits it to one library.
@spec total_count() :: non_neg_integer()
How many files the reconciler looks after (active and trashed ones).