PhoenixKit. Modules. Storage. Workers. ChecksumBackfillJob
(phoenix_kit v2.52.2)
Copy Markdown
View Source
Recomputes the MD5 checksums the upload API used to record (G12).
file_checksum is the dedup key's input, and every upload path but
UploadController hashed with SHA-256, so the same bytes uploaded through
the API and through the media browser were never recognised as the same
file. The controller hashes with SHA-256 now; this job finds the rows it
stored before (a 32-hex-character checksum), reads each file's original,
and records its SHA-256 and the matching user_file_checksum
(Storage.calculate_user_file_checksum/3, library-aware).
A row whose uploader already has the same bytes in the same library would
collide on the dedup index: it is left as it is (it keeps not deduping,
which is what it did before). A file whose original cannot be read is
left too. Either is marked in its metadata ("checksum_backfill"), so a
later pass does not download it again. The update is guarded on the old checksum, so a row changed
meanwhile (an image edit) is not overwritten.
Throttled and self-queuing like LocationBackfillJob: one batch per run,
the next five seconds later, queued after boot and by the daily trash
prune while any MD5 row is left.
Summary
Functions
Starts a pass as a job run (Storage.Jobs.ChecksumBackfill) when any file has an
MD5 checksum, unless one is already active. Never raises.
Whether any file still has an MD5 checksum to recompute.
How many files still have an MD5 checksum to recompute.
Runs a whole pass in the calling process; returns how many files were
:updated, :duplicate (left, their uploader has the bytes already),
:unreadable or :changed (edited meanwhile).
Functions
@spec maybe_enqueue() :: :queued | :nothing_to_do | :unavailable
Starts a pass as a job run (Storage.Jobs.ChecksumBackfill) when any file has an
MD5 checksum, unless one is already active. Never raises.
@spec pending?() :: boolean()
Whether any file still has an MD5 checksum to recompute.
@spec pending_count() :: non_neg_integer()
How many files still have an MD5 checksum to recompute.
@spec run_pass() :: %{required(atom()) => non_neg_integer()}
Runs a whole pass in the calling process; returns how many files were
:updated, :duplicate (left, their uploader has the bytes already),
:unreadable or :changed (edited meanwhile).