PhoenixKit. Modules. Storage. Workers. ChecksumBackfillJob
(phoenix_kit v2.41.0)
Copy Markdown
View Source
Recomputes the MD5 checksums the upload API used to record (G12).
file_checksum is the dedup key's input, and every upload path but
UploadController hashed with SHA-256, so the same bytes uploaded through
the API and through the media browser were never recognised as the same
file. The controller hashes with SHA-256 now; this job finds the rows it
stored before (a 32-hex-character checksum), reads each file's original,
and records its SHA-256 and the matching user_file_checksum
(Storage.calculate_user_file_checksum/3, library-aware).
A row whose uploader already has the same bytes in the same library would
collide on the dedup index: it is left as it is (it keeps not deduping,
which is what it did before). A file whose original cannot be read is
left too. Either is marked in its metadata ("checksum_backfill"), so a
later pass does not download it again. The update is guarded on the old checksum, so a row changed
meanwhile (an image edit) is not overwritten.
Throttled and self-queuing like LocationBackfillJob: one batch per run,
the next five seconds later, queued after boot and by the daily trash
prune while any MD5 row is left.
Summary
Functions
Queues a pass when any file has an MD5 checksum. Never raises.
Runs a whole pass in the calling process; returns how many files were
:updated, :duplicate (left, their uploader has the bytes already),
:unreadable or :changed (edited meanwhile).
Functions
@spec maybe_enqueue() :: :queued | :nothing_to_do | :unavailable
Queues a pass when any file has an MD5 checksum. Never raises.
@spec run_pass() :: %{required(atom()) => non_neg_integer()}
Runs a whole pass in the calling process; returns how many files were
:updated, :duplicate (left, their uploader has the bytes already),
:unreadable or :changed (edited meanwhile).