PhoenixKit.Modules.Storage.Workers.ChecksumBackfillJob (phoenix_kit v2.52.2)

Copy Markdown View Source

Recomputes the MD5 checksums the upload API used to record (G12).

file_checksum is the dedup key's input, and every upload path but UploadController hashed with SHA-256, so the same bytes uploaded through the API and through the media browser were never recognised as the same file. The controller hashes with SHA-256 now; this job finds the rows it stored before (a 32-hex-character checksum), reads each file's original, and records its SHA-256 and the matching user_file_checksum (Storage.calculate_user_file_checksum/3, library-aware).

A row whose uploader already has the same bytes in the same library would collide on the dedup index: it is left as it is (it keeps not deduping, which is what it did before). A file whose original cannot be read is left too. Either is marked in its metadata ("checksum_backfill"), so a later pass does not download it again. The update is guarded on the old checksum, so a row changed meanwhile (an image edit) is not overwritten.

Throttled and self-queuing like LocationBackfillJob: one batch per run, the next five seconds later, queued after boot and by the daily trash prune while any MD5 row is left.

Summary

Functions

Starts a pass as a job run (Storage.Jobs.ChecksumBackfill) when any file has an MD5 checksum, unless one is already active. Never raises.

Whether any file still has an MD5 checksum to recompute.

How many files still have an MD5 checksum to recompute.

Runs a whole pass in the calling process; returns how many files were :updated, :duplicate (left, their uploader has the bytes already), :unreadable or :changed (edited meanwhile).

Functions

maybe_enqueue()

@spec maybe_enqueue() :: :queued | :nothing_to_do | :unavailable

Starts a pass as a job run (Storage.Jobs.ChecksumBackfill) when any file has an MD5 checksum, unless one is already active. Never raises.

pending?()

@spec pending?() :: boolean()

Whether any file still has an MD5 checksum to recompute.

pending_count()

@spec pending_count() :: non_neg_integer()

How many files still have an MD5 checksum to recompute.

run_pass()

@spec run_pass() :: %{required(atom()) => non_neg_integer()}

Runs a whole pass in the calling process; returns how many files were :updated, :duplicate (left, their uploader has the bytes already), :unreadable or :changed (edited meanwhile).