Storage manager for handling file operations with redundancy and failover.
This module coordinates file storage across multiple buckets with automatic redundancy, failover, and variant generation.
Where an object is read from (V204)
Reads, existence checks, public URLs and serving go to the buckets the
object's location rows name first (PhoenixKit.Modules.Storage.Locations),
in the usual order: local buckets, then by priority. Only when none of
them has it are the other enabled buckets tried, and a bucket found to
hold it that way is recorded as a location. Deletes still remove an
unreferenced key from every enabled bucket.
Where an object is written (V205)
By the storage profile of the file's library (store_file/2): its
buckets, their roles and write priorities, and its copy counts.
Summary
Functions
The SHA-256 (lower-case hex) and size of the object at key in bucket
alone, read from that bucket and from no other: what "verify this copy"
needs, since retrieve_file/2 would fail over to a bucket that has it.
Copies the object at key from bucket from to bucket to, reading from
from alone (retrieve_file/2 would fail over to whichever bucket answers
first, perhaps the damaged one) and replacing what to holds under that key.
What repairs a copy that is missing or does not match its checksum from a
good one.
Deletes a file from all storage buckets.
Deletes the object at key from bucket only, never from any other
bucket. Storage.unlink_location/2 is what decides that nothing else on
that bucket needs it (G11).
Checks if a file exists in any storage bucket.
Returns file access info based on bucket access_type.
Returns the local filesystem path for a file if stored locally.
Whether bucket has the object at key: the reconciler's check that a
copy it made is really there before it unlinks another. A bucket that
raises does not have it.
Drops the cached list of enabled buckets, so the next read sees the database. Called whenever a bucket is created, updated or deleted. The cache only spares a query per file request; it expires on its own after five minutes, which is how long a bucket edit used to take to apply.
Gets a public URL for a file from the highest priority bucket that has it.
Replicates a file to specific target buckets.
Retrieves a file from storage with failover.
Stores a file across buckets.
Functions
@spec checksum_in(PhoenixKit.Modules.Storage.Bucket.t(), String.t()) :: {:ok, %{checksum: String.t(), size: non_neg_integer()}} | {:error, term()}
The SHA-256 (lower-case hex) and size of the object at key in bucket
alone, read from that bucket and from no other: what "verify this copy"
needs, since retrieve_file/2 would fail over to a bucket that has it.
{:error, :not_found} when the bucket does not hold it, {:error, reason}
when it could not be read. The object is streamed through the hash and the
temporary copy is removed.
@spec copy_object( PhoenixKit.Modules.Storage.Bucket.t(), PhoenixKit.Modules.Storage.Bucket.t(), String.t() ) :: :ok | {:error, term()}
Copies the object at key from bucket from to bucket to, reading from
from alone (retrieve_file/2 would fail over to whichever bucket answers
first, perhaps the damaged one) and replacing what to holds under that key.
What repairs a copy that is missing or does not match its checksum from a
good one.
Deletes a file from all storage buckets.
Deletes the object at key from bucket only, never from any other
bucket. Storage.unlink_location/2 is what decides that nothing else on
that bucket needs it (G11).
Checks if a file exists in any storage bucket.
Returns file access info based on bucket access_type.
Determines how a file should be served based on its storage location:
- Local files are always served directly
- Remote files depend on bucket's access_type setting
Returns
{:local, path}- File is local, serve directly from filesystem{:redirect, url}- Redirect to public URL (for public buckets){:signed_redirect, url}- Redirect to a short-lived signed URL (a public bucket, with:downloadgiven){:proxy, file_name}- Download and proxy through server (for private buckets){:error, :not_found}- File not found in any bucket
Options
:download- response headers (:disposition,:content_type) for a file that must download rather than render. A public bucket's plain object URL answers with the object's stored headers, so for such a file a public bucket hands out a signed URL that overrides them (PhoenixKit.Modules.Storage.Provider.signed_download_url/3), or is proxied when its provider cannot sign one.
Returns the local filesystem path for a file if stored locally.
This function checks enabled buckets with "local" provider and returns the direct path to the file if it exists. This allows serving files directly without copying to a temp file first.
Returns
{:ok, path}- File exists at the local path{:error, :not_local}- No local bucket contains the file
Whether bucket has the object at key: the reconciler's check that a
copy it made is really there before it unlinks another. A bucket that
raises does not have it.
Drops the cached list of enabled buckets, so the next read sees the database. Called whenever a bucket is created, updated or deleted. The cache only spares a query per file request; it expires on its own after five minutes, which is how long a bucket edit used to take to apply.
Gets a public URL for a file from the highest priority bucket that has it.
Replicates a file to specific target buckets.
Retrieves the file from any available bucket, then stores it to each target bucket. Returns the list of bucket UUIDs that succeeded.
Retrieves a file from storage with failover.
Tries each bucket in priority order until the file is found.
Stores a file across buckets.
Where it goes (V205)
By a storage profile: :profile (a StorageProfile with its buckets
preloaded, Storage.Profiles.for_library/1), or the Default profile
when none is given. :kind says what the object is: :original (an
original upload, the default) or :derived (a size, a tile, a render),
which only matters to the copies an upload must make. Every file gets the
profile's copies per kind of bucket: copies_local on local buckets and
copies_cloud on cloud ones. Buckets that are enabled, active in the
profile and not over their max_size_mb are written, primaries before
replicas before backups, each role by fixed write priority and then in
random order, up to the count for their kind.
An original fails unless at least the profile's min_copies_on_write
copies were written (what was written is removed again). The result
says whether every copy the profile wants was made (complete?); the
rest are made by the reconciler.
Options
:path_prefix- the object key:profile/:kind- see above:force_bucket_ids- exactly these buckets (every one that is enabled, in the order given), whatever any profile says:redundancy_copies/:priority_buckets- the pre-profile selection (the enabled buckets by priority, up to the count), kept for callers that ask for it
Returns
{:ok, file_result}-destination_path,bucket_ids(the buckets written),stored_in,successful_storages,complete?{:error, reason}- Failed to store file