Buckets.Cloud behaviour (buckets v1.0.0-rc.2)
Defines a cloud.
A cloud manages the movement of files and data between your application and a remote bucket for persistent storage.
When used, it expects :otp_app
as an option:
defmodule MyApp.Cloud do
use Buckets.Cloud,
otp_app: :my_app
end
Configuration is fetched from the application config, using a combination of
:otp_app
and the module that you defined:
config :my_app, MyApp.Cloud,
adapter: Buckets.Adapters.Volume,
bucket: "tmp/buckets_volume",
base_url: "http://localhost:4000"
Each Cloud module corresponds to a single storage backend, similar to how Ecto.Repo modules correspond to a single database. For multi-cloud applications, define multiple Cloud modules:
defmodule MyApp.VolumeCloud do
use Buckets.Cloud, otp_app: :my_app
end
defmodule MyApp.GCSCloud do
use Buckets.Cloud, otp_app: :my_app
end
defmodule MyApp.S3Cloud do
use Buckets.Cloud, otp_app: :my_app
end
You may also specify config dynamically at runtime, using the :config
opt
where it is supported.
Supervision
Cloud modules start a supervisor that manages any background processes required by the configured adapter. Some adapters (like GCS) need authentication servers, while others (like Volume, S3) don't need any supervised processes.
Only add Cloud modules to your supervision tree if they need background processes. If you add a Cloud module that doesn't need supervision (Volume, S3), you'll see a warning message suggesting you remove it to avoid unnecessary overhead.
# Only needed for adapters that require background processes (like GCS)
children = [
MyApp.GCSCloud # GCS needs auth servers
# MyApp.VolumeCloud - Not needed, would show warning
# MyApp.S3Cloud - Not needed, would show warning
]
Dynamic Configuration
For multi-tenant applications where cloud configurations are determined at runtime, every Cloud module supports dynamic configuration using the process dictionary, similar to Ecto's dynamic repositories.
Usage
There are two ways to use dynamic configuration:
1. Scoped Configuration (like Ecto transactions)
Use with_config/2
for temporary configuration:
# Define the runtime configuration
config = [
adapter: Buckets.Adapters.S3,
bucket: "user-bucket",
access_key_id: "AKIA...",
secret_access_key: "secret...",
region: "us-east-1"
]
# Execute operations with the dynamic config
{:ok, object} = MyApp.Cloud.with_config(config, fn ->
MyApp.Cloud.insert("file.pdf")
MyApp.Cloud.insert("another.pdf") # Same config
end)
2. Process-Local Configuration (like Ecto.Repo.put_dynamic_repo)
Use put_dynamic_config/1
for persistent configuration in the current process:
# Set dynamic config for this process
:ok = MyApp.Cloud.put_dynamic_config([
adapter: Buckets.Adapters.GCS,
bucket: "tenant-specific-bucket",
service_account_credentials: tenant.credentials
])
# All subsequent operations use the dynamic config
{:ok, object1} = MyApp.Cloud.insert("file1.pdf")
{:ok, object2} = MyApp.Cloud.insert("file2.pdf")
Auth Server Management
Auth servers (for GCS) are automatically started and cached per-process as needed. You don't need any special configuration or supervision setup for dynamic clouds.
Summary
Callbacks
Deletes a Buckets.Object
permanently.
Same as delete/1
but returns the object or raises if there is an error.
Inserts a Buckets.Object
or a file from a path into a bucket.
Same as insert/2
but returns the object or raises if there is an error.
Returns a map to be used as configuration for a LiveView live upload. The configuration
contains a signed URL that permits the upload to a remote bucket. Requires that an :uploader
is configured for the location that the file is being uploaded to.
Same as c:live_upload/1
but returns the upload config or raises if there is an error.
Loads the data for a Buckets.Object
, placing it in memory by default. It will load data
lazily, doing nothing if data is already present.
Same as c:load/1
but returns the object or raises if there is an error.
An overridable function that processes filenames before encoding them in a remote path.
Reads the data for Buckets.Object
, preferring local data first and fetching from
the remote bucket if needed.
Same as read/1
but returns the data or raises if there is an error.
An overridable function that specifies the temporary directory in which objects are stored.
Unloads the data for a Buckets.Object
. If the data is stored in a local file, the file will
be deleted.
Returns a SignedURL struct for a Buckets.Object
.
Same as url/1
but returns the SignedURL raises if there is an error.
Callbacks
delete(object)
@callback delete(object :: Buckets.Object.t()) :: {:ok, Buckets.Object.t()} | {:error, term()}
Deletes a Buckets.Object
permanently.
delete!(object)
@callback delete!(object :: Buckets.Object.t()) :: Buckets.Object.t()
Same as delete/1
but returns the object or raises if there is an error.
insert(object_or_path, opts)
@callback insert( object_or_path :: Buckets.Object.t() | String.t(), opts :: Keyword.t() ) :: {:ok, Buckets.Object.t()} | {:error, term()}
Inserts a Buckets.Object
or a file from a path into a bucket.
insert!(object_or_path, opts)
@callback insert!( object_or_path :: Buckets.Object.t() | String.t(), opts :: Keyword.t() ) :: Buckets.Object.t()
Same as insert/2
but returns the object or raises if there is an error.
live_upload(entry, opts)
@callback live_upload( entry :: Phoenix.LiveView.UploadEntry.t(), opts :: Keyword.t() ) :: {:ok, map()} | {:error, term()}
Returns a map to be used as configuration for a LiveView live upload. The configuration
contains a signed URL that permits the upload to a remote bucket. Requires that an :uploader
is configured for the location that the file is being uploaded to.
live_upload!(entry, opts)
@callback live_upload!( entry :: Phoenix.LiveView.UploadEntry.t(), opts :: Keyword.t() ) :: map()
Same as c:live_upload/1
but returns the upload config or raises if there is an error.
load(object, opts)
@callback load( object :: Buckets.Object.t(), opts :: Keyword.t() ) :: {:ok, Buckets.Object.t()} | {:error, term()}
Loads the data for a Buckets.Object
, placing it in memory by default. It will load data
lazily, doing nothing if data is already present.
Options
* `:to` - A location to store the loaded data to on disk, if this is preferred over
loading to memory. May be a path, `:tmp` (to load to the configured tmp dir), or
`{:tmp, path}` (to load to a path in the configured tmp dir).
* `:force` - If data is already present, first unload it with `c:unload/1` before loading
new data. Warning: if data is stored in a file, it will be deleted.
load!(object, opts)
@callback load!(object :: Buckets.Object.t(), opts :: Keyword.t()) :: Buckets.Object.t()
Same as c:load/1
but returns the object or raises if there is an error.
normalize_filename(filename)
An overridable function that processes filenames before encoding them in a remote path.
By default, replaces whitespace with "_"
and removes all non-alphanumeric characters.
read(object)
@callback read(object :: Buckets.Object.t()) :: {:ok, binary()} | {:error, term()}
Reads the data for Buckets.Object
, preferring local data first and fetching from
the remote bucket if needed.
read!(object)
@callback read!(object :: Buckets.Object.t()) :: binary()
Same as read/1
but returns the data or raises if there is an error.
tmp_dir()
@callback tmp_dir() :: String.t()
An overridable function that specifies the temporary directory in which objects are stored.
Defaults to System.tmp_dir!()
.
unload(object)
@callback unload(object :: Buckets.Object.t()) :: Buckets.Object.t()
Unloads the data for a Buckets.Object
. If the data is stored in a local file, the file will
be deleted.
url(object)
@callback url(object :: Buckets.Object.t()) :: Buckets.Object.t()
Returns a SignedURL struct for a Buckets.Object
.
url!(object, opts)
@callback url!(object :: Buckets.Object.t(), opts :: Keyword.t()) :: Buckets.Object.t()
Same as url/1
but returns the SignedURL raises if there is an error.