Buckets.Cloud behaviour (buckets v1.0.0-rc.1)

Defines a cloud.

A cloud manages the movement of files and data between your application and a remote bucket for persistent storage.

When used, it expects :otp_app as an option:

defmodule MyApp.Cloud do
  use Buckets.Cloud,
    otp_app: :my_app
end

Configuration is fetched from the application config, using a combination of :otp_app and the module that you defined:

config :my_app, MyApp.Cloud,
  adapter: Buckets.Adapters.Volume,
  bucket: "tmp/buckets_volume",
  base_url: "http://localhost:4000"

Each Cloud module corresponds to a single storage backend, similar to how Ecto.Repo modules correspond to a single database. For multi-cloud applications, define multiple Cloud modules:

defmodule MyApp.VolumeCloud do
  use Buckets.Cloud, otp_app: :my_app
end

defmodule MyApp.GCSCloud do
  use Buckets.Cloud, otp_app: :my_app
end

defmodule MyApp.S3Cloud do
  use Buckets.Cloud, otp_app: :my_app
end

You may also specify config dynamically at runtime, using the :config opt where it is supported.

Supervision

Cloud modules start a supervisor that manages any background processes required by the configured adapter. Some adapters (like GCS) need authentication servers, while others (like Volume, S3) don't need any supervised processes.

Only add Cloud modules to your supervision tree if they need background processes. If you add a Cloud module that doesn't need supervision (Volume, S3), you'll see a warning message suggesting you remove it to avoid unnecessary overhead.

# Only needed for adapters that require background processes (like GCS)
children = [
  MyApp.GCSCloud  # GCS needs auth servers
  # MyApp.VolumeCloud - Not needed, would show warning
  # MyApp.S3Cloud - Not needed, would show warning
]

Dynamic Configuration

For multi-tenant applications where cloud configurations are determined at runtime, every Cloud module supports dynamic configuration using the process dictionary, similar to Ecto's dynamic repositories.

Usage

There are two ways to use dynamic configuration:

1. Scoped Configuration (like Ecto transactions)

Use with_config/2 for temporary configuration:

# Define the runtime configuration
config = [
  adapter: Buckets.Adapters.S3,
  bucket: "user-bucket",
  access_key_id: "AKIA...",
  secret_access_key: "secret...",
  region: "us-east-1"
]

# Execute operations with the dynamic config
{:ok, object} = MyApp.Cloud.with_config(config, fn ->
  MyApp.Cloud.insert("file.pdf")
  MyApp.Cloud.insert("another.pdf")  # Same config
end)

2. Process-Local Configuration (like Ecto.Repo.put_dynamic_repo)

Use put_dynamic_config/1 for persistent configuration in the current process:

# Set dynamic config for this process
:ok = MyApp.Cloud.put_dynamic_config([
  adapter: Buckets.Adapters.GCS,
  bucket: "tenant-specific-bucket",
  service_account_credentials: tenant.credentials
])

# All subsequent operations use the dynamic config
{:ok, object1} = MyApp.Cloud.insert("file1.pdf")
{:ok, object2} = MyApp.Cloud.insert("file2.pdf")

Auth Server Management

Auth servers (for GCS) are automatically started and cached per-process as needed. You don't need any special configuration or supervision setup for dynamic clouds.

Summary

Callbacks

Deletes a Buckets.Object permanently.

Same as delete/1 but returns the object or raises if there is an error.

Inserts a Buckets.Object or a file from a path into a bucket.

Same as insert/2 but returns the object or raises if there is an error.

Returns a map to be used as configuration for a LiveView live upload. The configuration contains a signed URL that permits the upload to a remote bucket. Requires that an :uploader is configured for the location that the file is being uploaded to.

Same as c:live_upload/1 but returns the upload config or raises if there is an error.

Loads the data for a Buckets.Object, placing it in memory by default. It will load data lazily, doing nothing if data is already present.

Same as c:load/1 but returns the object or raises if there is an error.

An overridable function that processes filenames before encoding them in a remote path.

Reads the data for Buckets.Object, preferring local data first and fetching from the remote bucket if needed.

Same as read/1 but returns the data or raises if there is an error.

An overridable function that specifies the temporary directory in which objects are stored.

Unloads the data for a Buckets.Object. If the data is stored in a local file, the file will be deleted.

Returns a SignedURL struct for a Buckets.Object.

Same as url/1 but returns the SignedURL raises if there is an error.

Callbacks

@callback delete(object :: Buckets.Object.t()) ::
  {:ok, Buckets.Object.t()} | {:error, term()}

Deletes a Buckets.Object permanently.

Link to this callback

delete!(object)

@callback delete!(object :: Buckets.Object.t()) :: Buckets.Object.t()

Same as delete/1 but returns the object or raises if there is an error.

Link to this callback

insert(object_or_path, opts)

@callback insert(
  object_or_path :: Buckets.Object.t() | String.t(),
  opts :: Keyword.t()
) :: {:ok, Buckets.Object.t()} | {:error, term()}

Inserts a Buckets.Object or a file from a path into a bucket.

Link to this callback

insert!(object_or_path, opts)

@callback insert!(
  object_or_path :: Buckets.Object.t() | String.t(),
  opts :: Keyword.t()
) :: Buckets.Object.t()

Same as insert/2 but returns the object or raises if there is an error.

Link to this callback

live_upload(entry, opts)

@callback live_upload(
  entry :: Phoenix.LiveView.UploadEntry.t(),
  opts :: Keyword.t()
) :: {:ok, map()} | {:error, term()}

Returns a map to be used as configuration for a LiveView live upload. The configuration contains a signed URL that permits the upload to a remote bucket. Requires that an :uploader is configured for the location that the file is being uploaded to.

Link to this callback

live_upload!(entry, opts)

@callback live_upload!(
  entry :: Phoenix.LiveView.UploadEntry.t(),
  opts :: Keyword.t()
) :: map()

Same as c:live_upload/1 but returns the upload config or raises if there is an error.

Link to this callback

load(object, opts)

@callback load(
  object :: Buckets.Object.t(),
  opts :: Keyword.t()
) :: {:ok, Buckets.Object.t()} | {:error, term()}

Loads the data for a Buckets.Object, placing it in memory by default. It will load data lazily, doing nothing if data is already present.

Options

* `:to` - A location to store the loaded data to on disk, if this is preferred over
  loading to memory. May be a path, `:tmp` (to load to the configured tmp dir), or
  `{:tmp, path}` (to load to a path in the configured tmp dir).
* `:force` - If data is already present, first unload it with `c:unload/1` before loading
  new data. Warning: if data is stored in a file, it will be deleted.
Link to this callback

load!(object, opts)

@callback load!(object :: Buckets.Object.t(), opts :: Keyword.t()) :: Buckets.Object.t()

Same as c:load/1 but returns the object or raises if there is an error.

Link to this callback

normalize_filename(filename)

@callback normalize_filename(filename :: String.t()) :: String.t()

An overridable function that processes filenames before encoding them in a remote path.

By default, replaces whitespace with "_" and removes all non-alphanumeric characters.

@callback read(object :: Buckets.Object.t()) :: {:ok, binary()} | {:error, term()}

Reads the data for Buckets.Object, preferring local data first and fetching from the remote bucket if needed.

@callback read!(object :: Buckets.Object.t()) :: binary()

Same as read/1 but returns the data or raises if there is an error.

@callback tmp_dir() :: String.t()

An overridable function that specifies the temporary directory in which objects are stored.

Defaults to System.tmp_dir!().

@callback unload(object :: Buckets.Object.t()) :: Buckets.Object.t()

Unloads the data for a Buckets.Object. If the data is stored in a local file, the file will be deleted.

@callback url(object :: Buckets.Object.t()) :: Buckets.Object.t()

Returns a SignedURL struct for a Buckets.Object.

Link to this callback

url!(object, opts)

@callback url!(object :: Buckets.Object.t(), opts :: Keyword.t()) :: Buckets.Object.t()

Same as url/1 but returns the SignedURL raises if there is an error.