Waffle.Definition.Storage (waffle v2.0.0)

Copy Markdown View Source

Uploader configuration.

Add use Waffle.Definition inside your module to use it as uploader.

Storage directory

By default, the storage directory is uploads. But, it can be customized in two ways.

By setting up configuration

Customize storage directory via configuration option :storage_dir.

config :waffle,
  storage_dir: "my/dir"

By overriding the relevant functions in definition modules

Every definition module has a default storage_dir/2 which is overridable.

For example, a common pattern for user avatars is to store each user's uploaded images in a separate subdirectory based on primary key:

def storage_dir(version, {file, scope}) do
  "uploads/users/avatars/#{scope.id}"
end

Note: If you are "attaching" a file to a record on creation (eg, while inserting the record at the same time), then you cannot use the model's id as a path component. You must either (1) use a different storage path format, such as UUIDs, or (2) attach and update the model after an id has been given. Read more about how to integrate it with Ecto

Note: The storage directory is used for both local filestorage (as the relative or absolute directory), and S3 storage, as the path name (not including the bucket).

Asynchronous File Uploading

If you specify multiple versions in your definition module, each version is processed and stored concurrently as independent Tasks. To prevent an overconsumption of system resources, each Task is given a specified timeout to wait, after which the process will fail. By default, the timeout is 15_000 milliseconds.

If you wish to change the time allocated to version transformation and storage, you can add a configuration option:

config :waffle,
  :version_timeout, 15_000 # milliseconds

To disable asynchronous processing, add @async false to your definition module.

Storage of files

Waffle currently supports:

Override the __storage function in your definition module if you want to use a different type of storage for a particular uploader.

Community adapters

File Validation

While storing files on S3 eliminates some malicious attack vectors, it is strongly encouraged to validate the extensions of uploaded files as well.

Waffle delegates validation to a validate/1 function with a tuple of the file and scope. Validation will be considered successful if the function returns true or :ok. A customized error message can be returned in the form of {:error, message}. Any other return value will return {:error, :invalid_file}.

Validating with external library (e.g. magic_bytes):

Extension-based validation can be trivially bypassed by renaming a file. Because ImageMagick (and tools built on it) historically delegate format detection to GhostScript via file extension. Validating by magic bytes, reading the actual file header - eliminates that vector.

The magic_bytes package provides exactly this. %Waffle.File{} exposes path (local, remote, and binary uploads) and stream (set for streaming uploads), which map directly to MagicBytes.from_path/1 and MagicBytes.from_stream/1:

defmodule Avatar do
  use Waffle.Definition

  @allowed_types ~w(image/jpeg image/png image/gif image/webp)

  def validate({%{path: path}, _}) when not is_nil(path) do
    case MagicBytes.from_path(path) do
      {:ok, mime} when mime in @allowed_types -> :ok
      {:ok, _mime} -> {:error, "invalid file type"}
      {:error, :unknown} -> {:error, "invalid file type"}
      {:error, _} -> {:error, "could not read file"}
    end
  end

  def validate({%{stream: stream}, _}) when not is_nil(stream) do
    case MagicBytes.from_stream(stream) do
      {:ok, mime} when mime in @allowed_types -> :ok
      {:ok, _mime} -> {:error, "invalid file type"}
      {:error, :unknown} -> {:error, "invalid file type"}
      {:error, _} -> {:error, "could not read file"}
    end
  end
end

Validating by file extension

Extension-based validation is simpler but less secure (see above). It may be appropriate when you control the upload source or as an additional layer:

defmodule Avatar do
  use Waffle.Definition
  @extension_whitelist ~w(.jpg .jpeg .gif .png)

  def validate({file, _}) do
    file_extension = file.file_name |> Path.extname() |> String.downcase()

    case Enum.member?(@extension_whitelist, file_extension) do
      true -> :ok
      false -> {:error, "invalid file type"}
    end
  end
end

Fetching remote files

When a remote URL is passed to an uploader, Waffle downloads it to a temporary file before validation, transformation, and storage.

See Waffle.HTTPClient.Req for HTTP client setup and Req-specific options. See Waffle.HTTPClient.Request for timeouts, redirects, retries, and response body limits.

Remote-file HTTP client configuration does not affect requests made by Waffle.Storage.S3, which are handled separately by ExAws.

Filenames from Content-Disposition

When a remote response provides a filename through Content-Disposition, Waffle uses it instead of the URL's basename. Both filename and RFC 6266 filename* parameters are supported, with filename* taking precedence.

Filenames containing path separators or control characters are rejected. In that case, Waffle falls back to the filename derived from the URL.

Passing custom request headers

Waffle does not add custom headers when downloading remote files. Override remote_file_headers/1 in your definition module to provide them. For example:

defmodule Avatar do
  use Waffle.Definition

  def remote_file_headers(%URI{host: "elixir-lang.org"}) do
    credentials = Application.get_env(:my_app, :avatar_credentials)
    token = Base.encode64(credentials[:username] <> ":" <> credentials[:password])

    [{"Authorization", "Basic #{token}"}]
  end
end

This authenticates requests only to the specified domain. Requests to other domains use no custom headers.

Temporary Directory

When processing files, Waffle creates temporary files for operations like downloading remote files or applying transformations. By default, these files are stored in the system's temporary directory (as returned by System.tmp_dir/0).

You can customize this location by setting the :tmp_dir configuration option:

config :waffle,
  tmp_dir: "/path/to/custom/tmp"

Waffle may fail to remove temporary files if the process using them crashes.