Fil.Adapter behaviour (Fil v0.2.0)

The contract every adapter implements.

An adapter is a stateless module. init/1 turns the options into a state term once, when Fil.disk/1 builds the disk, and every other callback gets that state. There's no process to start, supervise or shut down.

Plugins run in Fil before an adapter is called, so an adapter never needs to know about them.

Two rules keep adapters small:

Mutations return a bare :ok. Fil wraps results into {:ok, %Fil.Ref{}} and handles cross-disk copies, so adapters don't have to. Errors are Fil's error structs (see Errors), and Fil fills in the operation and the path.
Paths arrive normalized and jailed. Fil has already resolved . and .., collapsed repeated / and rejected escapes, and "." means the disk root. Adapters may check again (Fil.Adapter.Local does), but they don't need to parse paths defensively.

Contract

Fil uses the vocabulary of Elixir's File module, but every adapter behaves like an object store, so the semantics differ where the two disagree. Beyond the callback signatures, this is the contract:

Situation	`File`	Every `Fil` adapter
`stream/3` on a missing file	`File.stream!/3` raises when it's read	`{:error, %Fil.NotFoundError{}}`
`write/4` into a missing directory	`{:error, :enoent}`	creates the missing parents
exclusive `write/4` on an existing file	`{:error, :eexist}`	`{:error, %Fil.AlreadyExistsError{}}`
exclusive `cp/4` or `rename/4` onto an existing file	replaces it	`{:error, %Fil.AlreadyExistsError{}}`
`cp/4` or `rename/4` into a missing directory	`{:error, :enoent}`	creates the missing parents
`rm/3` on a missing file	`{:error, :enoent}`	`:ok`
`ls/3` on a missing directory	`{:error, :enoent}`	`{:ok, []}`
`ls/3` results	names, one level deep	`{path, stat}` pairs, directories included, also in recursive listings
`rm_rf/3` results	`{:ok, removed_paths}`, directories included	`{:ok, count}` of removed files
`rm_rf/3` on `"reports"`	the directory `reports`	`reports` and all of `reports/`, not `reports.txt`
paths	relative to the working directory, or absolute	relative to the disk root, never above it

An exclusive write is File.write/3 with [:exclusive], and write/4 with if_exists: :error. An exclusive copy or move is cp/4 or rename/4 with if_exists: :error. It fails with the destination's path and leaves both files as they were.

A copy or a move onto its own path never loses the file. The adapter leaves it where it is, fails with Fil.AlreadyExistsError when the call is exclusive, or refuses the call with Fil.InvalidRequestError. A copy keeps the content type, where the storage keeps one.

The :etag that stat/3 returns changes when a write changes the size of the file. A weak etag may miss other changes. Which result an adapter gives for a copy onto itself, and what its etag covers, is under Operations on the adapter's page.

Where an adapter returns :ok, the Fil function returns {:ok, %Fil.Ref{}}, and Fil.ls/2 turns the pairs into refs.

Options that File has no equivalent for:

size: on write/4 is the size of a stream, when the caller knows it or Fil.write/4 found it. Storage that needs the size before the content uses it, and uploads a stream without one in parts
checksum: on write/4 sends a checksum of the content where the storage keeps one, and storage that finds the content doesn't match fails with Fil.ChecksumMismatchError. Storage without checksums ignores the option
checksum: on stat/3 fills in Fil.Stat's :checksum, from the storage or computed from the content. A directory has no checksum, and a copy keeps the checksum of a file written in one part
verify_checksum: true on read/3 fails with Fil.ChecksumMismatchError when the content doesn't match a stored checksum
offset: and length: on read/3 and stream/3 read only part of the file: up to length: bytes from byte offset: on, or to the end without length:. A part that reaches past the end is shorter, and one that starts at or after the end is empty, without an error. stream/3 returns the size of the part. Fil never passes them together with verify_checksum: true

Fil.AdapterCase (internal for now) tests this contract against a live disk.

Every adapter Fil ships reads, writes and streams files, lists, copies and checks checksums, and every disk builds public and signed URLs (S3 itself, the others with Fil.Plugin.URL), so code written against one disk runs on the others. What's left are edge cases, such as whether directories exist on their own, and each adapter lists them under Operations on its own page.

Errors

Every error is an exception struct, listed under Errors in the sidebar. Which struct you get tells you what to do next, and it's the same on every adapter: a missing file is a Fil.NotFoundError on the local disk and on S3. Fil.UnavailableError and Fil.ConflictError are the ones where retrying can help.

Every error has the same fields:

:op: the operation, such as :read (see Fil.Op.name/0)
:path: the path as the caller passed it (before plugins rewrote it), relative to the disk root. For a copy or a move, the side that failed
:disk: the Fil.Disk. Fil.ref(error.disk, error.path) is a ref to the file, and Fil.Disk.adapter(error.disk) returns the adapter
:reason: the exact cause, as the storage reported it, in the form its adapter documents (:enoent from Fil.Adapter.Local, "NoSuchKey" from Fil.Adapter.S3). nil only when a plugin built the error

Fil returns two of them itself: Fil.InvalidRequestError with reason: :ebadpath for a path that escapes the disk root, and Fil.UnsupportedError with reason: :no_callback for a URL on an adapter without that callback.

An adapter returns the struct with :reason set and leaves :op, :path and :disk nil: Fil fills in those that are still nil. An adapter that fails on the destination of a cp/4 or rename/4 sets :path to the destination, because Fil fills in the source. Anything in {:error, _} that isn't an exception raises ArgumentError.

A failure that fits a struct is returned as that struct, whatever the storage reported: Fil.Adapter.Local turns the :eperm that deleting a directory gives into :eisdir, and Fil.Adapter.S3 checks a copy that the server refused with a plain 400 for a missing source. Directory errors only come from storage with directories of its own, such as the filesystem. Where directories are only key prefixes, reading or copying one is a Fil.NotFoundError.

Summary

Types

error()

opts()

path()

state()

Callbacks

address(state)

Returns the address of the files: what decides which file a path names, such as the root directory, or the bucket and prefix.

cp(state, path, path, opts)

Copies a file within the same disk.

init(opts)

Validates options and builds the state passed to every other callback.

ls(state, path, opts)

Lists prefix, one level deep unless recursive: true.

read(state, path, opts)

Reads the whole file.

rename(state, path, path, opts)

Moves a file within the same disk.

rm(state, path, opts)

Deletes a file. Idempotent: a missing file is still :ok.

rm_rf(state, path, opts)

Removes everything under prefix and returns the number of deleted files.

signed_url(state, path, opts)

Builds a URL that grants temporary access to a file.

stat(state, path, opts)

Returns metadata for a file or directory.

stream(state, path, opts)

Checks that a file can be read and returns a stream of its content.

url(state, path, opts)

Builds the public URL of a file, without a signature.

write(state, path, arg3, opts)

Writes content, creating parent directories as needed.

Types

error()

@type error() :: {:error, Fil.error()}

opts()

@type opts() :: keyword()

path()

@type path() :: String.t()

state()

@type state() :: term()

Callbacks

address(state)

(optional)

@callback address(state()) :: term()

Returns the address of the files: what decides which file a path names, such as the root directory, or the bucket and prefix.

Optional. Fil.Disk.same_storage?/2 compares two disks of the adapter by it. Leave out credentials and options that only change how the storage is reached, so a disk with rotated credentials is still the same storage. Without this callback, the whole state is compared.

cp(state, path, path, opts)

@callback cp(state(), path(), path(), opts()) :: :ok | error()

Copies a file within the same disk.

A copy across two disks doesn't call this: Fil reads the file with read/3 on the source adapter and writes it with write/4 on the destination adapter.

With if_exists: :error, an existing destination fails the copy with Fil.AlreadyExistsError, :path set to the destination, and the destination keeps its content.

init(opts)

@callback init(opts()) :: {:ok, state()} | {:error, term()}

Validates options and builds the state passed to every other callback.

Not an operation, so an error is anything that explains the bad option, such as a NimbleOptions.ValidationError. Fil.Disk.new/1 raises it as an ArgumentError.

ls(state, path, opts)

@callback ls(state(), path(), opts()) :: {:ok, [{path(), Fil.Stat.t()}]} | error()

Lists prefix, one level deep unless recursive: true.

Returns {path, stat} pairs. Paths are relative to the disk root, and each stat holds what the listing returned. Fil turns the pairs into Fil.Refs; adapters don't build those themselves.

A recursive listing includes the directories of the subtree, at least every directory that holds a file, so storage without directories derives them from its keys. Fil filters by type: and leaves out the directories of a recursive listing unless the caller asked for them, so adapters can ignore that option. The pattern of Fil.wildcard/3 doesn't reach the adapter either: Fil matches it with one listing per directory level, and a recursive one from a ** on.

read(state, path, opts)

@callback read(state(), path(), opts()) :: {:ok, binary()} | error()

Reads the whole file.

rename(state, path, path, opts)

@callback rename(state(), path(), path(), opts()) :: :ok | error()

Moves a file within the same disk.

A move across two disks doesn't call this: Fil copies the file as described in cp/4, then removes the source with rm/3.

With if_exists: :error, an existing destination fails the move as it fails cp/4, and the source stays where it is.

rm(state, path, opts)

@callback rm(state(), path(), opts()) :: :ok | error()

Deletes a file. Idempotent: a missing file is still :ok.

rm_rf(state, path, opts)

@callback rm_rf(state(), path(), opts()) :: {:ok, non_neg_integer()} | error()

Removes everything under prefix and returns the number of deleted files.

signed_url(state, path, opts)

(optional)

@callback signed_url(state(), path(), opts()) :: {:ok, String.t()} | error()

Builds a URL that grants temporary access to a file.

Optional. Adapters whose storage can't sign URLs leave it out, and Fil.signed_url/2 then returns a Fil.UnsupportedError. Fil.Plugin.URL signs URLs for those disks, and Fil.Plug serves them.

opts has :method, :expires_in and :query as Fil.signed_url/3 validated them, and for a download with disposition:, :disposition as the finished content-disposition header value. The storage has to send that header with the download, and the URL's signature has to cover it and the :query parameters. A request with another path, method or parameter than the URL was signed for gets a status in the 4xx range, and an upload through it writes nothing.

An upload URL may get :content_type, :size and :if_exists. The signature has to cover them, so the storage refuses an upload with another content-type or content-length, and with if_exists: :error one that finds a file at the path. The client sends the headers Fil.signed_url/3 lists under Uploads.

stat(state, path, opts)

@callback stat(state(), path(), opts()) :: {:ok, Fil.Stat.t()} | error()

Returns metadata for a file or directory.

With type: :regular or type: :directory, the adapter may skip looking for the other type and return Fil.NotFoundError instead, so a check for a file costs no directory probe. Fil checks the type of the result either way, so adapters can ignore the option.

stream(state, path, opts)

(optional)

@callback stream(state(), path(), opts()) ::
  {:ok, Enumerable.t()} | {:ok, Enumerable.t(), non_neg_integer()} | error()

Checks that a file can be read and returns a stream of its content.

Optional. Without it, Fil.stream/3 calls read/3 and streams the file as one chunk.

Returns {:ok, stream, size} when the check found the size of the file, {:ok, stream} otherwise. Fil.write/4 passes the size on when the stream is written, to another disk or in a copy across disks, so storage that needs the size before the content (S3) can stream it too.

The stream is lazy: it reads the file only when it's enumerated, and again each time it is, in whatever process enumerates it. It yields binaries of any size and raises an error struct when reading fails (Fil fills in its context). opts are those of read/3, and a verify_checksum: true mismatch raises Fil.ChecksumMismatchError at the latest after the last chunk.

url(state, path, opts)

(optional)

@callback url(state(), path(), opts()) :: {:ok, String.t()} | error()

Builds the public URL of a file, without a signature.

Optional. Adapters whose storage has no URLs leave it out, and Fil.url/2 then returns a Fil.UnsupportedError. Fil.Plugin.URL builds URLs for those disks, and Fil.Plug serves them.

write(state, path, arg3, opts)

@callback write(state(), path(), iodata() | Enumerable.t(), opts()) :: :ok | error()

Writes content, creating parent directories as needed.

content is iodata, or a stream of non-empty binaries. Fil checks a stream against the :size option while the adapter reads it, when the caller gave one or Fil.write/4 found one. If the stream raises, at any point and also after its last chunk, the adapter lets the error propagate and writes nothing: the destination keeps what it had before. That's how a plugin refuses content while it's written (Fil.Op.scan_content/4).