Fil is a pluggable file storage abstraction for Elixir.
Mix.install([
{:fil, "~> 0.2"}
])
disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
disk
|> Fil.write!("hello.txt", "World")
|> Fil.read!()
#=> "World"With adapter: Fil.Adapter.S3, a bucket and credentials, the same code writes to S3.
To try Fil without a project, run the tour in Livebook.
Features and goals
- One API for many kinds of storage. Local disk, S3 (and S3-compatible stores) and an in-memory disk for async
tests, with the same behaviour on every adapter.
cpandrenamework across disks. - Just values. A disk is a plain value: no application config, no registry, nothing to supervise. It works in a
script or a Livebook with
Mix.install/1. - Pluggable. Anything that isn't about where files are stored is a plugin, such as setting content types or logging.
- Few dependencies. Req, NimbleOptions, MIME and Telemetry, plus Plug if you serve files. Cloud adapters use Req instead of their own SDKs.
- Streaming. Files can be read and written as streams on every disk, so large files don't have to fit in memory.
- Integrations. Each one is compiled only when its dependency is in your project:
- Plug:
Fil.Plugserves public and signed URLs from your endpoint or router, and sends files from controllers. - Phoenix LiveView:
Fil.LiveViewstores uploads on any disk, also straight from the browser, and has an upload field for forms (see the Phoenix guide). - Ecto:
Fil.Ecto.Refstores refs in Ecto schemas. - Livebook:
Fil.Kinobrowses a disk, uploads files to it, and adds a smart cell that builds one.
- Plug:
- URLs on every disk. Public and signed GET and PUT URLs, from S3 itself or from
Fil.Plugin.URLandFil.Plug. - Safe by default. Paths can't climb out of the disk root,
if_exists: :errornever replaces a file, and S3 verifies checksums. - Errors you can act on. Each error, such as
Fil.NotFoundErrororFil.UnavailableError, says what to do next and is the same on every adapter. - Telemetry. Every operation emits Telemetry events with its duration, bytes and
error, for metrics, logs and traces (see
Fil.Telemetry).
Concepts
Fil uses the function names of Elixir's File module (read, write,
stream, stat, ls, cp, rename, rm, rm_rf), but on every adapter they behave like an object store:
writecreates missing parent directoriesrmon a missing file succeedslson a missing directory returns an empty list- paths are always relative to the disk root, and
.is the root - a path that climbs above the root fails with a
Fil.InvalidRequestError
The contract in Fil.Adapter lists every difference from
File. Every function that can fail returns {:ok, result} or {:error, error}, with an error from the Errors
section below.
Disks
A disk says where files are stored: an adapter and its options. It's a plain value, so there's no application config
and nothing to add to your supervision tree (Fil starts one process of its own, for the Memory adapter). Every
function takes a disk as its first argument and a path relative to the disk's root:
disk = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")
{:ok, _} = Fil.write(disk, "hello.txt", "World")
Fil.read(disk, "hello.txt")
#=> {:ok, "World"}Refs
A Fil.Ref is a single value for a file on a disk, built with Fil.ref/2. Every
function that takes a disk and a path as two arguments also takes a ref as one argument in their place:
hello = Fil.ref(disk, "hello.txt")
#=> #Fil.Ref<local:hello.txt>
{:ok, _} = Fil.write(hello, "World")
Fil.read(hello)
#=> {:ok, "World"}Actions on files return the ref they acted on, which keeps cross-disk code short:
with {:ok, report} <- Fil.write(s3, "reports/q3.pdf", pdf),
{:ok, _backup} <- Fil.cp(report, Fil.ref(local, "backups/q3.pdf")) do
{:ok, report}
endThe bang variants return the ref itself instead of {:ok, ref}, so you can pipe one call into the next:
disk
|> Fil.write!("hello.txt", "World")
|> Fil.cp!("backup/hello.txt")
|> Fil.read!()
#=> "World"Plugins
Plugins attach to a disk and see every operation on it. Fil ships one that sets the content type from the file
extension, so S3 serves reports/q3.pdf as application/pdf:
s3 =
Fil.disk(adapter: Fil.Adapter.S3, bucket: "my-bucket", region: "eu-central-1")
|> Fil.Plugin.ContentType.attach()
{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)For a disk built from config, Fil.disk/1 takes plugins as data: plugins: [{Fil.Plugin.ContentType, :call, []}].
Your own plugin is a function. It gets the operation, calls next to run the rest, and returns the result:
local =
Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")
|> Fil.attach(:log, fn op, next, _opts ->
IO.puts("#{op.name} #{op.path}")
next.(op)
end)The plugins guide explains how to write plugins: matching on operations, changing content, handling errors and answering without the adapter.
Errors
Every error is an exception struct that says what you can do about it, and means the same on every disk:
case Fil.read(disk, "report.txt") do
{:ok, content} -> content
{:error, %Fil.NotFoundError{}} -> nil
{:error, %Fil.UnavailableError{}} -> :retry_later
endThe structs contain the operation, the path, the disk and what the storage reported (:reason), so a log line says
which file failed and why. Each error has its own page under Errors in the docs, and
Errors in Fil.Adapter explains how adapters use them.
Every function that can fail has a bang variant that returns the bare result and raises the same struct instead.
Usage
The installation guide is the full setup for an application: a module for your disks, the config for each environment (local disks in development, memory disks in tests and S3 in production), signed URLs and tests. This section is only a quick tour of the API.
Build one disk per kind of storage:
local = Fil.disk(adapter: Fil.Adapter.Local, root: "priv/storage")
s3 =
Fil.disk(
adapter: Fil.Adapter.S3,
bucket: "my-bucket",
region: "eu-central-1",
access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
)Every operation works the same on every disk:
{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf)
{:ok, pdf} = Fil.read(report)
{:ok, stat} = Fil.stat(report)
{:ok, reports} = Fil.ls(s3, "reports/", recursive: true)
{:ok, backup} = Fil.cp(report, Fil.ref(local, "backups/q3.pdf"))
{:ok, _} = Fil.rm(report)Large files don't have to fit in memory. stream returns a file's content as a stream of chunks, and write takes a
stream as well as a binary, so a file can go from one disk to another piece by piece. {:file, path} streams a local
file:
{:ok, video} = Fil.stream(s3, "videos/intro.mp4")
{:ok, _} = Fil.write(local, "cache/intro.mp4", video)
{:ok, _} = Fil.write(s3, "backups/db.dump", {:file, "db.dump"})With a local file, a File.stream! of bytes or a stream from stream, Fil knows the size, and S3 sends the stream as
it's read, in one request. For other streams, pass size: if you know it, or S3 uploads the stream in parts of 8 MiB
(the local and memory disks don't need the size).
offset: and length: read part of a file, and the storage reads only that part (S3 with a Range header):
{:ok, header} = Fil.read(s3, "videos/intro.mp4", length: 4096)
{:ok, rest} = Fil.stream(s3, "videos/intro.mp4", offset: 4096)url returns the public URL of a file, and signed_url an expiring one, so clients can download or upload a file
directly instead of going through your application code. S3 serves its URLs itself. For local and in-memory disks,
Fil.Plugin.URL builds them and Fil.Plug serves them from your application, with etags, 304s and range requests,
so browsers can cache files and seek in videos:
{:ok, logo_url} = Fil.url(s3, "logo.png")
{:ok, url} = Fil.signed_url(report, expires_in: 900)
{:ok, upload_url} = Fil.signed_url(s3, "inbox/new.bin", method: :put)With checksum:, a write sends a checksum of the content. S3 rejects the upload if what it received doesn't match,
and stores the checksum with the object, so later reads can check it:
{:ok, report} = Fil.write(s3, "reports/q3.pdf", pdf, checksum: :sha256)
{:ok, pdf} = Fil.read(report, verify_checksum: true)
{:ok, %Fil.Stat{checksum: {:sha256, checksum}}} = Fil.stat(report, checksum: :sha256)To create a file only if it doesn't exist yet, pass if_exists: :error. If the file is already there, nothing is
overwritten and the write returns a Fil.AlreadyExistsError. That
makes a simple lock: whoever creates the file first runs the job.
case Fil.write(s3, "jobs/today.lock", "started", if_exists: :error) do
{:ok, _lock} -> :run_the_job
{:error, %Fil.AlreadyExistsError{}} -> :someone_else_won
endtmp returns a ref to a file in a temporary directory, which is removed when the calling process exits. It's a local
file, so a copy from any disk gives a tool the path it needs:
report = Fil.tmp("report.pdf")
{:ok, _} = Fil.cp(Fil.ref(s3, "reports/q3.pdf"), report)
{text, 0} = System.cmd("pdftotext", [Fil.Tmp.path(report), "-"])Development
mix test runs the unit tests, which need no network. The integration tests run the conformance suite against RustFS,
started with Docker Compose:
docker compose up -d
mix test.integration
FIL_S3_ENDPOINT, FIL_S3_ACCESS_KEY_ID, FIL_S3_SECRET_ACCESS_KEY and FIL_S3_REGION point the integration tests
at another S3 endpoint.
Acknowledgments
Fil builds on ideas from these projects:
- Flysystem (PHP): the scope, one API across many kinds of storage
- Req: the API design, with disks as plain values and plugins that attach to them
- Plug: plugins as small units you add to a disk, each doing one thing to every operation that passes through
Thanks to their authors and contributors.
Summary
Building
Attaches a plugin callback to a disk under a name.
Removes a plugin from a disk. Removing a name that isn't attached returns the disk unchanged.
Builds a disk. Shorthand for Fil.Disk.new/1, which lists the options.
Builds a ref. Shorthand for Fil.Ref.new/2.
Creates a temporary directory and returns a ref to it. Shorthand for Fil.Tmp.new/0, which lists what it raises.
Creates a temporary directory and returns a ref to the file name in it. Shorthand for Fil.Tmp.new/1, which lists
the names it takes.
Operations
Copies a file and returns the destination ref.
Copies a file. See cp/2.
Copies a file. See cp/2.
Whether this path is a directory, which on object stores means a non-empty prefix. Same as
exists?(ref, type: :directory), see exists?/1.
Whether this path is a directory. See dir?/1.
Whether anything exists at this path, or with type: whether a file or a directory does.
Whether anything exists at this path. See exists?/1.
Whether anything exists at this path. See exists?/1.
Lists a directory. The whole listing is loaded into memory.
Lists a directory. See ls/1.
Lists a directory. See ls/1.
Reads a file.
Reads a file. See read/1.
Reads a file. See read/1.
Whether this path is a file. Same as exists?(ref, type: :regular), see exists?/1.
Whether this path is a file. See regular?/1.
Moves a file and returns the destination ref.
Moves a file. See rename/2.
Moves a file. See rename/2.
Deletes a file.
Deletes a file. See rm/1.
Deletes a file. See rm/1.
Removes everything under a prefix and returns the number of deleted files.
Removes everything under a prefix. See rm_rf/1.
Removes everything under a prefix. See rm_rf/1.
Builds a URL that grants temporary access to a file.
Builds a signed URL. See signed_url/1.
Builds a signed URL. See signed_url/1.
Returns metadata for a file or directory.
Returns metadata for a file or directory. See stat/1.
Returns metadata for a file or directory. See stat/1.
Streams a file: checks that it can be read and returns its content as a stream of binaries.
Streams a file. See stream/1.
Streams a file. See stream/1.
Builds the public URL of a file.
Builds a public URL. See url/1.
Builds a public URL. See url/1.
Lists the paths that match a pattern, like Path.wildcard/2 with the disk root as the working directory.
Lists the paths that match a pattern. See wildcard/2.
Writes a file, creating missing parent directories.
Writes a file. See write/2.
Writes a file. See write/2.
Bang variants
Same as cp/2, raising the error on failure.
Same as cp/3, raising the error on failure.
Same as cp/4, raising the error on failure.
Same as ls/1, raising the error on failure.
Same as ls/3, raising the error on failure.
Same as read/1, raising the error on failure.
Same as read/2, raising the error on failure.
Same as read/3, raising the error on failure.
Same as rename/2, raising the error on failure.
Same as rename/3, raising the error on failure.
Same as rename/4, raising the error on failure.
Same as rm/3, raising the error on failure.
Same as rm_rf/1, raising the error on failure.
Same as rm_rf/2, raising the error on failure.
Same as rm_rf/3, raising the error on failure.
Same as signed_url/1, raising the error on failure.
Same as signed_url/2, raising the error on failure.
Same as signed_url/3, raising the error on failure.
Same as stat/1, raising the error on failure.
Same as stat/2, raising the error on failure.
Same as stat/3, raising the error on failure.
Same as stream/1, raising the error on failure.
Same as stream/2, raising the error on failure.
Same as stream/3, raising the error on failure.
Same as url/2, raising the error on failure.
Same as url/3, raising the error on failure.
Same as wildcard/2, raising the error on failure.
Same as wildcard/3, raising the error on failure.
Same as write/2, raising the error on failure.
Same as write/3, raising the error on failure.
Same as write/4, raising the error on failure.
Types
What write/4 takes: iodata, a stream of it (any Enumerable that isn't a list, such as a Stream or a
File.Stream), or {:file, path} for a file on the local filesystem. A list is always iodata.
An error from an adapter or from Fil itself. Each struct stands for what the caller can do about it, and means the
same on every adapter; see Errors in Fil.Adapter.
A plugin callback: a function or a {module, function} pair. See the Plugins guide.
A result. Plugins may return their own exceptions, so an error isn't always one of error/0.
Building
@spec attach(Fil.Disk.t(), atom(), plugin_callback(), keyword()) :: Fil.Disk.t()
Attaches a plugin callback to a disk under a name.
iex> disk =
...> Fil.disk(adapter: Fil.Adapter.Memory)
...> |> Fil.attach(:shout, fn op, next, _opts ->
...> op |> Fil.Op.update_content(iodata: &String.upcase/1) |> next.()
...> end)
iex> Fil.write!(disk, "hello.txt", "world")
iex> Fil.read(disk, "hello.txt")
{:ok, "WORLD"}The callback is a function of arity 3 or a {module, function} pair naming a public function of arity 3. opts are
passed to it on every call. Attaching a name that's already attached replaces the callback and its options in the
same position. The Plugins guide explains how to write one.
@spec detach(Fil.Disk.t(), atom()) :: Fil.Disk.t()
Removes a plugin from a disk. Removing a name that isn't attached returns the disk unchanged.
@spec disk(keyword()) :: Fil.Disk.t()
Builds a disk. Shorthand for Fil.Disk.new/1, which lists the options.
iex> Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
#Fil.Disk<local>
@spec ref(Fil.Disk.t(), Path.t()) :: Fil.Ref.t()
Builds a ref. Shorthand for Fil.Ref.new/2.
iex> disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
iex> Fil.ref(disk, "uploads/a.txt")
#Fil.Ref<local:uploads/a.txt>
@spec tmp() :: Fil.Ref.t()
Creates a temporary directory and returns a ref to it. Shorthand for Fil.Tmp.new/0, which lists what it raises.
iex> Fil.tmp()
#Fil.Ref<local:.>The directory is removed when the calling process exits, see Fil.Tmp.
Creates a temporary directory and returns a ref to the file name in it. Shorthand for Fil.Tmp.new/1, which lists
the names it takes.
iex> Fil.tmp("report.pdf")
#Fil.Ref<local:report.pdf>The directory is removed when the calling process exits, see Fil.Tmp.
Operations
Copies a file and returns the destination ref.
Within one disk, Fil uses the adapter's native copy. Across disks, it streams the file from the source to the
destination (see stream/3), with the size the source's adapter found, so S3 sends the upload in one request too.
When a plugin on the source changes the content, or the adapter doesn't know the size, S3 uploads the file in parts.
A source that changes size while it's copied fails the copy with a Fil.ConflictError. The destination may be a
ref, or a bare path on the source's disk.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> other = Fil.disk(adapter: Fil.Adapter.Memory, root: "other")
iex> Fil.write!(disk, "hello.txt", "World")
iex> {:ok, backup} = Fil.cp(disk, "hello.txt", "backup/hello.txt")
iex> backup
#Fil.Ref<memory:backup/hello.txt>
iex> {:ok, copy} = Fil.cp(disk, "hello.txt", Fil.ref(other, "hello.txt"))
iex> Fil.read(copy)
{:ok, "World"}
@spec cp(Fil.Disk.t(), Path.t(), Fil.Ref.t() | Path.t()) :: result(Fil.Ref.t())
@spec cp(Fil.Ref.t(), Fil.Ref.t() | Path.t(), keyword()) :: result(Fil.Ref.t())
Copies a file. See cp/2.
Copies a file. See cp/2.
Whether this path is a directory, which on object stores means a non-empty prefix. Same as
exists?(ref, type: :directory), see exists?/1.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "reports/q3.pdf", "%PDF")
iex> Fil.dir?(disk, "reports")
true
iex> Fil.dir?(disk, "reports/q3.pdf")
false
@spec dir?(Fil.Disk.t(), Path.t()) :: boolean()
Whether this path is a directory. See dir?/1.
Whether anything exists at this path, or with type: whether a file or a directory does.
Predicates return a plain boolean, so unreachable storage or an invalid path is false. Use stat/1 when you need
to tell those cases apart.
regular?/1 is exists?(ref, type: :regular) and dir?/1 is exists?(ref, type: :directory). Pass the type when
you know what you're looking for: S3 then needs one request where it otherwise needs two for a missing path (a HEAD
for the file, then a listing for the directory).
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "reports/q3.pdf", "%PDF")
iex> Fil.exists?(disk, "reports/q3.pdf")
true
iex> Fil.exists?(disk, "reports", type: :regular)
false
iex> Fil.exists?(disk, "nope.txt")
falseOptions
:type- Counts only a file (:regular) or only a directory (:directory), the values ofFil.Stat's:type. Anything else is aFil.NotFoundError. Adapters can skip looking for the other type, which saves S3 a request.
@spec exists?(Fil.Disk.t(), Path.t()) :: boolean()
@spec exists?(Fil.Ref.t(), keyword()) :: boolean()
Whether anything exists at this path. See exists?/1.
@spec exists?(Fil.Disk.t(), Path.t(), keyword()) :: boolean()
Whether anything exists at this path. See exists?/1.
Lists a directory. The whole listing is loaded into memory.
One level deep by default; pass recursive: true to walk the whole subtree. Paths are relative to the disk root, and
a missing prefix returns an empty list.
A one-level listing includes directories. A recursive listing returns files only, unless you ask for its directories
with type: :directory: at least every directory that holds a file, because object stores have directories only
implicitly. Whether empty directories show up too is on the adapter's page.
To list by a pattern, at any depth, use wildcard/2.
Options
:recursive(boolean/0) - Walk the whole subtree instead of one level. The default value isfalse.:type- Lists only files (:regular) or only directories (:directory), the values ofFil.Stat's:type. Without it, a one-level listing returns both and a recursive one returns files.
Examples
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "hello.txt", "World")
iex> Fil.write!(disk, "reports/q3.pdf", "%PDF")
iex> {:ok, [hello, reports]} = Fil.ls(disk)
iex> hello
#Fil.Ref<memory:hello.txt>
iex> reports.stat.type
:directory
iex> {:ok, [_hello, report]} = Fil.ls(disk, ".", recursive: true)
iex> report
#Fil.Ref<memory:reports/q3.pdf>
iex> {:ok, [reports]} = Fil.ls(disk, ".", type: :directory)
iex> reports
#Fil.Ref<memory:reports>The results are ordinary refs with :stat filled in from the listing, so you can pass them to any other
function:
{:ok, reports} = Fil.ls(disk, "reports", recursive: true)
reports
|> Enum.filter(&(&1.stat.size == 0))
|> Enum.each(&Fil.rm/1)
@spec ls(Fil.Disk.t(), Path.t()) :: result([Fil.Ref.t()])
@spec ls(Fil.Ref.t(), keyword()) :: result([Fil.Ref.t()])
Lists a directory. See ls/1.
@spec ls(Fil.Disk.t(), Path.t(), keyword()) :: result([Fil.Ref.t()])
Lists a directory. See ls/1.
Reads a file.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "hello.txt", "World")
iex> Fil.read(disk, "hello.txt")
{:ok, "World"}
iex> Fil.read(disk, "hello.txt", offset: 1, length: 3)
{:ok, "orl"}
iex> {:error, %Fil.NotFoundError{path: "nope.txt", reason: :enoent}} = Fil.read(disk, "nope.txt")Options
:verify_checksum(boolean/0) - Checks the content against the checksum stored with it and returnsFil.ChecksumMismatchErrorif they differ. Content without a stored checksum (see the:checksumoption ofwrite/4), or on storage that stores none, is returned unchecked. A stored checksum covers the whole file, so it can't be combined with:offsetor:length. The default value isfalse.:offset(non_neg_integer/0) - Reads from this byte on, counted from 0. The storage reads only the part that's asked for (S3 with aRangeheader). From the end of the file on, the content is empty.:length(pos_integer/0) - Reads at most this many bytes, from:offseton. Without it, the read goes to the end of the file. Content that ends sooner is shorter, without an error. Plugins that transform the content on the way back get the whole file, and the range is cut from what they return, so offsets count bytes of the content as you get it (see Plugins).
@spec read(Fil.Disk.t(), Path.t()) :: result(binary())
@spec read(Fil.Ref.t(), keyword()) :: result(binary())
Reads a file. See read/1.
@spec read(Fil.Disk.t(), Path.t(), keyword()) :: result(binary())
Reads a file. See read/1.
Whether this path is a file. Same as exists?(ref, type: :regular), see exists?/1.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "reports/q3.pdf", "%PDF")
iex> Fil.regular?(disk, "reports/q3.pdf")
true
iex> Fil.regular?(disk, "reports")
false
@spec regular?(Fil.Disk.t(), Path.t()) :: boolean()
Whether this path is a file. See regular?/1.
Moves a file and returns the destination ref.
Within one disk, Fil uses the adapter's native rename. Across disks, it copies the file and then deletes the source.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "draft.txt", "World")
iex> {:ok, final} = Fil.rename(disk, "draft.txt", "final.txt")
iex> final
#Fil.Ref<memory:final.txt>
iex> Fil.exists?(disk, "draft.txt")
false
@spec rename(Fil.Disk.t(), Path.t(), Fil.Ref.t() | Path.t()) :: result(Fil.Ref.t())
@spec rename(Fil.Ref.t(), Fil.Ref.t() | Path.t(), keyword()) :: result(Fil.Ref.t())
Moves a file. See rename/2.
Moves a file. See rename/2.
Deletes a file.
Deleting is idempotent: a missing file still returns {:ok, ref}. The returned ref is meant for a restore
function in a future release.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "hello.txt", "World")
iex> {:ok, deleted} = Fil.rm(disk, "hello.txt")
iex> deleted
#Fil.Ref<memory:hello.txt>
iex> {:ok, ^deleted} = Fil.rm(disk, "hello.txt")
iex> Fil.exists?(disk, "hello.txt")
false
@spec rm(Fil.Disk.t(), Path.t()) :: result(Fil.Ref.t())
@spec rm(Fil.Ref.t(), keyword()) :: result(Fil.Ref.t())
Deletes a file. See rm/1.
@spec rm(Fil.Disk.t(), Path.t(), keyword()) :: result(Fil.Ref.t())
Deletes a file. See rm/1.
@spec rm_rf(Fil.Ref.t()) :: result(non_neg_integer())
Removes everything under a prefix and returns the number of deleted files.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "reports/2026/q3.pdf", "%PDF")
iex> Fil.write!(disk, "reports/2026/q4.pdf", "%PDF")
iex> Fil.rm_rf(disk, "reports/2026")
{:ok, 2}On object stores the prefix is matched as a directory: "reports" removes "reports" and everything under
"reports/", but not "reports.txt".
@spec rm_rf(Fil.Disk.t(), Path.t()) :: result(non_neg_integer())
@spec rm_rf(Fil.Ref.t(), keyword()) :: result(non_neg_integer())
Removes everything under a prefix. See rm_rf/1.
@spec rm_rf(Fil.Disk.t(), Path.t(), keyword()) :: result(non_neg_integer())
Removes everything under a prefix. See rm_rf/1.
Builds a URL that grants temporary access to a file.
Options
A disk can set defaults for :expires_in, :disposition and :if_exists with the :signed_url option of
Fil.disk/1. The options of a call replace them.
:method-:getfor a download URL,:putfor a direct upload. The default value is:get.:expires_in- How long the URL stays valid, in seconds. At most 7 days (604800), the cap of S3. The default value is900.:disposition- Thecontent-dispositionof the download, formethod: :getonly.:inlinelets the browser show the file,:attachmentsaves it under the file's name, and{:attachment, filename}under another name, which can be any UTF-8 string. Without it, S3 sends the disposition an object was uploaded with outsideFil, and the other disks send none.:content_type(String.t/0) - The content type of the upload, formethod: :putonly. The client has to send it as thecontent-typeheader, and the file is stored with it. See Uploads below.:size(non_neg_integer/0) - The size of the upload in bytes, formethod: :putonly. The request'scontent-lengthhas to be the same. Browsers set it from the body.:if_exists- What an upload does if a file is already at the path, formethod: :putonly. With:errorthe client sendsif-none-match: *, and an upload that finds a file there fails and leaves it alone, so the URL writes a file once and can't replace it later. Without it (or with:overwrite), an upload replaces the file.:query(list of tuple ofString.t/0,String.t/0values) - Extra query parameters, e.g.[{"trackingInfo", "42"}], for a page that reads them from its own URL. They're signed with the URL, so they can't be changed or added afterwards. Names that signed URLs use themselves on some disk raise on every disk:expires,disposition,content_type,size,if_exists,signature, and anything starting withX-Amz-orresponse-. The default value is[].
Examples
Fil.signed_url(s3, "cv.pdf", expires_in: 300)
#=> {:ok, "https://bucket.s3.eu-central-1.amazonaws.com/cv.pdf?X-Amz-Algorithm=..."}
Fil.signed_url(s3, "uploads/7f3a.pdf", disposition: {:attachment, "Invoice 2026-09.pdf"})
#=> {:ok, "https://bucket.s3.eu-central-1.amazonaws.com/uploads/7f3a.pdf?...&response-content-disposition=..."}S3 signs its own URLs. Local and memory disks can't, so they need Fil.Plugin.URL with a :secret, and Fil.Plug
serves the URLs from your application:
iex> disk =
...> Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
...> |> Fil.Plugin.URL.attach(base_url: "http://localhost:4000/storage", secret: "secret")
iex> {:ok, url} = Fil.signed_url(disk, "cv.pdf")
iex> url =~ ~r"^http://localhost:4000/storage/cv.pdf[?]expires=[0-9]+&signature="
trueWithout the plugin, a local or memory disk returns an error:
iex> disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
iex> {:error, %Fil.UnsupportedError{op: :signed_url, reason: :no_callback}} = Fil.signed_url(disk, "cv.pdf")Uploads
A URL signed with method: :put takes a PUT with the file as its body. :content_type, :size and
if_exists: :error tie it to one upload, because the URL's signature covers them:
Fil.signed_url(disk, "avatars/7f3a.png",
method: :put,
content_type: "image/png",
size: 48_213,
if_exists: :error
)The client then has to send these headers, on every disk:
content-type: image/pngcontent-length: 48213, which browsers set from the body (a script can't set it)if-none-match: *
A request with another content type or length is refused with a 403, by S3 and by Fil.Plug. With
if_exists: :error, an upload to a path that has a file fails and leaves the file alone: S3 answers 412,
Fil.Plug answers 409. So the URL writes once, and nobody who has it can replace the file later. Some
S3-compatible servers ignore if-none-match (see Fil.Adapter.S3).
@spec signed_url(Fil.Disk.t(), Path.t()) :: result(String.t())
@spec signed_url(Fil.Ref.t(), keyword()) :: result(String.t())
Builds a signed URL. See signed_url/1.
@spec signed_url(Fil.Disk.t(), Path.t(), keyword()) :: result(String.t())
Builds a signed URL. See signed_url/1.
@spec stat(Fil.Ref.t()) :: result(Fil.Stat.t())
Returns metadata for a file or directory.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "hello.txt", "World")
iex> {:ok, stat} = Fil.stat(disk, "hello.txt")
iex> {stat.size, stat.type}
{5, :regular}
iex> {:error, %Fil.NotFoundError{path: "nope.txt", reason: :enoent}} = Fil.stat(disk, "nope.txt")Options
:checksum- Fills inFil.Stat's:checksumfor this algorithm. S3 returns the checksum stored with the object, ornilif it was written without one. The local filesystem computes it by reading the file.:type- Counts only a file (:regular) or only a directory (:directory), the values ofFil.Stat's:type. Anything else is aFil.NotFoundError. Adapters can skip looking for the other type, which saves S3 a request.
@spec stat(Fil.Disk.t(), Path.t()) :: result(Fil.Stat.t())
@spec stat(Fil.Ref.t(), keyword()) :: result(Fil.Stat.t())
Returns metadata for a file or directory. See stat/1.
@spec stat(Fil.Disk.t(), Path.t(), keyword()) :: result(Fil.Stat.t())
Returns metadata for a file or directory. See stat/1.
@spec stream(Fil.Ref.t()) :: result(Enumerable.t())
Streams a file: checks that it can be read and returns its content as a stream of binaries.
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "hello.txt", "World")
iex> {:ok, hello} = Fil.stream(disk, "hello.txt")
iex> Enum.join(hello)
"World"
iex> {:error, %Fil.NotFoundError{path: "nope.txt"}} = Fil.stream(disk, "nope.txt")The file is read only when the stream is enumerated, and again every time it is, so a stream can be read by another
process, more than once, or not at all. The size of the chunks depends on the adapter and says nothing about the
content. An error after the check, such as a file deleted in between or a lost connection, raises the same error
struct read/2 would return. With verify_checksum: true, a mismatch raises Fil.ChecksumMismatchError at the
latest after the last chunk, so treat the content as unverified until then.
A stream can go straight into a write, to the same disk or another one:
{:ok, backup} = Fil.stream(s3, "backups/2026-09.tar")
Fil.write(local, "restore/2026-09.tar", backup)With :offset and :length, the stream reads only that part of the file, as Fil.Plug does for a Range request:
{:ok, clip} = Fil.stream(disk, "videos/intro.mp4", offset: 1_048_576, length: 65_536)Options
:verify_checksum(boolean/0) - Checks the content against the checksum stored with it and returnsFil.ChecksumMismatchErrorif they differ. Content without a stored checksum (see the:checksumoption ofwrite/4), or on storage that stores none, is returned unchecked. A stored checksum covers the whole file, so it can't be combined with:offsetor:length. The default value isfalse.:offset(non_neg_integer/0) - Reads from this byte on, counted from 0. The storage reads only the part that's asked for (S3 with aRangeheader). From the end of the file on, the content is empty.:length(pos_integer/0) - Reads at most this many bytes, from:offseton. Without it, the read goes to the end of the file. Content that ends sooner is shorter, without an error. Plugins that transform the content on the way back get the whole file, and the range is cut from what they return, so offsets count bytes of the content as you get it (see Plugins).
@spec stream(Fil.Disk.t(), Path.t()) :: result(Enumerable.t())
@spec stream(Fil.Ref.t(), keyword()) :: result(Enumerable.t())
Streams a file. See stream/1.
@spec stream(Fil.Disk.t(), Path.t(), keyword()) :: result(Enumerable.t())
Streams a file. See stream/1.
Builds the public URL of a file.
The URL has no signature and doesn't expire, so it only works where the file can be downloaded by anyone: a public
bucket, a CDN, or Fil.Plug with public: true.
Examples
Fil.url(s3, "logo.png")
#=> {:ok, "https://bucket.s3.eu-central-1.amazonaws.com/logo.png"}S3 builds the URL from the bucket. Local and memory disks have no URL of their own, so they get one from
Fil.Plugin.URL, and Fil.Plug serves it from your application:
iex> disk =
...> Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
...> |> Fil.Plugin.URL.attach(base_url: "http://localhost:4000/avatars")
iex> Fil.url(disk, "1.png")
{:ok, "http://localhost:4000/avatars/1.png"}Without the plugin, a local or memory disk returns an error:
iex> disk = Fil.disk(adapter: Fil.Adapter.Local, root: "/tmp/fil")
iex> {:error, %Fil.UnsupportedError{op: :url, reason: :no_callback}} = Fil.url(disk, "1.png")
@spec url(Fil.Disk.t(), Path.t()) :: result(String.t())
@spec url(Fil.Ref.t(), keyword()) :: result(String.t())
Builds a public URL. See url/1.
@spec url(Fil.Disk.t(), Path.t(), keyword()) :: result(String.t())
Builds a public URL. See url/1.
@spec wildcard(Fil.Disk.t(), String.t()) :: result([Fil.Ref.t()])
Lists the paths that match a pattern, like Path.wildcard/2 with the disk root as the working directory.
The syntax is Path.wildcard/2's: * matches any characters within a name, ? a single one, [a-z] one of a
class, {pdf,csv} one of the alternatives, and a ** segment any number of directories. \ makes the next
character literal. The pattern is relative to the disk root, so a leading / changes nothing, and neither do // or
a trailing /. Names that
start with a dot are matched only by the literal start of a pattern (.cache/*), unless you pass match_dot: true.
The result holds files and directories, sorted by path, unless type: picks one:
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> Fil.write!(disk, "videos/intro/withText/intro.mp4", "...")
iex> Fil.write!(disk, "videos/outro/withoutText/outro.mp4", "...")
iex> {:ok, [with_text, without_text]} = Fil.wildcard(disk, "videos/*/{withText,withoutText}")
iex> with_text
#Fil.Ref<memory:videos/intro/withText>
iex> without_text
#Fil.Ref<memory:videos/outro/withoutText>
iex> {:ok, [intro, outro]} = Fil.wildcard(disk, "videos/**/*.mp4", type: :regular)
iex> {intro.path, outro.path}
{"videos/intro/withText/intro.mp4", "videos/outro/withoutText/outro.mp4"}No storage matches patterns itself, so Fil lists the directories one level at a time with ls/3: the directory
that the literal start of the pattern names (videos for videos/*/withText), then each directory a segment
matched. On S3 that's one request per listed directory and level, the same as nested calls to ls/3. A ** segment
lists the whole subtree of each directory it starts from, so "**/*.mp4" is one recursive listing of the disk, and
on a large bucket a prefix in front of it ("videos/**/*.mp4") keeps that listing small. Each listing is an
operation of its own, so plugins and [:fil, :op] telemetry events see every one of them. The first one that fails
is the result.
A pattern built from a variable needs that part escaped if it can hold *, ?, [, { or \, as with
Path.wildcard/2. An invalid pattern raises ArgumentError.
Options
:type- Returns only files (:regular) or only directories (:directory). Without it, both.:match_dot(boolean/0) - Lets*,?and the other wildcards match names that start with a dot, as inPath.wildcard/2. The default value isfalse.
@spec wildcard(Fil.Disk.t(), String.t(), keyword()) :: result([Fil.Ref.t()])
Lists the paths that match a pattern. See wildcard/2.
Writes a file, creating missing parent directories.
content is iodata, a stream of it, or {:file, path} for a file on the local filesystem (see content/0). A
stream is written as it's read, without holding the content in memory at once, and a file is only there once the
stream has ended. If reading the stream raises, nothing is written: one of Fil's errors (from a stream of
stream/3, say) comes back as {:error, error}, and any other exception propagates. S3 sends a stream whose size it
knows in one request, and uploads any other in parts. Fil finds the size of a local file, of a File.Stream of
bytes and of a stream from stream/3 itself; for any other stream, pass it with :size if you know it.
{:file, path} streams the file in chunks of 64 KiB, the same as File.stream!(path, 65_536). A file that doesn't
exist, or a directory, raises File.Error before anything runs, as it's the caller's input, like the content itself.
Any other failure to read it raises File.Error while it's written, as a File.Stream does, and nothing is written.
The local file's name plays no part: the content type comes from :content_type or from Fil.Plugin.ContentType,
which looks at the destination path.
Options
:if_exists- What to do if the file (for a copy or a move, the destination) already exists.:overwritereplaces it.:errorwrites nothing and returns aFil.AlreadyExistsError, likeFile.write/3with[:exclusive], and a move leaves the source where it is. That check is atomic on local disk, in memory and on AWS S3, so two processes can't both create the file. Some S3-compatible servers ignore it. The default value is:overwrite.:content_type(String.t/0) - Stored as the object's content type where the storage keeps one.:checksum- Computes a checksum of the content with this algorithm (:sha256,:sha1or:crc32) and sends it along, where the storage supports it. S3 rejects the write withFil.ChecksumMismatchErrorif the content it received doesn't match, and stores the checksum with the object. The local filesystem stores nothing. S3 uploads a stream with a checksum in parts, and for SHA-1 and SHA-256 then stores a checksum of the parts' checksums instead of the file's, so use:crc32for large streams whose checksum you need later (see Checksums inFil.Adapter.S3).:size- The size of the content in bytes. S3 sends a stream of known size as it's read, in one request, and uploads one without a size in parts. Content of another size raisesArgumentErrorand writes nothing, whatever the plugins do with it. Plugins that transform the content drop the size.Without it,
{:file, path}has the size of its file, and so does aFile.Streamthat reads bytes (File.stream!(path, 65_536), not lines) without a mode that changes them (:compressed,:trim_bom, an encoding), minus its:read_offset. A stream fromstream/3has the size its adapter found. When such content turns out to have another size, because the file changed while it was read, the write returnsFil.ConflictErrorand writes nothing.size: :unknownwrites the stream as it's read, without a size (S3 uploads it in parts). Use it for a file that grows while it's written, such as a log that's still appended to, which is then written as far as it was read, and for files whose stat size may be wrong, such as those under/sysor on a network or FUSE file system.
Examples
iex> disk = Fil.disk(adapter: Fil.Adapter.Memory)
iex> {:ok, report} = Fil.write(disk, "reports/q3.pdf", ["%PDF", "-1.7"])
iex> report
#Fil.Ref<memory:reports/q3.pdf>
iex> {:error, %Fil.AlreadyExistsError{reason: :eexist}} =
...> Fil.write(disk, "reports/q3.pdf", "again", if_exists: :error)
iex> {:ok, _} = Fil.write(disk, "numbers.txt", Stream.map(1..3, &Integer.to_string/1))
iex> Fil.read(disk, "numbers.txt")
{:ok, "123"}A local file, such as an upload's temporary file, is written as it's read:
Fil.write(s3, "videos/intro.mp4", {:file, "intro.mp4"})
Fil.write(s3, "documents/upload.pdf", {:file, upload.path})
@spec write(Fil.Disk.t(), Path.t(), content()) :: result(Fil.Ref.t())
@spec write(Fil.Ref.t(), content(), keyword()) :: result(Fil.Ref.t())
Writes a file. See write/2.
Writes a file. See write/2.
Bang variants
Same as cp/2, raising the error on failure.
@spec cp!(Fil.Disk.t(), Path.t(), Fil.Ref.t() | Path.t()) :: Fil.Ref.t()
@spec cp!(Fil.Ref.t(), Fil.Ref.t() | Path.t(), keyword()) :: Fil.Ref.t()
Same as cp/3, raising the error on failure.
Same as cp/4, raising the error on failure.
@spec ls!(Fil.Disk.t()) :: [Fil.Ref.t()]
@spec ls!(Fil.Ref.t()) :: [Fil.Ref.t()]
Same as ls/1, raising the error on failure.
@spec ls!(Fil.Disk.t(), Path.t()) :: [Fil.Ref.t()]
@spec ls!(Fil.Ref.t(), keyword()) :: [Fil.Ref.t()]
Same as ls/2, raising the error on failure.
@spec ls!(Fil.Disk.t(), Path.t(), keyword()) :: [Fil.Ref.t()]
Same as ls/3, raising the error on failure.
Same as read/1, raising the error on failure.
Same as read/2, raising the error on failure.
@spec read!(Fil.Disk.t(), Path.t(), keyword()) :: binary()
Same as read/3, raising the error on failure.
Same as rename/2, raising the error on failure.
@spec rename!(Fil.Disk.t(), Path.t(), Fil.Ref.t() | Path.t()) :: Fil.Ref.t()
@spec rename!(Fil.Ref.t(), Fil.Ref.t() | Path.t(), keyword()) :: Fil.Ref.t()
Same as rename/3, raising the error on failure.
Same as rename/4, raising the error on failure.
Same as rm/1, raising the error on failure.
Same as rm/2, raising the error on failure.
@spec rm!(Fil.Disk.t(), Path.t(), keyword()) :: Fil.Ref.t()
Same as rm/3, raising the error on failure.
@spec rm_rf!(Fil.Ref.t()) :: non_neg_integer()
Same as rm_rf/1, raising the error on failure.
@spec rm_rf!(Fil.Disk.t(), Path.t()) :: non_neg_integer()
@spec rm_rf!(Fil.Ref.t(), keyword()) :: non_neg_integer()
Same as rm_rf/2, raising the error on failure.
@spec rm_rf!(Fil.Disk.t(), Path.t(), keyword()) :: non_neg_integer()
Same as rm_rf/3, raising the error on failure.
Same as signed_url/1, raising the error on failure.
@spec signed_url!(Fil.Disk.t(), Path.t()) :: String.t()
@spec signed_url!(Fil.Ref.t(), keyword()) :: String.t()
Same as signed_url/2, raising the error on failure.
@spec signed_url!(Fil.Disk.t(), Path.t(), keyword()) :: String.t()
Same as signed_url/3, raising the error on failure.
@spec stat!(Fil.Ref.t()) :: Fil.Stat.t()
Same as stat/1, raising the error on failure.
@spec stat!(Fil.Disk.t(), Path.t()) :: Fil.Stat.t()
@spec stat!(Fil.Ref.t(), keyword()) :: Fil.Stat.t()
Same as stat/2, raising the error on failure.
@spec stat!(Fil.Disk.t(), Path.t(), keyword()) :: Fil.Stat.t()
Same as stat/3, raising the error on failure.
@spec stream!(Fil.Ref.t()) :: Enumerable.t()
Same as stream/1, raising the error on failure.
@spec stream!(Fil.Disk.t(), Path.t()) :: Enumerable.t()
@spec stream!(Fil.Ref.t(), keyword()) :: Enumerable.t()
Same as stream/2, raising the error on failure.
@spec stream!(Fil.Disk.t(), Path.t(), keyword()) :: Enumerable.t()
Same as stream/3, raising the error on failure.
Same as url/1, raising the error on failure.
Same as url/2, raising the error on failure.
@spec url!(Fil.Disk.t(), Path.t(), keyword()) :: String.t()
Same as url/3, raising the error on failure.
@spec wildcard!(Fil.Disk.t(), String.t()) :: [Fil.Ref.t()]
Same as wildcard/2, raising the error on failure.
@spec wildcard!(Fil.Disk.t(), String.t(), keyword()) :: [Fil.Ref.t()]
Same as wildcard/3, raising the error on failure.
Same as write/2, raising the error on failure.
@spec write!(Fil.Disk.t(), Path.t(), content()) :: Fil.Ref.t()
@spec write!(Fil.Ref.t(), content(), keyword()) :: Fil.Ref.t()
Same as write/3, raising the error on failure.
@spec write!(Fil.Disk.t(), Path.t(), content(), keyword()) :: Fil.Ref.t()
Same as write/4, raising the error on failure.
Types
@type content() :: iodata() | Enumerable.t() | {:file, String.t()}
What write/4 takes: iodata, a stream of it (any Enumerable that isn't a list, such as a Stream or a
File.Stream), or {:file, path} for a file on the local filesystem. A list is always iodata.
@type error() :: Fil.NotFoundError.t() | Fil.AccessDeniedError.t() | Fil.InvalidRequestError.t() | Fil.InvalidContentError.t() | Fil.AlreadyExistsError.t() | Fil.ConflictError.t() | Fil.ChecksumMismatchError.t() | Fil.StorageFullError.t() | Fil.UnsupportedError.t() | Fil.ConfigurationError.t() | Fil.UnavailableError.t() | Fil.UnknownError.t()
An error from an adapter or from Fil itself. Each struct stands for what the caller can do about it, and means the
same on every adapter; see Errors in Fil.Adapter.
@type plugin_callback() :: (Fil.Op.t(), (Fil.Op.t() -> Fil.Op.t()), keyword() -> Fil.Op.t()) | {module(), atom()}
A plugin callback: a function or a {module, function} pair. See the Plugins guide.
@type result(value) :: {:ok, value} | {:error, error() | Exception.t()}
A result. Plugins may return their own exceptions, so an error isn't always one of error/0.