Amazon S3 and services that implement its API, such as RustFS, SeaweedFS, Adobe S3Mock, Cloudflare R2, Backblaze B2, Tigris and Ceph.

disk =
  Fil.disk(
    adapter: Fil.Adapter.S3,
    bucket: "invoices",
    region: "eu-central-1",
    access_key_id: System.fetch_env!("AWS_ACCESS_KEY_ID"),
    secret_access_key: System.fetch_env!("AWS_SECRET_ACCESS_KEY")
  )

Requests are sent and signed (SigV4) by Req, configured with :req_options. Listings are parsed with OTP's :xmerl_sax_parser.

A stream of known size (see the :size option of Fil.write/4) is sent as it's read, and that needs an HTTP/1 connection pool, which is what Req uses unless :req_options asks for HTTP/2 (for example with connect_options: [protocols: [:http2]] or a :finch pool for HTTP/2). On HTTP/2, Finch reads a request body in the pool's process instead of the caller's, and a stream that only the caller's process can read fails or stalls there: a Fil.Plug upload, or a stream from another S3 disk (Fil.stream/3, or a copy across disks). Content in memory, uploads in parts and streams that any process can read, such as a File.Stream, are fine.

Options

• :bucket (String.t/0) - Required. The bucket all paths are stored in.

• :region (String.t/0) - The bucket's region. The default value is "us-east-1".

• :root (String.t/0) - The key prefix every path is resolved against, e.g. "uploads". Paths on the disk stay relative to it, and listings strip it again. Defaults to the bucket root. The default value is ".".

• :access_key_id (String.t/0) - The access key. Fil doesn't look up credentials itself (no environment variables, no instance metadata), so where they come from is up to the application. Leave out all credentials to access a public bucket without signing.

• :secret_access_key (String.t/0) - The secret matching :access_key_id.

• :session_token (String.t/0) - The session token, for temporary STS credentials.

• :endpoint (String.t/0) - A base URL for S3-compatible services, e.g. "http://localhost:9000". Setting it turns :path_style on.

• :public_endpoint (String.t/0) - The base URL clients reach the storage at, when it isn't :endpoint, e.g. "http://localhost:9090" for a container the application reaches as "http://s3mock:9090". Fil.url/2 and Fil.signed_url/3 build their URLs with it, and the requests the disk makes itself still go to :endpoint. A signature covers the host, so a signed URL can't be rewritten to another host afterwards. :path_style applies to both, but only :endpoint turns it on by default.

• :path_style (boolean/0) - Put the bucket in the path (https://host/bucket/key) instead of the hostname (https://bucket.host/key). Defaults to true when :endpoint is set, false otherwise.

• :part_size - The size in bytes of the parts an upload in parts uses (a stream without a size or with checksum:, and content over 5 GiB), 8 MiB by default and at least 5 MiB (S3's minimum). An upload keeps one part in memory at a time. S3 allows 10,000 parts, so a stream without a size can be at most 78 GiB at the default. Pass size: or raise :part_size for larger ones: with size:, the parts grow to fit. See Uploads in parts. The default value is 8388608.

• :req_options (keyword/0) - Options for every Req request the disk makes, such as :receive_timeout, :connect_options or a shared :finch pool. The adapter always sets :method, :url, :headers and :body, plus retry: false (retrying is up to the caller) and raw: true (no decompression and no body decoding, so a file reads back exactly as it was written). A stream sent with its size needs HTTP/1, see above. The default value is [].

Operations

For each Fil function, this section says which S3 requests it sends and where it departs from the contract that every adapter follows. Anything it doesn't mention follows the contract. Directories exist only as key prefixes, so there are no empty directories.

Fil.read/3

• Uses GetObject.
• Reading a directory returns a Fil.NotFoundError.
• offset: and length: send a Range header. S3 answers a range that starts at the end of the object with a 416, which is read as empty content.
• verify_checksum: true asks S3 for the stored checksum (x-amz-checksum-mode: ENABLED) and compares it with the downloaded content, including the composite checksum of an upload in parts (see Checksums). Objects stored with another algorithm, or with none, are read without a check.

Fil.stream/3

• Uses HeadObject first, then GetObject each time the stream is read.
• The download runs in a process of its own and goes only as fast as the stream is read.
• offset: and length: are clamped to the size from the HeadObject and sent as a Range header. An empty range downloads nothing.
• verify_checksum: true computes the checksum while streaming.

Fil.write/4

• Uses PutObject for content in memory and for a stream of known size (size:, a File.Stream, a stream from Fil.stream/3), which is sent as it's read. The last chunk goes out only once the stream has ended, so a stream that raises after its last chunk leaves no object.
• Uploads in parts instead for a stream without a size (or with size: :unknown), a stream with checksum:, and anything over 5 GiB (see Uploads in parts).
• With if_exists: :error, sends If-None-Match: *.
• checksum: (:sha256, :sha1 or :crc32) sends the checksum of the content in x-amz-checksum-*. S3 rejects the upload if what it received doesn't match, and stores the checksum with the object.
• Writing to report.txt/x when report.txt is an object writes a second object and leaves the first alone.

Fil.rm/3

• Uses DeleteObject, which S3 already treats as idempotent. A 404 for a missing bucket is still an error.
• Removing a directory succeeds and removes nothing.

Fil.stat/3

• Uses HeadObject. If there's no object, it checks for keys with the path as their prefix (ListObjectsV2 with max-keys=2), so Fil.dir?/1 works.
• type: :regular sends only the HeadObject, and type: :directory (Fil.dir?/1) only the prefix check. So a key that is an object and a prefix at once is both a file and a directory.
• A directory marker alone doesn't make a directory for the prefix check, though a listing of its parent lists it.
• :etag and :content_type are the ones S3 returns. The ETag of an upload in parts ends in - and the number of parts.
• checksum: returns the checksum S3 stored if the write used the same algorithm, and nil otherwise. It's nil for the composite checksum of an upload in parts too.

Fil.ls/3

• Uses ListObjectsV2, with delimiter=/ unless recursive, and fetches every page.
• A recursive listing finds the directories in the keys, so an empty directory shows up only as a directory marker (an empty object whose key ends in /, as consoles create for a new folder), which is listed as the directory.

Fil.cp/4

• Uses CopyObject.
• With if_exists: :error, sends If-None-Match: *. AWS checks it on copies since October 2025, and RustFS 1.0.0 does too.
• Copying a directory returns a Fil.NotFoundError.
• S3 refuses to copy an object onto itself, so that's a Fil.InvalidRequestError with reason: "InvalidRequest", with or without if_exists: :error. The other adapters copy the file onto itself, or return a Fil.AlreadyExistsError with if_exists: :error.
• A copy of a file uploaded in parts gets a new checksum of the whole file, and doesn't keep the one of the parts.

Fil.rename/4

• Uses CopyObject, then DeleteObject.
• A copy that fails (with if_exists: :error too) fails before the source is deleted.
• Renaming an object onto itself fails like the copy.
• The DeleteObject removes whatever is at the source by then, so a write that replaces the source between the two requests is lost. The other adapters keep it, and with if_exists: :error fail the move with a Fil.ConflictError.

Fil.rm_rf/3

• Uses ListObjectsV2, then one DeleteObject per key.

Fil.url/3

• Returns the object URL, without a signature, so it works for public objects only.

Fil.signed_url/3

• Returns a presigned GET or PUT URL.
• disposition: becomes response-content-disposition, and the query: parameters are added to the URL.
• For an upload, content_type:, size: and if_exists: :error are signed headers (content-type, content-length and if-none-match: *). S3 refuses a PUT without them, or with other values, with a 403, and answers 412 when if_exists: :error finds an object.

Fil.Disk.same_storage?/2

• Compares :endpoint, :region, :bucket and the prefix from :root. Disks that differ only in credentials, :public_endpoint, :path_style or :req_options are the same storage.

Details

Uploads in parts

A stream without a size (or with size: :unknown), a stream with checksum:, and content over 5 GiB (the largest PutObject) are read one part at a time, :part_size bytes each, so an upload keeps about one part and one chunk in memory. Content that ends within the first part goes out as one PutObject once it has ended. Anything longer is a multipart upload:

• CreateMultipartUpload once the second part begins, with the content type and the checksum algorithm
• UploadPart for each part once more content has arrived after it, signed with its SHA-256, which S3 checks. The parts go out one after the other from the calling process, so this works on HTTP/2 too. A part that fails with Fil.UnavailableError (throttling such as SlowDown included) is sent once more, a second later. It's the only request Fil repeats: nobody sees a part before the upload completes
• CompleteMultipartUpload after the stream has ended, so nothing is written unless it ends. if_exists: :error sends If-None-Match: * with it (S3 takes it nowhere else), so a write that finds the file already there fails only at the end

S3 allows 10,000 parts. A stream without a size that would need more fails with Fil.InvalidRequestError, reason: :too_many_parts, before its last part goes out: that's 78 GiB at the default part size, so pass size: or raise :part_size for larger ones. With size:, the parts are as large as the size needs, rounded up to a whole MiB (525 MiB for 5 TiB, the largest object S3 stores). A size: over 5 TiB fails with Fil.InvalidRequestError, reason: "EntityTooLarge", before the stream is read.

Completing a large upload can take a while. AWS answers with a 200 right away and sends whitespace until it's done, and :receive_timeout applies between packets, so it doesn't cut the completion off. An error that comes after the 200 is read like any other. Whether RustFS keeps the connection alive the same way hasn't been checked.

Any failure aborts the upload (AbortMultipartUpload), and so does a stream that raises. When the writing process is killed (a supervisor shutdown, or Cowboy stopping the request of a client that disconnected), a process of its own that watches the writer aborts the upload. The abort can still be missed: when the node goes down, or when the abort request fails. S3 doesn't list the parts of an incomplete upload, but bills them, so give the bucket a lifecycle rule that aborts incomplete multipart uploads (AbortIncompleteMultipartUpload, for example after one day). It also aborts uploads that are still running after that time.

For Fil.Telemetry, an upload in parts is one :write, with no events for the parts. Its requests are Finch events, and the creation and the abort run in a process of their own (see HTTP requests in Fil.Telemetry).

Checksums

checksum: :crc32 covers the whole file, however it's uploaded. Each part is sent with its CRC32, and the completion with the CRC32 of all of the content (FULL_OBJECT), which S3 checks and stores as it does for a PutObject.

S3 can't compute a SHA-1 or SHA-256 over an upload in parts. With :sha1 or :sha256, each part is sent with its checksum, which S3 checks, and S3 stores a composite checksum: the checksum of the parts' checksums, followed by - and the number of parts. Fil.stat/3 returns nil for it, because it isn't the checksum of the file. verify_checksum: true asks for the size of the first part (a HeadObject with partNumber=1), checksums the content part by part and compares the result. That works for uploads whose parts all have one size but the last, as Fil's and those of AWS's tools do. Other objects, and objects on a server that ignores partNumber (RustFS 1.0.0 does), are read without a check. Use :crc32 for large streams if you need the checksum of the file later.

Errors

:reason is the error code from the response body ("NoSuchKey"), or {:http_status, status} when there's none. Failures before a response keep Req's reason (:timeout, an exception). The error code decides first, whatever the status, because S3-compatible servers don't all send the same one.