Fil.Op (Fil v0.2.0)

Copy Markdown View Source

One call to a Fil function, as plugins see it.

Every operation builds a %Fil.Op{} and passes it through the disk's plugins to the adapter. See the Plugins guide for how to write a plugin. Each run emits the events in Fil.Telemetry, and their duration includes the plugins.

  • :disk: the Fil.Disk the operation runs on
  • :name: the operation, :read, :write, :stat, :ls, :rm, :rm_rf, :cp, :rename, :url or :signed_url
  • :path: the normalized path, relative to the disk root
  • :dest: the destination path of a :cp or :rename on the same disk, nil otherwise
  • :content: the content of a :write, nil otherwise: iodata, or a stream of it. Change it with update_content/2, and read it without changing it with scan_content/4
  • :streaming: true for a :read from Fil.stream/3, whose result is a stream instead of a binary
  • :options: the validated options of the call
  • :result: nil on the way in, then {:ok, value} or {:error, exception} with the same value the Fil function returns. Change a read result with update_result/2, and read it with scan_result/4
  • :private: a map for plugins to pass data along

Summary

Functions

Returns an option of the call.

Returns a private value.

Collects the content of a :write into one binary, for plugins that need all of it at once.

Sets an option unless the call already has it.

Sets an option before the adapter sees it.

Stores a private value, for a later plugin or for the way back.

Sets the result.

Reads the content of a :write as it passes, without changing it. Other operations are returned unchanged.

Reads the content returned by a successful :read as it passes, without changing it. Other operations and errors are returned unchanged.

Transforms the content of a :write. Other operations are returned unchanged.

Transforms the content returned by a successful :read. Other operations and errors are returned unchanged.

Types

name()

@type name() ::
  :read
  | :write
  | :stat
  | :ls
  | :rm
  | :rm_rf
  | :cp
  | :rename
  | :url
  | :signed_url

t()

@type t() :: %Fil.Op{
  content: iodata() | Enumerable.t() | nil,
  dest: Path.t() | nil,
  disk: Fil.Disk.t(),
  name: name(),
  options: keyword(),
  path: Path.t(),
  private: map(),
  result: {:ok, term()} | {:error, Exception.t()} | nil,
  streaming: boolean()
}

transform()

@type transform() :: [
  iodata: (binary() -> iodata()),
  stream: (Enumerable.t() -> Enumerable.t())
]

Functions

get_option(op, key, default \\ nil)

@spec get_option(t(), atom(), term()) :: term()

Returns an option of the call.

iex> op = %Fil.Op{disk: nil, name: :write, path: "a.txt", options: [content_type: "text/plain"]}
iex> Fil.Op.get_option(op, :content_type)
"text/plain"
iex> Fil.Op.get_option(op, :missing, :default)
:default

get_private(op, key, default \\ nil)

@spec get_private(t(), term(), term()) :: term()

Returns a private value.

iex> op = %Fil.Op{disk: nil, name: :read, path: "a.txt"}
iex> Fil.Op.get_private(op, :seen, false)
false

materialize(op)

@spec materialize(t()) :: t()

Collects the content of a :write into one binary, for plugins that need all of it at once.

Iodata is flattened into a binary. A stream is collected into memory, so use this only when a plugin can't work chunk by chunk.

iex> op = %Fil.Op{disk: nil, name: :write, path: "a.txt", content: ["a", ["b"]]}
iex> Fil.Op.materialize(op).content
"ab"

put_new_option(op, key, value)

@spec put_new_option(t(), atom(), term()) :: t()

Sets an option unless the call already has it.

iex> op = %Fil.Op{disk: nil, name: :write, path: "a.txt", options: [content_type: "text/plain"]}
iex> op |> Fil.Op.put_new_option(:content_type, "text/csv") |> Fil.Op.get_option(:content_type)
"text/plain"

put_option(op, key, value)

@spec put_option(t(), atom(), term()) :: t()

Sets an option before the adapter sees it.

iex> op = %Fil.Op{disk: nil, name: :write, path: "a.txt", options: [content_type: "text/plain"]}
iex> op |> Fil.Op.put_option(:content_type, "text/csv") |> Fil.Op.get_option(:content_type)
"text/csv"

put_private(op, key, value)

@spec put_private(t(), term(), term()) :: t()

Stores a private value, for a later plugin or for the way back.

iex> op = %Fil.Op{disk: nil, name: :read, path: "a.txt"}
iex> op |> Fil.Op.put_private(:seen, true) |> Fil.Op.get_private(:seen)
true

put_result(op, result)

@spec put_result(t(), {:ok, term()} | {:error, Exception.t()}) :: t()

Sets the result.

A plugin that sets the result instead of calling next answers the call itself, and the adapter never runs. An error is an exception: one of Fil's (see Errors) or the plugin's own. Fil fills in the :op, :path and :disk of its own errors where the plugin left them nil. On a read with offset: or length:, content set instead of calling next is taken as the whole file, and Fil cuts the part from it. Content set after next returned the part (to wrap the stream, say) stays the part (see Ranges).

scan_content(op, acc, fun, done \\ &scanned/1)

@spec scan_content(t(), acc, (binary(), acc -> acc), (acc -> term())) :: t()
when acc: term()

Reads the content of a :write as it passes, without changing it. Other operations are returned unchanged.

fun gets each chunk with the accumulator and returns the next one, and done gets the last accumulator after the last chunk. Content in memory is one chunk, read right away. A stream is read when the adapter reads it, chunk by chunk, and each chunk is passed on only after fun has seen it. Empty content never reaches fun, but always done.

The content, its chunks and the :size option stay as they are, so S3 still sends a stream of known size in one request. To refuse the content, raise Fil.InvalidContentError from fun or done: the write returns it, and nothing is written, even when done raises after the last chunk. Any other exception propagates.

This one refuses content over 1 MB as it arrives:

Fil.Op.scan_content(op, 0, fn chunk, size ->
  size = size + byte_size(chunk)
  if size > 1_000_000, do: raise(%Fil.InvalidContentError{reason: :too_large}), else: size
end)

See Inspecting content.

scan_result(op, acc, fun, done \\ &scanned/1)

@spec scan_result(t(), acc, (binary(), acc -> acc), (acc -> term())) :: t()
when acc: term()

Reads the content returned by a successful :read as it passes, without changing it. Other operations and errors are returned unchanged.

Takes the same functions as scan_content/4. Content in memory is read right away, and one of Fil's errors raised from fun or done, such as Fil.InvalidContentError, turns the read into that error. The result of Fil.stream/3 is read when the caller reads it, and the error is raised to the caller then, with the operation, the path and the disk filled in. done runs only when the caller reads to the end: Enum.take/2 stops before it. On a read with offset: or length:, the scan reads the part the caller gets, not the whole file.

update_content(op, funs)

@spec update_content(t(), transform()) :: t()

Transforms the content of a :write. Other operations are returned unchanged.

The content is iodata or a stream, and each has its transform:

Fil.Op.update_content(op, iodata: &:zlib.gzip/1, stream: &MyApp.Gzip.stream/1)
  • iodata: gets all of the content as one binary and returns iodata. Without stream:, it gets a stream too: Fil collects the stream into memory first
  • stream: gets the stream, an enumerable of binaries, and returns an enumerable of iodata. A transform of each chunk on its own is &Stream.map(&1, fun). Without iodata:, it gets content in memory too, as a stream of one chunk (none if it's empty), and the result is collected again

The Plugins guide describes what a chunk is. A transform drops the :size option of the write, because the size can change. A plugin that knows the new size declares it again with put_option(op, :size, size).

update_result(op, funs)

@spec update_result(t(), transform()) :: t()

Transforms the content returned by a successful :read. Other operations and errors are returned unchanged.

Takes the same iodata: and stream: functions as update_content/2. The result of Fil.stream/3 is a stream, and the transforms run when the caller reads it. With only iodata:, the stream is collected then too.

On a read with offset: or length:, the adapter reads only that part of the file, but a transform gets the whole file (it's read again) and Fil cuts the part from what the plugins return. See Ranges.

A transform that raises one of Fil's errors, such as Fil.ChecksumMismatchError for content that fails a check, turns a read that returns a binary into that error. On a stream, the error is raised when the caller reads it, with the operation, the path and the disk filled in.