resx v0.1.0 Resx.Producers.File

A producer to handle file URIs.

Resx.Producers.File.open("file:///path/to/file.txt")

Types

MIME types are inferred from file extension names. A chained file extension will result in multiple MIME types (e.g. file.jpg.txt => ["text/plain", "image/jpeg"]). Unsupported types will default to "application/octet-stream". Custom MIME types can be added to the config.

Hostnames

Valid hostnames can either be an erlang node name, localhost, or not specified.

When referencing a file without providing a hostname or using localhost, the file will be referenced from the calling node. This means that if a reference was created and has been passed to another node, any requests that need to access this file will then be sent back to the original node to be processed.

Files/Directory Access

Files that can be opened need to be explicitly included. This can be done by configuring the :access configuration option for Resx.Producers.File.

config :resx, Resx.Producers.File,
    access: [
        "path/to/file.txt",
        "path/*/*.jpg",
        "**/*.{ex,exs}",
        ~r/.*?/to/another.txt/,
        { MyFileAccessGranter, :can_access?, 1 },
        { :"foo@127.0.0.1", "**/some-file.txt" },
        { &(&1 in [:"foo@127.0.0.1", :"bar@127.0.0.1"]), "**/another-file.txt" }
    ]

The :access field should contain either a list of strings, regexes, or callback functions, which will be applied to every node or it can be tagged with the node ({ node, pattern }) if the rule should only be applied to files found at that node. Callback functions expect a string (glob pattern) and return a boolean. The node may also be a callback function that expects a node and returns a boolean. Valid function formats are any callback variant, see Callback for more information.

File access rules are applied both on the node making the request and the node processing the request. This means that if node foo@127.0.0.1 has the access rules:

[
    "/one.txt",
    {:"foo@127.0.0.1", "/two.txt"},
    {:"bar@127.0.0.1", "/three.txt"}
]

And node bar@127.0.0.1 has the access rules:

[
    "/two.txt",
    {:"bar@127.0.0.1", "/three.txt"}
]

If node foo@127.0.0.1 attempted to open the following files it would get these responses:

# Allowed
Resx.Producers.File.open("file:///one.txt") # => open file on node foo@127.0.0.1
Resx.Producers.File.open("file://foo@127.0.0.1/one.txt") # => open file on node foo@127.0.0.1

Resx.Producers.File.open("file:///two.txt") # => open file on node foo@127.0.0.1
Resx.Producers.File.open("file://foo@127.0.0.1/two.txt") # => open file on node foo@127.0.0.1

Resx.Producers.File.open("file://bar@127.0.0.1/three.txt") # => open file on node bar@127.0.0.1

# Not Allowed
Resx.Producers.File.open("file://bar@127.0.0.1/one.txt")

Resx.Producers.File.open("file://bar@127.0.0.1/two.txt")

Resx.Producers.File.open("file:///three.txt")
Resx.Producers.File.open("file://foo@127.0.0.1/three.txt")

If node bar@127.0.0.1 attempted to open the following files it would get these responses:

# Allowed
Resx.Producers.File.open("file:///two.txt") # => open file on node bar@127.0.0.1
Resx.Producers.File.open("file://foo@127.0.0.1/two.txt") # => open file on node foo@127.0.0.1
Resx.Producers.File.open("file://bar@127.0.0.1/two.txt") # => open file on node bar@127.0.0.1

Resx.Producers.File.open("file:///three.txt") # => open file on node bar@127.0.0.1
Resx.Producers.File.open("file://bar@127.0.0.1/three.txt") # => open file on node bar@127.0.0.1

# Not Allowed
Resx.Producers.File.open("file:///one.txt")
Resx.Producers.File.open("file://foo@127.0.0.1/one.txt")
Resx.Producers.File.open("file://bar@127.0.0.1/one.txt")

Resx.Producers.File.open("file://foo@127.0.0.1/three.txt")

One common rule for nodes that might access files from other nodes, is a generic catch-all. This rule can be written as: { &(&1 != node()), "**" } This will allow the calling node to attempt to access any file on the recieving node.

Glob Pattern

Glob pattern rules follow the syntax of Path.wildcard/2, with the addition of a negative character match [!char1,char2,...]. e.g. [!abc] or [!a-c] (match any character other than a, b, or c).

Distribution

The following operations need to talk to the node the file originates from.

  • Opening a file
  • Checking if a file exists
  • Accessing a file's attributes

These distributed requests are done using the rpc module provided by the erlang runtime. This can be overridden by configuring the :rpc field to a callback function that will be used as the replacement rpc handler. The callback function expects 4 arguments (node, module, fun, args) and should return the result of target function otherwise any non-ok/error tuple to be used as the internal error. Valid function formats are any callback variant, see Callback for more information.

config :resx, Resx.Producers.File,
    rpc: { :gen_rpc, :call, 4 }

Resources contain a reference to the node they came from. So if a resource is passed around to other nodes, it will still be able to guarantee access.

An example of this is in the diagram below. If foo@127.0.0.1 requests to open a file on bar@127.0.0.1, and then passes the resource to baz@127.0.0.1 which then wants to repoen it (get the latest changes) the request will go back to bar@127.0.0.1.

open foo@127.0.0.1 resource resource bar@127.0.0.1 reopen baz@127.0.0.1 resource

Streams

File streaming relies on the File.Stream type, but allows for distributed access. These follow the same distribution rules as previously outlined, with the addition that operations on the stream will be sent back to the node the file originates from.

Caveats

One caveat to this approach however, is because access to the file is deferred, the identity of this resource will not be accurate. The timestamp will be the file timestamp at the time of first creating the resource, not necessarily the file timestamp at the time of operating on the content stream.

Sources

File sources are file references with a backup data source, so if the file no longer exists it will revert back to getting the data from the source and creating the file again. The data source is any compatible URI.

Resx.Producers.File.open("file:///hi.txt?source=ZGF0YTp0ZXh0L3BsYWluO2NoYXJzZXQ9VVMtQVNDSUk7YmFzZTY0LGFHaz0=")

If the source cannot be accessed anymore but the file exists, it will access the file. If both cannot be accessed then the request will fail.

Caching

File sources can act a form of cache, a couple of scenarios you might want to cache could be:

  • Storing a copy of a file locally that originates from another node in the network. So for as long as the file remains on the local node, any future requests won't need to be sent across the network.
  • Storing a file that is created from a chain of operations on a resource. So future requests won't need to reprocess the original pipeline but can simply access the file directly.

However there are some downsides to relying on it as a proper cache (or at least require additional work):

  • As the file reference will not be recreated unless it no longer exists, this can mean that the data source is does not represent the current state of the file. In order to invalidate the file it will need to be deleted.
  • Recreating a file from a data source is not atomic, this can mean that if there are multiple processes trying to operate on the file, some or all of them may go through the process of creating the file from the data source. If the data source contains a side-effect, or simply the referenced data changes, the new file could be different from what some of the other processes will receive.

Link to this section Summary

Functions

Discard a file resource

Get the MIME type list for the filename

Store a resource as a file

Creates a file resource with streamable contents

Link to this section Functions

Link to this function discard(reference, opts \\ [])
discard(Resx.ref(), meta: boolean(), content: boolean()) ::
  :ok | Resx.error(Resx.resource_error() | Resx.reference_error())

Discard a file resource.

The following options are all optional:

  • :meta - specify whether the meta file should also be deleted. By default it is.
  • :content - specify whether the content file should also be deleted. By default it is.

Get the MIME type list for the filename.

iex> Resx.Producers.File.mime("foo.txt")
["text/plain"]

iex> Resx.Producers.File.mime("foo.txt.png.jpg")
["image/jpeg", "image/png", "text/plain"]

iex> Resx.Producers.File.mime("foo")
["application/octet-stream"]

iex> Resx.Producers.File.mime("a/b/foo.txt")
["text/plain"]

iex> Resx.Producers.File.mime("a.png/b.exe/foo.txt")
["text/plain"]

iex> Resx.Producers.File.mime(".txt")
["application/octet-stream"]

iex> Resx.Producers.File.mime(".txt.png")
["image/png"]
Link to this function store(resource, options)
store(Resx.Resource.t(),
  path: String.t(),
  node: node(),
  modes: File.stream_mode(),
  bytes: pos_integer()
) ::
  {:ok, resource :: Resx.Resource.t(Resx.Resource.Content.Stream.t())}
  | Resx.error()

Store a resource as a file.

File stores are deferred, this means the returned resource will contain a content stream. When the content stream is processed the store operation will be performed.

It should also be noted that like file sources, file stores are non-atomic.

The required options are:

  • :path - expects a string denoting the path the file will be stored at. If it is not an absolute path, the path will be expanded by the calling node (this may result in the wrong path if it's storing on an external node).

The following options are all optional:

  • :node - expects a node name for where the resource should be stored.
  • :modes - expects a value of type File.stream_mode. If no modes are provided it defaults to the default modes File.stream!/3 opens with.
  • :bytes - expects a positive integer for the number bytes to read per request. If it is not provided, it defaults to reading line by line.

Link to this function stream(reference, opts \\ [])

Creates a file resource with streamable contents.

The options expose the additional arguments that can normally be passed to File.stream!/3.

  • :modes - expects a value of type File.stream_mode. If no modes are provided it defaults to the default modes File.stream!/3 opens with.
  • :bytes - expects a positive integer for the number bytes to read per request. If it is not provided, it defaults to reading line by line.