ExDir v0.1.2 ExDir View Source

ExDir is an iterative directory listing for Elixir.

Elixir function File.ls/1 return files from directories after reading them from the filesystem. When you have an humongous number of files on a single folder, File.ls/1 will block for a certain time.

In these cases you may not be interested in returning the full list of files, but instead you may want to list them iteratively, returning each entry after the another to your process, at the moment they are taken from readdir.

The Enumerable protocol has been implemented for ExDir, so all usual functions such as Enum.map, Enum.reduce, etc are available.

For example, you can return all files in a given directory in a list with:

{:ok, dir} = ExDir.opendir(".")
Enum.map(dir, &(&1))

Or count the number of files in a directory:

{:ok, dir} = ExDir.opendir(".")
Enum.count(dir)

The above examples aren't practical when you have tons of files in the directory, which makes the above functions very similar to File.ls/1.

If your intention is to consume files from a very large folder, then you might be interested in reading the file names and distribute them to worker processes to do some job. In this case the following example is the most suitable:

{:ok, dir} = ExDir.opendir(".")
Enum.each(fn file_path ->
  push_to_worker(file_path)
end)

So you can start consuming files straight away, without having to wait for File.ls/1 to complete as you would normally do.

Notice that ExDir is a system resource and thus it is mutable. It means that after reading all files from the directory, the only way to read it a second time is by opening the directory again.

The order of the files is highly dependent of the filesystem, and no ordering is guaranteeded. This is intentional as large directories is the main purpose of this library. If reading tons of files in a specific order is important for your application, you should think twice: or you read all files from system and order them by yourself, which will be very time consuming for very long directories, or better accept listing them unordered.

Link to this section Summary

Functions

Opens the given path.

Reads the opened directory.

Set controlling affinity.

Returns a ExDir.Stream for the given path with the given options.

Link to this section Types

Link to this type

file_type()

View Source
file_type() :: :device | :directory | :symlink | :regular | :other | :undefined
Link to this type

filename()

View Source
filename() :: Path.t()
Link to this type

option()

View Source
option() :: {:read, :type | :raw}
Link to this type

options()

View Source
options() :: [option()]
Link to this type

posix_error()

View Source
posix_error() ::
  :enoent | :eacces | :emfile | :enfile | :enomem | :enotdir | atom()

Link to this section Functions

Link to this function

open(path \\ ".")

View Source
open(dirname()) :: {:ok, t()} | {:error, posix_error()}

Opens the given path.

Possible error reasons:

  • :enoent: Directory does not exist, or path is an empty string.
  • :eacces: Permission denied.
  • :emfile: The per-process limit on the number of open file descriptors has been reached.
  • :enfile: The system-wide limit on the total number of open files has been reached.
  • :enomem: Insufficient memory to complete the operation.
  • :enotdir: path is not a directory.

Example

ExDir.open(".")
{:ok, #Reference<0.3456274719.489029636.202763>}

Reads the opened directory.

The only available option is :read. You can choose one of the following:

  • :type - if the filesystem supports it, returns the file type along the file name while reading. It will skip filenames with invalid Unicode characters.
  • :raw - returns the file type, and doesn't skip filenames containing invalid Unicode characters (use with care).

If not specified, the readdir will not return file types and will skip invalid filenames.

This function returns each entry in the directory iteratively. Filenames contain the full path, including the start path passed to opendir/1.

This function breaks the general immutability of the language in the sense that ExDir is actually a system resource identifier, and thus it is mutable internally. It means that calling this function twice for the same dir will result in different results.

Link to this function

set_controlling_process(dir, owner)

View Source
set_controlling_process(t(), pid()) :: :ok

Set controlling affinity.

Once created, ExDir resources are associated to the calling process and readdir/1 should be executed by the same process. If passing to another process is required, then this function should be called from the process owner, delegating control to another process indicated by pid.

Link to this function

stream!(path, recursive \\ false, options \\ [])

View Source

Returns a ExDir.Stream for the given path with the given options.

The stream implements only the Enumerable protocol, which means it can be used for read only.

The options argument configures how the filenames are returned when streaming. It can be:

  • :raw - all filenames will be returned, even invalid Unicode filenames. Case this option is false (default) and the filename can't be translated to Unicode, then an exception ExDir.Error will be raised.
  • type - filenames will be returned along with their types in tuples {file_type, file_path}, otherwise just file_path.

If the recursive argument is true, all subdirectories will be recursed, except directory entries themselves.