Reads and writes CSV, TSV, fixed-width, and other flat files through a declared schema.
defmodule Employee do
use Delimited.Schema
delimited_schema do
field :id, :integer, header: "Employee ID"
field :name, :string, required: true
field :hired_on, :date, header: "Hire Date"
field :active, :boolean, default: true
end
end
{:ok, employees} = Delimited.read(Employee, "employees.csv")
:ok = Delimited.write(Employee, "employees.tsv", employees, :tsv)The schema is the contract. With a header row, a column the file holds and the schema does not declare is ignored. A column the schema declares and the file does not hold is an error, because that is the shape a renamed column takes. Without a header row, every row must contain the declared number of cells because no name identifies an extra column.
Choosing a function
| You have | You want | Use |
|---|---|---|
| a path | every row, or the errors | read/3 |
| a path | every row, or an exception | read!/3 |
| a path, or a stream of slices | rows as they arrive | stream/3 |
| a binary in memory | every row, or the errors | decode/3 |
| rows | a file on disk | write/4 |
| rows | a stream of iodata | encode!/3 |
read/3 collects every row before returning. stream/3 holds one slice and
one row at a time, and is the one to reach for when a file is larger than the
memory you want to spend on it.
Errors
Every failure is a Delimited.Error carrying a :reason to match on and as
much of the path, line, column, and field as the failure knows.
A parse failure ends the file: after a misplaced quote, no later row can be trusted. A cast failure fails only its own row, so one unreadable date does not cost you the other rows.
case Delimited.read(Employee, "employees.csv") do
{:ok, employees} -> employees
{:error, errors} -> Enum.map(errors, &Exception.message/1)
endFixed-width files
A file with no delimiters is the same declaration with positions on it:
delimited_schema :fixed do
field :account, :string, at: 2..9
field :amount, :integer, at: 12..19, pad: ?0
endPositions are 1-based and inclusive, as a file specification writes them. See
Delimited.Field for :at, :align, and :pad, and Delimited.Dialect for
framing a file that has no line terminators at all.
Options
Every function accepts the runtime options in Delimited.Dialect, or a
format name such as :tsv, applied on top of what the schema declared. A call
cannot change the schema's layout because the layout determined field
positions and embedded shapes when the schema compiled.
Read Delimited.Dialect before writing a file that another program will open
as a spreadsheet. Its :escape_formulas note describes an injection that this
library does not defend against by default, and why.
Summary
Types
Dialect options, or the name of a format such as :csv or :tsv.
A row to write: the schema's struct, or a map with the same top-level keys.
A path to read, or an enumerable of binary slices.
Functions
Reads rows from a binary, or an enumerable of binary slices, already in memory.
Reads rows from a binary or an enumerable of binary slices, or raises the first error.
Returns the dialect that schema declared, before any call-site options.
Encodes rows as a stream of iodata, one element per line.
Returns the header row that a file for schema would have.
Reads every row of the file at path.
Reads every row of the file at path, or raises the first error.
Reads a file, or a stream of binary slices, one row at a time.
Writes rows to the file at path, replacing whatever is there.
Writes rows to the file at path, or raises.
Types
Dialect options, or the name of a format such as :csv or :tsv.
A row to write: the schema's struct, or a map with the same top-level keys.
@type source() :: Path.t() | Enumerable.t()
A path to read, or an enumerable of binary slices.
Functions
@spec decode(module(), binary() | Enumerable.t(), options()) :: {:ok, [row()]} | {:error, [Delimited.Error.t()]}
Reads rows from a binary, or an enumerable of binary slices, already in memory.
The counterpart of read/3 for data that never was a file: a response body, a
database column, a fixture written inline.
{:ok, [employee]} = Delimited.decode(Employee, "Employee ID,name\n1,Ada\n")@spec decode!(module(), binary() | Enumerable.t(), options()) :: [row()]
Reads rows from a binary or an enumerable of binary slices, or raises the first error.
@spec dialect(module()) :: Delimited.Dialect.t()
Returns the dialect that schema declared, before any call-site options.
@spec encode!(module(), Enumerable.t(row()), options()) :: Enumerable.t(iodata())
Encodes rows as a stream of iodata, one element per line.
Raises Delimited.Error for a row that cannot be written, because a value
that does not match its declared type is a fault in the program rather than in
the data. Use write/4 where you want that as a value.
The stream is lazy, so an export never exists in memory all at once:
Employee
|> Delimited.encode!(Repo.stream(query))
|> Enum.into(File.stream!("employees.csv"))For an export small enough to hold:
Employee
|> Delimited.encode!(employees)
|> Enum.to_list()
|> IO.iodata_to_binary()Returns the header row that a file for schema would have.
Use it to generate a blank template for whoever has to fill one in, so that the schema and the template cannot disagree.
Delimited.headers(Employee)
#=> ["Employee ID", "name", "Hire Date", "active"]@spec read(module(), Path.t(), options()) :: {:ok, [row()]} | {:error, [Delimited.Error.t()]}
Reads every row of the file at path.
Returns {:ok, rows}, or {:error, errors} holding every error found. One
unreadable cell does not discard the rows around it, so the error list is
where to look for what a supplier keeps getting wrong.
{:ok, employees} = Delimited.read(Employee, "employees.csv")
{:ok, employees} = Delimited.read(Employee, "employees.txt", delimiter: "|")Both the rows and the errors are held in memory. Use stream/3 for a file too
large for that.
Reads every row of the file at path, or raises the first error.
Use it where an unreadable file is a broken deployment rather than a case to handle: a fixture, a build step, a one-off script.
@spec stream(module(), source(), options()) :: Enumerable.t({:ok, row()} | {:error, Delimited.Error.t()})
Reads a file, or a stream of binary slices, one row at a time.
Emits {:ok, row} and {:error, error} in the order they occur. The stream
ends at the first parse error, because a file whose quoting or row boundaries
are wrong cannot be read further. It continues past a cast error.
Give a path to read a file, or any enumerable of binaries to read something that arrives in pieces, such as an upload or a decompressed response.
Employee
|> Delimited.stream("employees.csv")
|> Stream.each(&report_error/1)
|> Stream.filter(&match?({:ok, _row}, &1))
|> Enum.each(fn {:ok, employee} -> insert(employee) end)Reading a path opens the file when the stream is first enumerated, and raises
File.Error if it cannot be opened, in the manner of File.stream!/3. Use
read/3 where an unopenable file should be a value rather than an exception.
@spec write(module(), Path.t(), Enumerable.t(row()), options()) :: :ok | {:error, Delimited.Error.t()}
Writes rows to the file at path, replacing whatever is there.
A row may be the schema's struct or a map with the same top-level keys. Embedded keys hold the nested structs, maps, or lists declared by the schema.
:ok = Delimited.write(Employee, "employees.csv", employees)
:ok = Delimited.write(Employee, "employees.tsv", employees, :tsv)Returns {:error, error} for the first row that cannot be written, or if the
file cannot be opened, written, or closed. Before emitting a field, the writer
checks that the declared reader returns the same value. The rows before a
failed row are already on disk. Write to a temporary path and rename it if a
partial file would be worse than no file.
@spec write!(module(), Path.t(), Enumerable.t(row()), options()) :: :ok
Writes rows to the file at path, or raises.
This function has the same replacement and partial-write behaviour as
write/4. Write to a temporary path and rename it if a partial file would be
worse than no file.