Delimited.Dialect (Delimited v0.4.0)

Copy Markdown View Source

How a file is punctuated, separately from what its columns mean.

A schema carries the dialect it was declared with. Every read and write accepts the runtime options and applies them on top, so one schema can read the comma-separated export and the tab-separated feed without being declared twice. A call cannot change the layout because the layout determined field positions and embedded shapes when the schema compiled.

delimited_schema :tsv do
  field :sku, :string
end

Delimited.read(Product, "supplier.csv", delimiter: ",")

Formats

:csv uses a comma. :tsv uses a tab and is otherwise identical, which means quoting still applies. That is deliberate: the tab-separated files that tools actually produce quote fields containing tabs, while the IANA text/tab-separated-values registration forbids such fields outright. Reading a file that follows the registration works either way; writing produces a file the registration does not describe only when a value contains a tab, a quote, or a line break.

:psv uses a pipe and :ssv a single space. A space-separated file is the one to be most careful with: a single space is a delimiter like any other, so two spaces make an empty cell between them rather than one wider gap. A file that aligns its columns with runs of spaces is a fixed-width file, and layout: :fixed reads it.

:fixed selects the fixed-width layout and turns :headers off, since a fixed-width file rarely carries a header row.

A format applies exactly the options it names and leaves every other option as it found it, so merge!(dialect, :tsv) changes the delimiter without disturbing the null strings.

Layouts

layout: :delimited (the default) finds a cell between two delimiters. layout: :fixed takes a cell from the byte range each field declares with :at, and the delimiter and quote character are then unused. See Delimited.Field for declaring positions.

The layout also decides how records are framed:

  • record_length: :line (the default) frames a record as a line, however long the line is.
  • record_length: N frames a record as exactly N bytes, for a file with no line terminators at all. A file that has terminators is framed by its lines whatever its record length, so there is never a question of whether a terminator is part of the record or between two of them.

Options

Reading and writing:

  • :layout - :delimited (default) or :fixed. The schema declaration selects it; a read or write cannot change it after compilation.
  • :record_length - :line (default) or a positive integer. Fixed layout only.
  • :delimiter - the byte between cells, as a one-character string or a codepoint. Defaults to ?,. Delimited layout only.
  • :quote_char - the byte that quotes a cell. Defaults to ?". Delimited layout only.
  • :headers - whether the file has a header row. Under the delimited layout, false matches columns by declaration order. Under the fixed layout, positions always decide, so true only means that the first record is a header line to skip when reading and to write when writing. Defaults to true, and to false under the :fixed format.
  • :trim - strip surrounding whitespace from every cell. Defaults to false, so that a value is read exactly as the file holds it. Overridable per field.
  • :null - the strings that mean "no value". Defaults to [""]. When writing, nil becomes the first string in the list. Overridable per field.

Reading only:

  • :comment - a byte that marks a whole line as a comment, as a one-character string or a codepoint. Defaults to nil, meaning no line is a comment. A commented line is discarded while the file is being framed, before any cell is read, so it may hold anything at all including an unclosed quote.
  • :skip_rows - discard this many rows before the header row. Use it for the export that starts with a title and a blank line. Defaults to 0.
  • :skip_blank_lines - ignore lines holding no cells at all. Defaults to true. A line of only whitespace is not blank; it is a one-cell row.
  • :on_missing_header - :error (default) or :ignore. Ignoring leaves the field at its default for every row, which is why it is not the default: a renamed column would otherwise read as an entire column of nil.
  • :on_extra_header - :ignore (default) or :error. A schema names the columns it wants, so extra columns are normally the file's business.
  • :chunk_size - bytes read from the file at a time. Defaults to 65_536.

Writing only:

  • :newline - "\n" (default) or "\r\n". RFC 4180 specifies CRLF; most tools accept either.
  • :quoting - :as_needed (default) quotes a cell only when it holds a delimiter, a quote, or a line break. :always quotes every cell.
  • :bom - write a UTF-8 byte order mark. Defaults to false. Use it for a consumer that requires the mark to identify UTF-8. Reading strips a byte order mark either way.
  • :escape_formulas - prefix a cell that a spreadsheet might interpret as a formula with an apostrophe. Defaults to false. See the security note below.

Formula escaping

Some spreadsheet import paths interpret a cell beginning with =, +, -, @, a tab, or a carriage return as a formula. A malicious formula can disclose data or invoke an external action when the spreadsheet and its security settings permit one. escape_formulas: true prefixes such a cell with an apostrophe.

It is off by default because it changes the data: a file written with it and read back yields '=SUM(A1), not =SUM(A1). Correctness of the round trip wins over a defence against a hazard in a different program, and the choice is documented here rather than made silently.

Values that have numeric syntax are not prefixed, so -1.5 survives. This option is not a universal spreadsheet defence. It checks only the first ASCII byte after Delimited has chosen the cell boundary. It does not cover a leading line feed or full-width variants of formula characters. It also does not protect against a consumer that finds different cell boundaries or a program that removes the apostrophe when it saves and reopens the file. OWASP's CSV Injection guidance records the same portability limit. Importing the file as text remains the reliable control.

Summary

Functions

Applies call-site options, or a format name, to an existing dialect.

Builds a dialect from a format name, a keyword list of options, or both.

Types

t()

@type t() :: %Delimited.Dialect{
  bom: boolean(),
  chunk_size: pos_integer(),
  comment: byte() | nil,
  delimiter: byte(),
  escape_formulas: boolean(),
  headers: boolean(),
  layout: :delimited | :fixed,
  newline: String.t(),
  null: [String.t()],
  on_extra_header: :error | :ignore,
  on_missing_header: :error | :ignore,
  quote_char: byte(),
  quoting: :as_needed | :always,
  record_length: :line | pos_integer(),
  skip_blank_lines: boolean(),
  skip_rows: non_neg_integer(),
  trim: boolean()
}

Functions

merge!(dialect, format)

@spec merge!(t(), keyword() | atom()) :: t()

Applies call-site options, or a format name, to an existing dialect.

A format name applies that format's options over the dialect and leaves the rest of it alone, so merge!(dialect, :tsv) changes the delimiter without disturbing the null strings.

Raises ArgumentError on an unknown format, or an unknown or invalid option.

new!(format_or_opts \\ [])

@spec new!(atom() | keyword()) :: t()

Builds a dialect from a format name, a keyword list of options, or both.

Raises ArgumentError for an unknown format, an unknown option, or an option the dialect cannot honour. A dialect is programmer-owned configuration rather than data read from a file, so a mistake in one is a mistake in the program.

Delimited.Dialect.new!(:tsv)
Delimited.Dialect.new!(delimiter: ";", newline: "\r\n")
Delimited.Dialect.new!(:fixed, record_length: 100)

new!(format, opts)

@spec new!(
  atom() | keyword(),
  keyword()
) :: t()