Delimited.Dialect (Delimited v0.1.0)

Copy Markdown View Source

How a file is punctuated, separately from what its columns mean.

A schema carries the dialect it was declared with. Every read and write accepts the same options and applies them on top, so one schema can read the comma-separated export and the tab-separated feed without being declared twice.

delimited_schema :tsv do
  field :sku, :string
end

Delimited.read(Product, "supplier.csv", delimiter: ",")

Formats

:csv uses a comma. :tsv uses a tab and is otherwise identical, which means quoting still applies. That is deliberate: the tab-separated files that tools actually produce quote fields containing tabs, while the IANA text/tab-separated-values registration forbids such fields outright. Reading a file that follows the registration works either way; writing produces a file the registration does not describe only when a value contains a tab, a quote, or a line break.

Options

Reading and writing:

  • :delimiter - the byte between cells, as a one-character string or a codepoint. Defaults to ?,.
  • :quote_char - the byte that quotes a cell. Defaults to ?".
  • :headers - whether the file has a header row. When false, columns are matched by declaration order. Defaults to true.
  • :trim - strip surrounding whitespace from every cell. Defaults to false, so that a value is read exactly as the file holds it. Overridable per field.
  • :null - the strings that mean "no value". Defaults to [""]. When writing, nil becomes the first string in the list. Overridable per field.

Reading only:

  • :skip_rows - discard this many rows before the header row. Use it for the export that starts with a title and a blank line. Defaults to 0.
  • :skip_blank_lines - ignore lines holding no cells at all. Defaults to true. A line of only whitespace is not blank; it is a one-cell row.
  • :on_missing_header - :error (default) or :ignore. Ignoring leaves the field at its default for every row, which is why it is not the default: a renamed column would otherwise read as an entire column of nil.
  • :on_extra_header - :ignore (default) or :error. A schema names the columns it wants, so extra columns are normally the file's business.
  • :chunk_size - bytes read from the file at a time. Defaults to 65_536.

Writing only:

  • :newline - "\n" (default) or "\r\n". RFC 4180 specifies CRLF; most tools accept either.
  • :quoting - :as_needed (default) quotes a cell only when it holds a delimiter, a quote, or a line break. :always quotes every cell.
  • :bom - write a UTF-8 byte order mark. Defaults to false. Excel needs it to read UTF-8 correctly. Reading strips a byte order mark either way.
  • :escape_formulas - prefix a cell that a spreadsheet would evaluate with an apostrophe. Defaults to false. See the security note below.

Formula escaping

A cell beginning with =, +, -, @, a tab, or a carriage return is executed as a formula by Excel, LibreOffice, and Google Sheets when the file is opened. A file assembled from untrusted input can therefore run a command on the reader's machine. escape_formulas: true prefixes such a cell with an apostrophe, which those programs strip on display.

It is off by default because it changes the data: a file written with it and read back yields '=SUM(A1), not =SUM(A1). Correctness of the round trip wins over a defence against a hazard in a different program, and the choice is documented here rather than made silently.

Values that read as numbers are never prefixed, so -1.5 survives. The defence covers the leading character only. It does not sanitise a cell a spreadsheet interprets in some other way, does not protect a consumer that splits on delimiters differently, and is no substitute for the consumer opening the file as data rather than as a spreadsheet.

Summary

Functions

Applies call-site options, or a format name, to an existing dialect.

Builds a dialect from a format name, a keyword list of options, or both.

Types

t()

@type t() :: %Delimited.Dialect{
  bom: boolean(),
  chunk_size: pos_integer(),
  delimiter: byte(),
  escape_formulas: boolean(),
  headers: boolean(),
  newline: String.t(),
  null: [String.t()],
  on_extra_header: :error | :ignore,
  on_missing_header: :error | :ignore,
  quote_char: byte(),
  quoting: :as_needed | :always,
  skip_blank_lines: boolean(),
  skip_rows: non_neg_integer(),
  trim: boolean()
}

Functions

merge!(dialect, format)

@spec merge!(t(), keyword() | atom()) :: t()

Applies call-site options, or a format name, to an existing dialect.

Raises ArgumentError on an unknown format, or an unknown or invalid option.

new!(format_or_opts \\ [])

@spec new!(atom() | keyword()) :: t()

Builds a dialect from a format name, a keyword list of options, or both.

Raises ArgumentError for an unknown format, an unknown option, or an option the parser cannot honour. A dialect is programmer-owned configuration rather than data read from a file, so a mistake in one is a mistake in the program.

Delimited.Dialect.new!(:tsv)
Delimited.Dialect.new!(delimiter: ";", newline: "\r\n")
Delimited.Dialect.new!(:tsv, headers: false)

new!(format, opts)

@spec new!(
  atom() | keyword(),
  keyword()
) :: t()