Delimited.Schema (Delimited v0.4.0)

Copy Markdown View Source

Declares the columns of a delimited file as a struct.

defmodule Employee do
  use Delimited.Schema

  delimited_schema do
    field :id, :integer, header: "Employee ID"
    field :name, :string, required: true
    field :department, {:enum, [engineering: "ENG", sales: "SLS"]}
    field :hired_on, :date, header: "Hire Date"
    field :salary, :decimal
    field :active, :boolean, default: true
  end
end

The module gains a struct with one key per field, and Delimited gains everything it needs to read and write that file. A schema holds no behaviour of its own: Delimited.read/3 and Delimited.write/4 take the module the way an Ecto.Repo takes an Ecto.Schema.

What the declaration decides

The order of field/3 calls is the order of columns when writing, and the order columns are matched in when reading with headers: false. When reading with headers, order is irrelevant and only the :header name matters.

A field's name is the struct key. Its :header is the text in the file, and defaults to the field name. The two are separate because a file's column names are the file's business: renaming a column in the file changes one :header, not every call site.

See Delimited.Field for the field options and Delimited.Type for the types.

Dialect

delimited_schema/2 takes a format name, dialect options, or both, and they become the schema's defaults:

delimited_schema :tsv, headers: false do
  field :sku, :string
end

A read or write can override runtime options, so the dialect here should describe the file the schema was written for. A call cannot change the layout because the layout determines field positions and embedded shapes when the schema compiles. See Delimited.Dialect.

Repeated groups of columns

A file often carries the same group of columns more than once. Declare the group once and embed it:

defmodule Address do
  use Delimited.Schema

  delimited_schema do
    field :street, :string
    field :city, :string
  end
end

defmodule Order do
  use Delimited.Schema

  delimited_schema do
    field :id, :integer
    embeds_one :billing, Address, prefix: "billing_"
    embeds_one :shipping, Address, prefix: "shipping_"
    embeds_many :lines, LineItem, count: 2, prefix: "item_{n}_"
  end
end

That reads billing_street, shipping_street, item_1_sku, item_2_sku and the rest into %Order{billing: %Address{}, lines: [%LineItem{}, ...]}. Declaring the group once is the point: two copies of a column list cannot drift apart, and shipping_postcode cannot end up reading the billing one.

An embed is resolved when the schema compiles, so what the reader works with is still a flat list of columns. See embeds_one/3 and embeds_many/3.

A group whose every column is empty reads as nil, and nil writes its columns back empty, which is the same rule the fixed layout uses for a blank field. One column filled makes the group present.

Introspection

  • __delimited__(:fields) returns the Delimited.Field structs in the order the file holds them, with any embedded schema's fields expanded in place.
  • __delimited__(:dialect) returns the declared Delimited.Dialect.

Both are public, because generating a blank template, a column list for an upload form, or documentation from the schema is the point of declaring it once. __delimited__(:shape) also exists, and holds the nesting that the flat list cannot express; it is internal and may change.

Summary

Functions

Declares the columns, and optionally the dialect, of a delimited file.

Embeds another schema's columns a declared number of times.

Embeds another schema's columns in this one.

Functions

delimited_schema(format \\ :csv, opts \\ [], list)

(macro)

Declares the columns, and optionally the dialect, of a delimited file.

Defines a struct with one key per field, defaulting to the field's :default.

embeds_many(name, schema, opts)

(macro)

Embeds another schema's columns a declared number of times.

embeds_many :lines, LineItem, count: 3, prefix: "item_{n}_"

Reads item_1_sku, item_2_sku, and item_3_sku into %Order{lines: [%LineItem{}, %LineItem{}, %LineItem{}]}. {n} becomes each copy's number, and is required, because otherwise every copy would claim the same columns.

A row holds a fixed number of columns, so a repeated group has to say how many times it repeats. Under the fixed layout the copies follow one another by the embedded schema's own width, or by a declared :stride where the file leaves a gap between them. When writing, a shorter list leaves the remaining groups blank. A longer list is an error because the row has no columns for it.

embeds_one(name, schema, opts \\ [])

(macro)

Embeds another schema's columns in this one.

embeds_one :billing, Address, prefix: "billing_"

Reads billing_street and billing_city into %Order{billing: %Address{}}. Under the fixed layout there are no headers to prefix, so the embed says where its bytes start instead, and the embedded schema's own positions are counted from there:

embeds_one :payer, Party, at: 2

An embed whose every column is empty reads as nil, which is what an absent group means. required: true makes that an error instead.

field(name, type \\ :string, opts \\ [])

(macro)

Declares one column.

The type defaults to :string, which is the type of every cell before anything is decided about it. See Delimited.Field for the options and Delimited.Type for the types.