DataMorph v0.0.6 DataMorph

Create Elixir structs, maps with atom keys, and keyword lists from CSV/TSV data.

Examples

Define a struct and return stream of structs created from a tsv string, a namespace atom and name string.

iex> "name\tiso\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> DataMorph.structs_from_tsv(OpenRegister, "country") \
...> |> Enum.to_list
[
  %OpenRegister.Country{iso: "nz", name: "New Zealand"},
  %OpenRegister.Country{iso: "gb", name: "United Kingdom"}
]

Return stream of maps with atom keys created from a tsv stream.

iex> "name\tiso-code\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> String.split("\n") \
...> |> Stream.map(& &1) \
...> |> DataMorph.maps_from_tsv() \
...> |> Enum.to_list
[
  %{iso_code: "nz", name: "New Zealand"},
  %{iso_code: "gb", name: "United Kingdom"}
]

Return stream of keyword lists created from a tsv string.

iex> "name\tiso-code\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> DataMorph.keyword_lists_from_tsv() \
...> |> Enum.to_list
[
  [name: "New Zealand", "iso-code": "nz"],
  [name: "United Kingdom", "iso-code": "gb"]
]

Summary

Functions

Takes stream and applies filter regexp when not nil, and takes count when not nil

Returns stream of keyword_lists created from csv string or stream

Returns stream of keyword_lists created from tsv string or stream

Returns stream of maps with atom keys created from csv string or stream

Returns stream of maps with atom keys created from tsv string or stream

Encode stream of to TSV and write to standard out

Concat headers to stream, encode to TSV and write to standard out

Defines a struct and returns stream of structs created from csv string or stream, and a namespace and name

Defines a struct and returns stream of structs created from tsv string or stream, and a namespace and name

Functions

filter_and_take(stream, regex, count \\ nil)

Takes stream and applies filter regexp when not nil, and takes count when not nil.

Parmeters

  • stream: stream of string lines
  • regex: nil or regexp to match lines via Stream.filter/2 and String.match?/2
  • count: optional take count to apply via Stream.take/2
keyword_lists_from_csv(csv, options \\ [separator: ","])

Returns stream of keyword_lists created from csv string or stream.

Useful when you want to retain the field order of the original stream.

Parmeters

  • csv: CSV stream or string
  • options: optionally pass in separator, e.g. separator: “;”
keyword_lists_from_tsv(tsv)

Returns stream of keyword_lists created from tsv string or stream.

Useful when you want to retain the field order of the original stream.

Example

Return stream of keyword lists created from a tsv string.

iex> "name\tiso-code\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> DataMorph.keyword_lists_from_tsv() \
...> |> Enum.to_list
[
  [name: "New Zealand", "iso-code": "nz"],
  [name: "United Kingdom", "iso-code": "gb"]
]
maps_from_csv(csv, options \\ [separator: ","])

Returns stream of maps with atom keys created from csv string or stream.

Parmeters

  • csv: CSV stream or string
  • options: optionally pass in separator, e.g. separator: “;”
maps_from_tsv(tsv)

Returns stream of maps with atom keys created from tsv string or stream.

Example

Return stream of maps with atom keys created from a tsv stream.

iex> "name\tiso-code\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> String.split("\n") \
...> |> Stream.map(& &1) \
...> |> DataMorph.maps_from_tsv() \
...> |> Enum.to_list
[
  %{iso_code: "nz", name: "New Zealand"},
  %{iso_code: "gb", name: "United Kingdom"}
]

Parmeters

  • tsv: TSV stream or string
puts_tsv(stream)

Encode stream of to TSV and write to standard out.

Example

Write to standard out stream of string lists as TSV lines.

iex> "name\tiso\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> String.split("\n") \
...> |> DataMorph.structs_from_tsv("open-register", :iso_country) \
...> |> Stream.map(& [&1.iso, &1.name]) \
...> DataMorph.puts_tsv
nz\tNew Zealand
gb\tUnited Kingdom
puts_tsv(stream, headers)

Concat headers to stream, encode to TSV and write to standard out.

Example

Write to standard out stream of string lists as TSV lines with headers.

iex> "name\tiso\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> String.split("\n") \
...> |> DataMorph.structs_from_tsv("open-register", :iso_country) \
...> |> Stream.map(& [&1.iso, &1.name]) \
...> DataMorph.puts_tsv("iso-code","name")
iso-code\tname
nz\tNew Zealand
gb\tUnited Kingdom
structs_from_csv(csv, namespace, name, options \\ [separator: ","])

Defines a struct and returns stream of structs created from csv string or stream, and a namespace and name.

See structs_from_tsv/3 for examples.

Parmeters

  • csv: CSV stream or string
  • namespace: string or atom to form first part of struct alias
  • name: string or atom to form last part of struct alias
  • options: optionally pass in separator, e.g. separator: “;”
structs_from_tsv(tsv, namespace, name)

Defines a struct and returns stream of structs created from tsv string or stream, and a namespace and name.

Redefines struct when called again with same namespace and name but different fields. It sets struct fields to be the union of the old and new fields.

Example

Define a struct and return stream of structs created from a tsv stream, and a namespace string and name atom.

iex> "name\tiso\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> String.split("\n") \
...> |> Stream.map(& &1) \
...> |> DataMorph.structs_from_tsv("open-register", :iso_country) \
...> |> Enum.to_list
[
  %OpenRegister.IsoCountry{iso: "nz", name: "New Zealand"},
  %OpenRegister.IsoCountry{iso: "gb", name: "United Kingdom"}
]

Example

Add additional new fields to struct when called again with different tsv.

iex> "name\tiso\n" <>
...> "New Zealand\tnz\n" <>
...> "United Kingdom\tgb" \
...> |> DataMorph.structs_from_tsv(OpenRegister, "country") \
...> |> Enum.to_list
...>
...> "name\tacronym\n" <>
...> "New Zealand\tNZ\n" <>
...> "United Kingdom\tUK" \
...> |> DataMorph.structs_from_tsv(OpenRegister, "country") \
...> |> Enum.to_list
[
  %OpenRegister.Country{acronym: "NZ", iso: nil, name: "New Zealand"},
  %OpenRegister.Country{acronym: "UK", iso: nil, name: "United Kingdom"}
]

Parmeters

  • tsv: TSV stream or string
  • namespace: string or atom to form first part of struct alias
  • name: string or atom to form last part of struct alias