Parquex.Schema (parquex v0.3.0)

The ordered fields in a Parquet batch stream.

Types use stable Elixir descriptors rather than exposing Arrow or Parquet implementation types. Nested list descriptors retain their element field so element nullability and names are not lost.

Summary

Types

float_type()

integer_type()

t()

t_type()

time_unit()

Functions

field(schema, name)

Returns a field by string name.

fields(schema)

Returns fields in file order, filtered by projection when configured.

new(fields)

Creates an ordered schema from concise Elixir field descriptors.

new!(fields)

Creates a schema and raises ArgumentError when a descriptor is invalid.

Types

float_type()

@type float_type() :: {:float, 32 | 64}

integer_type()

@type integer_type() :: {:integer, 8 | 16 | 32 | 64, boolean()}

t()

@type t() :: %Parquex.Schema{fields: [Parquex.Schema.Field.t()]}

t_type()

@type t_type() ::
  :boolean
  | :utf8
  | :binary
  | :date32
  | :date64
  | :null
  | integer_type()
  | float_type()
  | {:fixed_binary, pos_integer()}
  | {:time, :second | :millisecond, 32}
  | {:time, :microsecond | :nanosecond, 64}
  | {:timestamp, time_unit(), String.t() | nil}
  | {:duration, time_unit()}
  | {:decimal, 32 | 64 | 128 | 256, pos_integer(), integer()}
  | {:list, Parquex.Schema.Field.t()}
  | {:large_list, Parquex.Schema.Field.t()}
  | {:fixed_list, Parquex.Schema.Field.t(), pos_integer()}
  | {:struct, [Parquex.Schema.Field.t()]}

time_unit()

@type time_unit() :: :second | :millisecond | :microsecond | :nanosecond

Functions

field(schema, name)

@spec field(t(), String.t()) :: {:ok, Parquex.Schema.Field.t()} | :error

Returns a field by string name.

fields(schema)

@spec fields(t()) :: [Parquex.Schema.Field.t()]

Returns fields in file order, filtered by projection when configured.

new(fields)

@spec new(keyword() | [tuple() | map()]) :: {:ok, t()} | {:error, Parquex.Error.t()}

Creates an ordered schema from concise Elixir field descriptors.

A two-element descriptor defaults to a nullable field. Use a three-element tuple or field map to set nullability explicitly.

Examples

iex> {:ok, schema} =
...>   Parquex.Schema.new([
...>     {:id, :int64, false},
...>     {:name, :string, true},
...>     {:occurred_at, {:timestamp, :millisecond}, false}
...>   ])
iex> Enum.map(schema.fields, &{&1.name, &1.type, &1.nullable})
[
  {"id", {:integer, 64, true}, false},
  {"name", :utf8, true},
  {"occurred_at", {:timestamp, :millisecond, "UTC"}, false}
]

Supported aliases include :string, signed and unsigned integer widths, :float32, :float64, :date, and two-element timestamp descriptors. The stable low-level descriptors in t_type/0 remain accepted.

new!(fields)

@spec new!(keyword() | [tuple() | map()]) :: t()

Creates a schema and raises ArgumentError when a descriptor is invalid.