ExArrow.Parquet.Writer (ex_arrow v0.8.0)

Parquet file writer: serialise Arrow record batches to a .parquet file or to an in-memory binary.

Accepts an ExArrow.Schema handle and a list of ExArrow.RecordBatch handles produced by any ExArrow source (IPC reader, ADBC execute, Flight do_get, or compute kernels).

Write options (v0.8+)

:compression — :none (alias :uncompressed) | :snappy | :zstd | {:zstd, level} (level 1..22) | :lz4 | :gzip
:row_group_size — positive integer (max rows per row group)
:dictionary — boolean

Examples

:ok =
  ExArrow.Parquet.Writer.to_file("/out/result.parquet", schema, batches,
    compression: :zstd,
    row_group_size: 64_000
  )

{:ok, bytes} =
  ExArrow.Parquet.Writer.to_binary(schema, batches, compression: :snappy)

Summary

Functions

to_binary(schema, batches, opts \\ [])

Serialise schema and batches to a Parquet binary in memory.

to_file(path, schema, batches, opts \\ [])

Write schema and batches to a Parquet file at path.

Functions

to_binary(schema, batches, opts \\ [])

@spec to_binary(ExArrow.Schema.t(), [ExArrow.RecordBatch.t()], keyword()) ::
  {:ok, binary()} | {:error, String.t()}

Serialise schema and batches to a Parquet binary in memory.

Returns {:ok, binary} or {:error, message}.

to_file(path, schema, batches, opts \\ [])

@spec to_file(Path.t(), ExArrow.Schema.t(), [ExArrow.RecordBatch.t()], keyword()) ::
  :ok | {:error, String.t()}

Write schema and batches to a Parquet file at path.

Creates or overwrites the file. Returns :ok or {:error, message}.