Parquex.
View Source
A validated Parquet dataset beneath one store prefix.
A dataset combines a reusable Parquex.Store, an object-key prefix, an
explicit schema and a UTC event-time partition specification.
Dataset writers route each row by its event timestamp into canonical UTC
paths and write uniquely named Parquet parts. Dataset streams lazily discover
only overlapping partitions and apply exact half-open time filtering.
Examples
iex> {:ok, store} = Parquex.Store.open(:local, root: System.tmp_dir!())
iex> {:ok, schema} = Parquex.Schema.new(timestamp: :int64, name: :string)
iex> {:ok, dataset} =
...> Parquex.Dataset.new(store, "event_log",
...> schema: schema,
...> partition_by: {:time, :timestamp, :hour},
...> timestamp_unit: :millisecond,
...> compression: :zstd
...> )
iex> Parquex.Dataset.prefix(dataset)
"event_log/"Summary
Functions
Returns the default Parquet compression.
Creates a validated time-partitioned Parquet dataset descriptor.
Creates a dataset and raises ArgumentError when its contract is invalid.
Opens an owner-bound writer with a bounded active-partition registry.
Returns the UTC time partition specification.
Returns the normalized dataset prefix.
Materializes all selected rows in one finite dataset time range.
Returns the explicit Parquet schema.
Returns the reusable store.
Plans and lazily streams one exact half-open UTC time range.
Writes finite rows or a finite/continuous enumerable into partition parts.
Types
Functions
@spec compression(t()) :: :zstd | :snappy | :uncompressed
Returns the default Parquet compression.
@spec new(Parquex.Store.t(), String.t(), keyword()) :: {:ok, t()} | {:error, Parquex.Error.t()}
Creates a validated time-partitioned Parquet dataset descriptor.
@spec new!(Parquex.Store.t(), String.t(), keyword()) :: t()
Creates a dataset and raises ArgumentError when its contract is invalid.
@spec open_writer( t(), keyword() ) :: {:ok, Parquex.Dataset.Writer.t()} | {:error, Parquex.Error.t()}
Opens an owner-bound writer with a bounded active-partition registry.
@spec partition(t()) :: Parquex.TimePartition.t()
Returns the UTC time partition specification.
Returns the normalized dataset prefix.
@spec read( t(), keyword() ) :: {:ok, [map()]} | {:error, Parquex.Error.t()}
Materializes all selected rows in one finite dataset time range.
@spec schema(t()) :: Parquex.Schema.t()
Returns the explicit Parquet schema.
@spec store(t()) :: Parquex.Store.t()
Returns the reusable store.
@spec stream( t(), keyword() ) :: {:ok, Parquex.Dataset.Stream.t()} | {:error, Parquex.Error.t()}
Plans and lazily streams one exact half-open UTC time range.
@spec write(t(), term(), keyword()) :: {:ok, Parquex.Dataset.WriteReport.t()} | {:error, Parquex.Error.t()}
Writes finite rows or a finite/continuous enumerable into partition parts.