ExZarr.GenStage (ExZarr v1.1.0)

View Source

GenStage integration for demand-driven Zarr chunk and slice processing.

GenStage producers emit chunks or slices only when downstream consumers request them, providing explicit backpressure for large array pipelines. Producers stop with :normal after the array is fully consumed.

GenStage is an optional dependency:

{:gen_stage, "~> 1.2"}

Modules

Examples

{:ok, producer} = ExZarr.GenStage.start_chunk_producer(array, metadata: true)

{:ok, consumer} = MyApp.ChunkConsumer.start_link(producer: producer)
GenStage.ask(producer, 10)

Backpressure

Downstream consumers control throughput by adjusting demand. When a consumer is overloaded, it stops asking for events and chunk reads pause automatically.

Summary

Functions

start_chunk_producer(array, opts \\ [])

Starts a supervised chunk producer for the given array.

start_slice_producer(array, along, opts \\ [])

Starts a supervised slice producer for the given array and dimension.

Functions

start_chunk_producer(array, opts \\ [])

@spec start_chunk_producer(
  ExZarr.Array.t(),
  keyword()
) :: {:ok, pid()} | {:error, term()}

Starts a supervised chunk producer for the given array.

Stream options may be passed flat (e.g. metadata: true) or nested under :stream_opts.

start_slice_producer(array, along, opts \\ [])

@spec start_slice_producer(ExZarr.Array.t(), non_neg_integer(), keyword()) ::
  {:ok, pid()} | {:error, term()}

Starts a supervised slice producer for the given array and dimension.

Stream options may be passed flat or nested under :stream_opts.