ExZarr.FormatConverter (ExZarr v1.1.0)

View Source

Convert Zarr arrays between v2 and v3 formats.

Provides utilities for converting entire Zarr arrays from one format version to another, including metadata conversion and chunk copying with proper key encoding.

Features

  • Convert v2 arrays to v3 format
  • Convert v3 arrays to v2 format (with limitations)
  • Preserve data during conversion
  • Handle different chunk key encodings
  • Clear error messages for incompatible features

Limitations

v3 → v2 Conversion

The following v3 features cannot be converted to v2:

  • Sharding codec
  • Dimension names (lost in conversion)
  • Irregular chunk grids
  • Array→Array codecs (transpose, quantize, bitround)
  • Custom chunk key encodings

v2 → v3 Conversion

v2 arrays can be fully converted to v3, with the following transformations:

  • Filters and compressor become unified codec pipeline
  • NumPy dtypes become v3 data types
  • Regular chunking becomes regular chunk grid
  • Attributes are preserved

Examples

# Convert v2 array to v3
ExZarr.FormatConverter.convert(
  source_path: "/data/array_v2.zarr",
  target_path: "/data/array_v3.zarr",
  target_version: 3
)

# Convert v3 array to v2 (if compatible)
ExZarr.FormatConverter.convert(
  source_path: "/data/array_v3.zarr",
  target_path: "/data/array_v2.zarr",
  target_version: 2
)

# Convert with explicit storage options
ExZarr.FormatConverter.convert(
  source_path: "/data/array_v2.zarr",
  source_storage: :filesystem,
  target_path: "/data/array_v3.zarr",
  target_storage: :filesystem,
  target_version: 3
)

Summary

Types

convert_opts()

Functions

check_v2_compatibility(metadata)

Check if a v3 array can be converted to v2 without data loss.

convert(opts)

Convert a Zarr array from one format version to another.

Types

convert_opts()

@type convert_opts() :: [
  source_path: String.t(),
  source_storage: atom(),
  target_path: String.t(),
  target_storage: atom(),
  target_version: 2 | 3
]

Functions

check_v2_compatibility(metadata)

@spec check_v2_compatibility(ExZarr.MetadataV3.t()) ::
  :ok | {:error, {atom(), String.t()}}

Check if a v3 array can be converted to v2 without data loss.

Returns :ok if conversion is possible, or {:error, reason} with details about incompatible features.

Parameters

  • metadata - MetadataV3 struct to check

Returns

  • :ok - Can be converted to v2
  • {:error, {reason, details}} - Cannot be converted

Examples

iex> metadata = %ExZarr.MetadataV3{
...>   zarr_format: 3,
...>   node_type: :array,
...>   shape: {100},
...>   data_type: "int32",
...>   chunk_grid: %{name: "regular", configuration: %{chunk_shape: {10}}},
...>   codecs: [%{name: "bytes"}]
...> }
iex> ExZarr.FormatConverter.check_v2_compatibility(metadata)
:ok

convert(opts)

@spec convert(convert_opts()) :: :ok | {:error, term()}

Convert a Zarr array from one format version to another.

Reads the source array, converts metadata to the target format, and copies all chunks with proper key encoding for the target format.

Parameters

  • opts - Keyword list with:
    • :source_path - Path to source array (required)
    • :source_storage - Source storage backend (default: :filesystem)
    • :target_path - Path to target array (required)
    • :target_storage - Target storage backend (default: :filesystem)
    • :target_version - Target Zarr version, 2 or 3 (required)

Returns

  • :ok - Conversion successful
  • {:error, reason} - Conversion failed

Examples

# Basic conversion from v2 to v3
:ok = ExZarr.FormatConverter.convert(
  source_path: "/data/my_array.zarr",
  target_path: "/data/my_array_v3.zarr",
  target_version: 3
)

# Conversion with custom storage
:ok = ExZarr.FormatConverter.convert(
  source_path: "/data/my_array.zarr",
  source_storage: :filesystem,
  target_path: "/data/my_array_v3.zarr",
  target_storage: :memory,
  target_version: 3
)