ExZarr.DataType (ExZarr v1.1.0)

View Source

Data type conversion between Zarr v2 and v3 formats.

Zarr v2 uses NumPy-style dtype strings (e.g., "<i4", "<f8", "|b1") while Zarr v3 uses simplified type names (e.g., "int32", "float64", "bool").

This module provides bidirectional conversion between:

  • Internal atoms (:int32, :float64, etc.)
  • v2 NumPy dtype strings ("<i4", "<f8", etc.)
  • v3 simplified type names ("int32", "float64", etc.)

Data Type Mapping

Internalv2 NumPyv3 Name
:int8"<i1""int8"
:int16"<i2""int16"
:int32"<i4""int32"
:int64"<i8""int64"
:uint8"<u1""uint8"
:uint16"<u2""uint16"
:uint32"<u4""uint32"
:uint64"<u8""uint64"
:float32"<f4""float32"
:float64"<f8""float64"

| :bool | "|b1" | "bool" | | :complex64 | "<c8" | "complex64" | | :complex128 | "<c16" | "complex128" | | :datetime64 | "<M8" | "datetime64" | | :timedelta64 | "<m8" | "timedelta64" |

Byte Order

v2 uses byte order prefixes:

  • < : little-endian
  • > : big-endian
  • | : not applicable (single byte or unicode)

v3 does not encode byte order in the type name (little-endian is default).

Examples

# Convert internal atom to v3 string
iex> ExZarr.DataType.to_v3(:int32)
"int32"

# Convert v3 string to internal atom
iex> ExZarr.DataType.from_v3("float64")
:float64

# Convert internal atom to v2 NumPy string
iex> ExZarr.DataType.to_v2(:int32)
"<i4"

# Convert v2 NumPy string to internal atom
iex> ExZarr.DataType.from_v2("<f8")
:float64

Summary

Functions

Returns whether a dtype represents a boolean type.

Returns whether a dtype represents a complex number type.

Returns whether a dtype represents a datetime type.

Convert an Elixir DateTime to microseconds since epoch.

Returns whether a dtype represents a floating point type.

Converts a v2 NumPy dtype string to an internal dtype atom.

Converts a v3 data type string to an internal dtype atom.

Returns the size in bytes for a given dtype.

Convert a datetime value (microseconds since epoch) to an Elixir DateTime.

Pack a value into binary format for a given dtype.

Parse an ISO 8601 datetime string into microseconds since epoch.

Returns whether a dtype represents a signed integer type.

Returns all supported internal dtype atoms.

Returns whether a dtype represents a timedelta type.

Converts an internal dtype atom to a v2 NumPy dtype string.

Converts an internal dtype atom to a v3 data type string.

Unpack a binary value into Elixir terms for a given dtype.

Returns whether a dtype represents an unsigned integer type.

Validates that a dtype is supported.

Types

dtype_atom()

@type dtype_atom() ::
  :int8
  | :int16
  | :int32
  | :int64
  | :uint8
  | :uint16
  | :uint32
  | :uint64
  | :float32
  | :float64
  | :bool
  | :complex64
  | :complex128
  | :datetime64
  | :timedelta64

v2_dtype_string()

@type v2_dtype_string() :: String.t()

v3_data_type()

@type v3_data_type() :: String.t()

Functions

bool?(arg1)

@spec bool?(dtype_atom()) :: boolean()

Returns whether a dtype represents a boolean type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if boolean, false otherwise

Examples

iex> ExZarr.DataType.bool?(:bool)
true

iex> ExZarr.DataType.bool?(:int32)
false

complex?(dtype)

@spec complex?(dtype_atom()) :: boolean()

Returns whether a dtype represents a complex number type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if complex, false otherwise

Examples

iex> ExZarr.DataType.complex?(:complex64)
true

iex> ExZarr.DataType.complex?(:complex128)
true

iex> ExZarr.DataType.complex?(:float32)
false

datetime?(arg1)

@spec datetime?(dtype_atom()) :: boolean()

Returns whether a dtype represents a datetime type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if datetime, false otherwise

Examples

iex> ExZarr.DataType.datetime?(:datetime64)
true

iex> ExZarr.DataType.datetime?(:int64)
false

datetime_to_micros(dt)

@spec datetime_to_micros(DateTime.t()) :: integer()

Convert an Elixir DateTime to microseconds since epoch.

Parameters

  • datetime - DateTime struct

Returns

  • Microseconds since Unix epoch

Examples

iex> ExZarr.DataType.datetime_to_micros(~U[2021-01-01 00:00:00.000000Z])
1609459200000000

float?(dtype)

@spec float?(dtype_atom()) :: boolean()

Returns whether a dtype represents a floating point type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if floating point, false otherwise

Examples

iex> ExZarr.DataType.float?(:float64)
true

iex> ExZarr.DataType.float?(:float32)
true

iex> ExZarr.DataType.float?(:int32)
false

from_v2(dtype_string)

@spec from_v2(v2_dtype_string()) :: dtype_atom()

Converts a v2 NumPy dtype string to an internal dtype atom.

Handles multiple byte order prefixes (<, >, |).

Parameters

  • dtype_string - v2 NumPy-style dtype string

Returns

  • Internal dtype atom

Examples

iex> ExZarr.DataType.from_v2("<i4")
:int32

iex> ExZarr.DataType.from_v2("<f8")
:float64

iex> ExZarr.DataType.from_v2("|u1")
:uint8

iex> ExZarr.DataType.from_v2(">i4")
:int32

from_v3(data_type_string)

@spec from_v3(v3_data_type()) :: dtype_atom()

Converts a v3 data type string to an internal dtype atom.

Parameters

  • data_type_string - v3 data type name

Returns

  • Internal dtype atom

Examples

iex> ExZarr.DataType.from_v3("int32")
:int32

iex> ExZarr.DataType.from_v3("float64")
:float64

iex> ExZarr.DataType.from_v3("uint8")
:uint8

itemsize(dtype)

@spec itemsize(dtype_atom() | v3_data_type()) :: 1 | 2 | 4 | 8 | 16

Returns the size in bytes for a given dtype.

Parameters

  • dtype - Internal dtype atom or v3 type string

Returns

  • Size in bytes

Examples

iex> ExZarr.DataType.itemsize(:int32)
4

iex> ExZarr.DataType.itemsize(:float64)
8

iex> ExZarr.DataType.itemsize("int16")
2

micros_to_datetime(micros)

@spec micros_to_datetime(integer()) :: {:ok, DateTime.t()} | {:error, term()}

Convert a datetime value (microseconds since epoch) to an Elixir DateTime.

Parameters

  • micros - Microseconds since Unix epoch

Returns

  • DateTime struct or error

Examples

iex> ExZarr.DataType.micros_to_datetime(1609459200000000)
{:ok, ~U[2021-01-01 00:00:00.000000Z]}

pack(value, atom)

@spec pack(term(), atom()) :: nonempty_binary()

Pack a value into binary format for a given dtype.

Parameters

  • value - Value to pack
  • dtype - Internal dtype atom

Returns

  • Binary data

Examples

iex> ExZarr.DataType.pack(42, :int32)
<<42, 0, 0, 0>>

iex> ExZarr.DataType.pack(true, :bool)
<<1>>

iex> ExZarr.DataType.pack({3.0, 4.0}, :complex64)
<<0, 0, 64, 64, 0, 0, 128, 64>>

parse_datetime(datetime_string)

@spec parse_datetime(binary()) ::
  {:error,
   :incompatible_calendars
   | :invalid_date
   | :invalid_format
   | :invalid_time
   | :missing_offset}
  | {:ok, integer()}

Parse an ISO 8601 datetime string into microseconds since epoch.

Parameters

  • datetime_string - ISO 8601 formatted string

Returns

  • {:ok, microseconds} or {:error, reason}

Examples

iex> ExZarr.DataType.parse_datetime("2021-01-01T00:00:00Z")
{:ok, 1609459200000000}

signed?(dtype)

@spec signed?(dtype_atom()) :: boolean()

Returns whether a dtype represents a signed integer type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if signed integer, false otherwise

Examples

iex> ExZarr.DataType.signed?(:int32)
true

iex> ExZarr.DataType.signed?(:uint32)
false

iex> ExZarr.DataType.signed?(:float64)
false

supported_types()

@spec supported_types() :: [dtype_atom(), ...]

Returns all supported internal dtype atoms.

Returns

  • List of dtype atoms

Examples

iex> ExZarr.DataType.supported_types()
[:int8, :int16, :int32, :int64, :uint8, :uint16, :uint32, :uint64, :float32, :float64, :bool, :complex64, :complex128, :datetime64, :timedelta64]

timedelta?(arg1)

@spec timedelta?(dtype_atom()) :: boolean()

Returns whether a dtype represents a timedelta type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if timedelta, false otherwise

Examples

iex> ExZarr.DataType.timedelta?(:timedelta64)
true

iex> ExZarr.DataType.timedelta?(:int64)
false

to_v2(dtype_atom)

@spec to_v2(dtype_atom()) :: v2_dtype_string()

Converts an internal dtype atom to a v2 NumPy dtype string.

Returns little-endian format by default ("<" prefix).

Parameters

  • dtype_atom - Internal data type atom

Returns

  • v2 NumPy dtype string

Examples

iex> ExZarr.DataType.to_v2(:int32)
"<i4"

iex> ExZarr.DataType.to_v2(:float64)
"<f8"

iex> ExZarr.DataType.to_v2(:uint8)
"<u1"

to_v3(dtype_atom)

@spec to_v3(dtype_atom()) :: v3_data_type()

Converts an internal dtype atom to a v3 data type string.

Parameters

  • dtype_atom - Internal data type atom

Returns

  • v3 data type string

Examples

iex> ExZarr.DataType.to_v3(:int32)
"int32"

iex> ExZarr.DataType.to_v3(:float64)
"float64"

iex> ExZarr.DataType.to_v3(:uint8)
"uint8"

unpack(arg, atom)

@spec unpack(nonempty_binary(), atom()) :: boolean() | number() | {float(), float()}

Unpack a binary value into Elixir terms for a given dtype.

Parameters

  • binary - Binary data to unpack
  • dtype - Internal dtype atom

Returns

  • Unpacked value

Examples

iex> ExZarr.DataType.unpack(<<42, 0, 0, 0>>, :int32)
42

iex> ExZarr.DataType.unpack(<<1>>, :bool)
true

iex> ExZarr.DataType.unpack(<<0, 0, 64, 64, 0, 0, 128, 64>>, :complex64)
{3.0, 4.0}

unsigned?(dtype)

@spec unsigned?(dtype_atom()) :: boolean()

Returns whether a dtype represents an unsigned integer type.

Parameters

  • dtype - Internal dtype atom

Returns

  • true if unsigned integer, false otherwise

Examples

iex> ExZarr.DataType.unsigned?(:uint32)
true

iex> ExZarr.DataType.unsigned?(:int32)
false

iex> ExZarr.DataType.unsigned?(:float64)
false

validate(dtype)

@spec validate(dtype_atom() | String.t()) :: :ok | {:error, :unsupported_dtype}

Validates that a dtype is supported.

Parameters

  • dtype - Dtype atom, v2 string, or v3 string

Returns

  • :ok if valid
  • {:error, reason} if invalid

Examples

iex> ExZarr.DataType.validate(:int32)
:ok

iex> ExZarr.DataType.validate("float64")
:ok

iex> ExZarr.DataType.validate("<i4")
:ok

iex> ExZarr.DataType.validate(:invalid)
{:error, :unsupported_dtype}