Arrow.Logical (Arrow v0.1.0)

Logical (null-aware) view of Arrow.Array.* columns.

Two Arrow arrays are logically equal if, for each slot, both are null or both carry the same value. Byte-level differences that don't affect that observation — trailing junk bits in a validity bitmap, arbitrary content at null positions in a value buffer, two equivalent encodings of the same logical value — should not register as a difference.

This module produces a canonical Elixir representation of each array:

to_list(%Arrow.Array.Int32{...})   #=> [1, nil, 3, nil, 5]
to_list(%Arrow.Array.Utf8{...})    #=> ["foo", nil, "bar"]
to_list(%Arrow.Array.List{...})    #=> [[1, 2], [], [3, nil, 5]]
to_list(%Arrow.Array.Struct{...})  #=> [[1, "a"], [2, nil], nil]

Equality is then plain to_list(a) == to_list(b). equal?/2, arrays_equal?/2, and batches_equal?/2 are shortcuts.

Caveats:

Field names are not preserved for Struct slots — children are represented positionally so two equivalent arrays compare equal without needing the schema. Use to_list/1 plus the schema if you want field-named output.
Float32/Float64 NaN slots do not compare equal to themselves, because nil == nil but NaN != NaN per IEEE-754. Callers comparing float data with NaNs should custom-compare.
Decimal128 values are returned as their unscaled integer. Scale/precision are properties of the type, not the value.

Summary

Types

dictionaries()

Dictionary registry: id → dictionary values array.

Functions

arrays_equal?(a, b, dicts_a \\ %{}, dicts_b \\ %{})

True iff two arrays are logically equal, given their dictionary registries.

batches_equal?(a, b, dicts_a \\ %{}, dicts_b \\ %{})

True iff two record batches are logically equal.

dictionaries_equivalent_via_schemas?(sa, da, sb, db)

True iff every dictionary-encoded field's dictionary is resolvable in both registries, given the schemas referring to them.

equal?(a, b)

Registry-less logical equality. Arrow.RecordBatch inputs dispatch to batches_equal?/2; any other pair of array structs dispatches to arrays_equal?/2. Neither path takes dictionary registries, so inputs containing Arrow.Array.Dictionary columns raise — use arrays_equal?/4 or batches_equal?/4 with registries instead.

payloads_equivalent?(map1, map2)

True iff a complete payload (schema + dictionaries + batches) is logically equivalent to another. Dictionary encoding details are not compared directly: ID assignment, dictionary entry order, and unreferenced dictionary entries may all differ between the two sides, provided the resolved per-slot values agree.

schemas_equivalent?(a, b)

True iff two schemas describe the same logical shape.

to_list(array, dictionaries \\ %{})

Converts an Arrow.Array.* into a list of native Elixir values, with nil for null slots. Pass a dictionaries registry to resolve Arrow.Array.Dictionary columns to their dictionary values.

Types

dictionaries()

@type dictionaries() :: %{optional(non_neg_integer()) => Arrow.Array.t()}

Dictionary registry: id → dictionary values array.

Functions

arrays_equal?(a, b, dicts_a \\ %{}, dicts_b \\ %{})

@spec arrays_equal?(Arrow.Array.t(), Arrow.Array.t(), dictionaries(), dictionaries()) ::
  boolean()

True iff two arrays are logically equal, given their dictionary registries.

batches_equal?(a, b, dicts_a \\ %{}, dicts_b \\ %{})

@spec batches_equal?(
  Arrow.RecordBatch.t(),
  Arrow.RecordBatch.t(),
  dictionaries(),
  dictionaries()
) :: boolean()

True iff two record batches are logically equal.

Schemas are compared via schemas_equivalent?/2, which treats dictionary IDs as opaque labels — producers may assign different IDs to semantically-identical dictionaries. The optional registries are used to resolve Arrow.Array.Dictionary columns to their values during comparison.

dictionaries_equivalent_via_schemas?(sa, da, sb, db)

@spec dictionaries_equivalent_via_schemas?(
  Arrow.Schema.t(),
  dictionaries(),
  Arrow.Schema.t(),
  dictionaries()
) :: boolean()

True iff every dictionary-encoded field's dictionary is resolvable in both registries, given the schemas referring to them.

Walks the field tree of both schemas in parallel; for every dict-encoded field pair, checks that the corresponding dictionary is present in each registry under the field's ID. ID values themselves are never compared directly, and the dictionary arrays are not compared element-wise — two payloads carrying permuted or superset dictionaries with correspondingly adjusted indices encode the same logical values, so the value-level comparison is left to batches_equal?/4, which resolves indices through the registries. This mirrors the official Arrow integration comparison, which compares resolved values only.

equal?(a, b)

@spec equal?(term(), term()) :: boolean()

payloads_equivalent?(map1, map2)

@spec payloads_equivalent?(
  %{
    schema: Arrow.Schema.t(),
    dictionaries: dictionaries(),
    batches: [Arrow.RecordBatch.t()]
  },
  %{
    schema: Arrow.Schema.t(),
    dictionaries: dictionaries(),
    batches: [Arrow.RecordBatch.t()]
  }
) :: boolean()

schemas_equivalent?(a, b)

@spec schemas_equivalent?(Arrow.Schema.t(), Arrow.Schema.t()) :: boolean()

True iff two schemas describe the same logical shape.

Identical to == except that:

Arrow.Field.dictionary.id is treated as an opaque label: two fields where one has dictionary.id = 0 and the other has dictionary.id = 7 are still equivalent provided the rest of their dictionary annotation (index_type, is_ordered) and surrounding field structure agrees.
The names of a Map field's children (the entries struct and its key/value fields) are ignored — per the spec they are recommendations, not part of the logical type.

Use == instead if you need strict structural equality.

to_list(array, dictionaries \\ %{})

@spec to_list(Arrow.Array.t(), dictionaries()) :: [term()]

Converts an Arrow.Array.* into a list of native Elixir values, with nil for null slots. Pass a dictionaries registry to resolve Arrow.Array.Dictionary columns to their dictionary values.