Arrow.Array (Arrow v0.1.0)

Copy Markdown

Per-type column structs. Each module under Arrow.Array.* represents one concrete Arrow array.

Shape

Primitive arrays carry a validity bitmap and a value buffer:

%Arrow.Array.Int64{length: 3, null_count: 1,
                   # LSB-first: slots 0 and 2 valid, slot 1 null
                   validity: <<0b00000101>>,
                   values:   <<1::little-signed-64,
                               0::little-signed-64,
                               3::little-signed-64>>}

Variable-binary arrays (Utf8, Binary) add an offsets buffer of length + 1 little-endian int32s; entry i of the value buffer is the byte range offsets[i]..offsets[i+1].

List arrays mirror that layout with a child array in place of the value bytes. Struct arrays carry one child array per struct member and no value buffer.

The validity bitmap is laid out as Arrow specifies: bit i of byte floor(i/8), with bit 0 being the least-significant bit. A bit value of 1 means the slot is valid; 0 means null. The bitmap may be omitted (set to nil) when null_count = 0.

Summary

Types

t()

Functions

length(map)

The number of slots in the array (length).

null_count(arg1)

The number of null slots in the array (null_count).

Types

t()

@type t() ::
  %Arrow.Array.Null{length: term()}
  | %Arrow.Array.Bool{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Int8{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Int16{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Int32{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Int64{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.UInt8{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.UInt16{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.UInt32{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.UInt64{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Float32{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Float64{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Utf8{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Binary{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Date32{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Date64{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Timestamp{
      length: term(),
      null_count: term(),
      timezone: term(),
      unit: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.List{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Struct{
      children: term(),
      length: term(),
      null_count: term(),
      validity: term()
    }
  | %Arrow.Array.Time32{
      length: term(),
      null_count: term(),
      unit: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Time64{
      length: term(),
      null_count: term(),
      unit: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Duration{
      length: term(),
      null_count: term(),
      unit: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.FixedSizeBinary{
      byte_width: term(),
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.FixedSizeList{
      length: term(),
      list_size: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Decimal32{
      length: term(),
      null_count: term(),
      precision: term(),
      scale: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Decimal64{
      length: term(),
      null_count: term(),
      precision: term(),
      scale: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Decimal128{
      length: term(),
      null_count: term(),
      precision: term(),
      scale: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Decimal256{
      length: term(),
      null_count: term(),
      precision: term(),
      scale: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Map{
      keys_sorted: term(),
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.Dictionary{dictionary_id: term(), indices: term()}
  | %Arrow.Array.IntervalYearMonth{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.IntervalDayTime{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.IntervalMonthDayNano{
      length: term(),
      null_count: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.LargeUtf8{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.LargeBinary{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }
  | %Arrow.Array.LargeList{
      length: term(),
      null_count: term(),
      offsets: term(),
      validity: term(),
      values: term()
    }

Functions

length(map)

@spec length(t()) :: non_neg_integer()

The number of slots in the array (length).

null_count(arg1)

@spec null_count(t()) :: non_neg_integer()

The number of null slots in the array (null_count).