Arrow.
Copy Markdown
Encoders and decoders between in-memory values and Arrow's raw buffer layout.
The Arrow columnar format spec describes the per-type set of buffers (a
validity bitmap, value buffer, optional offset buffer, etc.). This module
packs and unpacks each of those buffers in isolation. The higher-level
Arrow.Array.* structs hold the packed buffers directly, so the JSON
reader/writer and (eventually) the IPC reader/writer can share this code.
Conventions
- All multi-byte integers and floats are little-endian.
- Bitmaps (validity, Bool values) are LSB-first: slot
iis biti mod 8of bytefloor(i / 8), with the least-significant bit being slot 0. - For validity bitmaps,
1means valid,0means null. - When a buffer is materialised for the IPC body, callers are expected to
pad it to an 8-byte boundary using
pad_to_alignment/2.
Summary
Functions
Packs a list of 0 | 1 | true | false into a Bool values bitmap. Slots
whose validity bit is 0 may carry any value; callers typically pass 0.
Builds an int32 offsets buffer from a list of per-slot byte lengths.
int64 analogue of pack_int32_offsets/1, used by LargeUtf8,
LargeBinary, and LargeList.
Packs a list of numeric values into a little-endian fixed-width buffer.
Packs a list of 0 | 1 | true | false flags into Arrow's LSB-first validity
bitmap. Returns the bitmap padded to a whole number of bytes plus the null
count.
Pads binary with zero bytes up to the next multiple of alignment bytes
(default 8). Used at IPC body buffer boundaries.
Returns the number of pad bytes needed to reach the next alignment boundary.
Slices a variable-length value buffer using a list of int32 offsets,
returning one binary per slot.
int64 analogue of slice_variable/3, used by LargeUtf8 and
LargeBinary.
Unpacks a Bool values bitmap into a list of 0 | 1.
Unpacks an int32 offsets buffer of length + 1 entries.
Unpacks an int64 offsets buffer of length + 1 entries.
Unpacks a fixed-width buffer into a list of values.
Unpacks a validity bitmap into a list of 0 | 1 slots.
Types
Functions
Packs a list of 0 | 1 | true | false into a Bool values bitmap. Slots
whose validity bit is 0 may carry any value; callers typically pass 0.
iex> Arrow.Buffer.pack_bool_values([true, false, true, true])
<<0b00001101::8>>@spec pack_int32_offsets([non_neg_integer()]) :: binary()
Builds an int32 offsets buffer from a list of per-slot byte lengths.
The returned buffer has length(byte_lengths) + 1 entries, starting at 0
and ending at the sum.
iex> Arrow.Buffer.pack_int32_offsets([3, 2, 0])
<<0::little-signed-32, 3::little-signed-32,
5::little-signed-32, 5::little-signed-32>>@spec pack_int64_offsets([non_neg_integer()]) :: binary()
int64 analogue of pack_int32_offsets/1, used by LargeUtf8,
LargeBinary, and LargeList.
@spec pack_primitive([number()], primitive_kind()) :: binary()
Packs a list of numeric values into a little-endian fixed-width buffer.
kind is one of the atoms in primitive_kind/0. :int64-like atoms map
to signed packing; :uint* to unsigned; :float32 and :float64 to IEEE
floats. Null slots may carry any in-range placeholder (typically 0).
iex> Arrow.Buffer.pack_primitive([1, 2, 3], :int32)
<<1::little-signed-32, 2::little-signed-32, 3::little-signed-32>>@spec pack_validity([0 | 1 | boolean()]) :: {binary(), non_neg_integer()}
Packs a list of 0 | 1 | true | false flags into Arrow's LSB-first validity
bitmap. Returns the bitmap padded to a whole number of bytes plus the null
count.
iex> {bitmap, null_count} = Arrow.Buffer.pack_validity([1, 0, 1, 1, 0])
iex> null_count
2
iex> bitmap
<<0b00001101::8>>@spec pad_to_alignment(binary(), pos_integer()) :: binary()
Pads binary with zero bytes up to the next multiple of alignment bytes
(default 8). Used at IPC body buffer boundaries.
iex> Arrow.Buffer.pad_to_alignment(<<1, 2, 3>>)
<<1, 2, 3, 0, 0, 0, 0, 0>>@spec padding_size(non_neg_integer(), pos_integer()) :: non_neg_integer()
Returns the number of pad bytes needed to reach the next alignment boundary.
@spec slice_variable(binary(), binary(), non_neg_integer()) :: [binary()]
Slices a variable-length value buffer using a list of int32 offsets,
returning one binary per slot.
iex> offs = Arrow.Buffer.pack_int32_offsets([1, 2, 0, 3])
iex> Arrow.Buffer.slice_variable(offs, "abcdef", 4)
["a", "bc", "", "def"]@spec slice_variable_large(binary(), binary(), non_neg_integer()) :: [binary()]
int64 analogue of slice_variable/3, used by LargeUtf8 and
LargeBinary.
@spec unpack_bool_values(binary(), non_neg_integer()) :: [0 | 1]
Unpacks a Bool values bitmap into a list of 0 | 1.
@spec unpack_int32_offsets(binary(), non_neg_integer()) :: [integer()]
Unpacks an int32 offsets buffer of length + 1 entries.
@spec unpack_int64_offsets(binary(), non_neg_integer()) :: [integer()]
Unpacks an int64 offsets buffer of length + 1 entries.
@spec unpack_primitive(binary(), primitive_kind(), non_neg_integer()) :: [number()]
Unpacks a fixed-width buffer into a list of values.
iex> Arrow.Buffer.unpack_primitive(<<1::little-signed-32,
...> 2::little-signed-32,
...> 3::little-signed-32>>, :int32, 3)
[1, 2, 3]@spec unpack_validity(binary() | nil, non_neg_integer()) :: [0 | 1]
Unpacks a validity bitmap into a list of 0 | 1 slots.
iex> Arrow.Buffer.unpack_validity(<<0b00001101::8>>, 5)
[1, 0, 1, 1, 0]