Arrow.Ipc.Body (Arrow v0.1.0)

Copy Markdown

Encodes and decodes the body bytes of an Arrow IPC RecordBatch.

An Arrow IPC body is the concatenation of every column's raw buffers (validity, offsets, values, ...) in depth-first preorder, each padded to an 8-byte boundary. The associated RecordBatch metadata describes the body as two parallel flat lists:

  • nodes — one FieldNode per physical array (each nested array contributes its own node), carrying length and null_count.
  • buffers — one Buffer per physical buffer, carrying the byte offset into the body and the byte length.

This module owns the walking order, the per-type buffer set, and the alignment padding. It does not own the FlatBuffers encoding of those descriptors — that's the internal Ipc.Metadata module.

Per-type buffer layout

Null              0 buffers, 1 node
Bool              [validity, values]  (both bitmaps)
Int*/UInt*/Float* [validity, values]
Date32/Date64     [validity, values]
Timestamp         [validity, values]
Time32/Time64     [validity, values]
Duration          [validity, values]
Decimal128        [validity, values]
FixedSizeBinary   [validity, values]
Utf8/Binary       [validity, offsets, values]
List              [validity, offsets]  + 1 child array
FixedSizeList     [validity]           + 1 child array
Struct            [validity]           + N child arrays
Map               [validity, offsets]  + 1 entries struct array

Summary

Types

buffer_desc()

Buffer descriptor — a per-buffer (offset, length) into the body.

encoded()

The result of encode/1: descriptors plus the concatenated body bytes.

node_desc()

FieldNode descriptor — a per-array stats entry.

Functions

decode(schema, length, nodes, buffers, body)

Reverses encode/1: takes the schema, row count, and the descriptor + body produced by an encoder (ours or another implementation), and reconstructs the RecordBatch.

decode_array_buffers(field, nodes, buffers, body)

Decodes a single array out of pre-extracted descriptors + body bytes. Used by the IPC layer for DictionaryBatch decoding.

encode(record_batch)

Walks batch's columns and produces the body bytes plus the parallel nodes and buffers lists.

encode_array(array)

Encodes a single array's buffers (no surrounding RecordBatch). Used by the IPC layer to produce a DictionaryBatch body.

Types

buffer_desc()

@type buffer_desc() :: %{offset: non_neg_integer(), length: non_neg_integer()}

Buffer descriptor — a per-buffer (offset, length) into the body.

encoded()

@type encoded() :: %{
  length: non_neg_integer(),
  nodes: [node_desc()],
  buffers: [buffer_desc()],
  body: binary()
}

The result of encode/1: descriptors plus the concatenated body bytes.

node_desc()

@type node_desc() :: %{length: non_neg_integer(), null_count: non_neg_integer()}

FieldNode descriptor — a per-array stats entry.

Functions

decode(schema, length, nodes, buffers, body)

@spec decode(
  Arrow.Schema.t(),
  non_neg_integer(),
  [node_desc()],
  [buffer_desc()],
  binary()
) ::
  Arrow.RecordBatch.t()

Reverses encode/1: takes the schema, row count, and the descriptor + body produced by an encoder (ours or another implementation), and reconstructs the RecordBatch.

decode_array_buffers(field, nodes, buffers, body)

@spec decode_array_buffers(Arrow.Field.t(), [node_desc()], [buffer_desc()], binary()) ::
  Arrow.Array.t()

Decodes a single array out of pre-extracted descriptors + body bytes. Used by the IPC layer for DictionaryBatch decoding.

encode(record_batch)

@spec encode(Arrow.RecordBatch.t()) :: encoded()

Walks batch's columns and produces the body bytes plus the parallel nodes and buffers lists.

encode_array(array)

@spec encode_array(Arrow.Array.t()) :: %{
  nodes: [node_desc()],
  buffers: [buffer_desc()],
  body: binary()
}

Encodes a single array's buffers (no surrounding RecordBatch). Used by the IPC layer to produce a DictionaryBatch body.