Arrow.
Copy Markdown
Encodes and decodes the Arrow IPC streaming format.
An Arrow stream is a sequence of length-prefixed messages, each of
which carries a FlatBuffers-encoded Message metadata table and,
optionally, a raw body of column buffers.
Frame layout per message
continuation marker (u32 0xFFFFFFFF, 4 bytes)
metadata length (i32 little-endian, 4 bytes)
metadata bytes (variable, padded to 8-byte boundary)
body bytes (variable, already 8-byte aligned per buffer codec)An end-of-stream marker is the continuation marker followed by a zero
length: 0xFFFFFFFF 0x00000000. Some pre-0.15 producers omitted the
continuation marker — decode/1 accepts both forms.
Stream composition
A well-formed Arrow stream is:
<Schema message>
<DictionaryBatch message> (zero or more)
<RecordBatch message> (zero or more)
<EOS>DictionaryBatch messages are decoded into the result's dictionaries
registry (delta dictionaries are rejected). Footer interactions belong
to the file format — see Arrow.Ipc.File, which reuses the frame
builders exposed here.
Errors
decode/1 returns {:ok, payload} or {:error, %Arrow.DecodeError{}}
with kind :unsupported (the input uses a feature this library
deliberately rejects) or :malformed (the input is corrupt, truncated,
or internally inconsistent). decode!/1 raises the same error.
Summary
Functions
Parses a complete Arrow IPC stream binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.
Like decode/1, but returns the payload directly and raises
Arrow.DecodeError on failure.
Builds a single DictionaryBatch message frame from a dictionary id and its values array. Returns iodata + framing-inclusive metadata length + body length.
Builds a complete Arrow IPC stream from a schema, optional dictionaries registry, and zero or more record batches.
End-of-stream marker: continuation marker followed by a zero length.
Builds a single RecordBatch message frame: continuation + length + padded metadata + body. Returns iodata, the framing-inclusive metadata length, and the body length.
Builds a single Schema message frame: continuation + length + padded
metadata. No body. Returns iodata and the framing-inclusive metadata
length (i.e. 8 + padded_metadata).
Validates the schema ↔ dictionaries registry pairing before encoding:
every dictionary-encoded field in schema must have an entry for its
id in dictionaries, or the output would reference a dictionary that
is never written (spec-invalid). Raises ArgumentError otherwise.
Types
@type payload() :: %{ schema: Arrow.Schema.t(), dictionaries: %{optional(non_neg_integer()) => Arrow.Array.t()}, batches: [Arrow.RecordBatch.t()] }
Everything a decoded stream carries: schema, dictionary registry, batches.
Functions
@spec decode(binary()) :: {:ok, payload()} | {:error, Arrow.DecodeError.t()}
Parses a complete Arrow IPC stream binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.
Like decode/1, but returns the payload directly and raises
Arrow.DecodeError on failure.
@spec dictionary_batch_message_frame(non_neg_integer(), Arrow.Array.t()) :: {iodata(), non_neg_integer(), non_neg_integer()}
Builds a single DictionaryBatch message frame from a dictionary id and its values array. Returns iodata + framing-inclusive metadata length + body length.
@spec encode( Arrow.Schema.t(), [Arrow.RecordBatch.t()], %{optional(non_neg_integer()) => Arrow.Array.t()} ) :: binary()
Builds a complete Arrow IPC stream from a schema, optional dictionaries registry, and zero or more record batches.
@spec eos() :: binary()
End-of-stream marker: continuation marker followed by a zero length.
@spec record_batch_message_frame(Arrow.RecordBatch.t()) :: {iodata(), non_neg_integer(), non_neg_integer()}
Builds a single RecordBatch message frame: continuation + length + padded metadata + body. Returns iodata, the framing-inclusive metadata length, and the body length.
Exposed for Arrow.Ipc.File, same reason as schema_message_frame/1.
@spec schema_message_frame(Arrow.Schema.t()) :: {iodata(), non_neg_integer()}
Builds a single Schema message frame: continuation + length + padded
metadata. No body. Returns iodata and the framing-inclusive metadata
length (i.e. 8 + padded_metadata).
Exposed for Arrow.Ipc.File, which needs to record per-message file
offsets in the Footer's Block list.
@spec validate_dictionaries!(Arrow.Schema.t(), %{ optional(non_neg_integer()) => Arrow.Array.t() }) :: :ok
Validates the schema ↔ dictionaries registry pairing before encoding:
every dictionary-encoded field in schema must have an entry for its
id in dictionaries, or the output would reference a dictionary that
is never written (spec-invalid). Raises ArgumentError otherwise.
The converse direction (registry id with no referencing field) is checked where the DictionaryBatch messages are emitted.
Exposed for Arrow.Ipc.File, which builds its frames directly.