Arrow.Ipc.File (Arrow v0.1.0)

Copy Markdown

Encodes and decodes the Arrow IPC file format.

The file format wraps the streaming format with magic bytes and a Footer FlatBuffers table that supports random access — consumers can jump straight to the footer and read individual record batches by offset, without scanning the whole stream linearly.

On-disk layout

"ARROW1\0\0"          (8 bytes magic prefix)
<schema message>        (stream-format framing)
<batch 0 message>
<batch 1 message>
...
<EOS>                   (continuation + zero length; 8 bytes)
<Footer FlatBuffer>     (variable; contains schema + Block descriptors)
<footer length>         (4 bytes, i32 little-endian)
"ARROW1"                (6 bytes magic suffix)

Each Block in the Footer is {offset, metaDataLength, bodyLength}, pointing at one record batch message within the file:

  • offset — absolute byte offset where the message's continuation marker starts.
  • metaDataLength — the framing-inclusive metadata size: 8 + padded_metadata_bytes. Per the upstream spec the recorded value includes the 8-byte prefix.
  • bodyLength — the raw body byte count after the metadata.

Notes

  • Dictionaries are written as DictionaryBatch messages between the Schema and the record batches, with Block descriptors recorded in the Footer's dictionaries list; decode reads them back through those Blocks (delta dictionaries are rejected).
  • The Schema appears once in the inline stream prefix and again in the Footer. Decode uses only the Footer's copy — the inline stream prefix is never parsed — and does not cross-check the two.

Errors

decode/1 returns {:ok, payload} or {:error, %Arrow.DecodeError{}} with kind :unsupported (the input uses a feature this library deliberately rejects) or :malformed (the input is corrupt, truncated, or internally inconsistent). decode!/1 raises the same error.

Summary

Types

Everything a decoded file carries: schema, dictionary registry, batches.

Functions

Parses an Arrow IPC file binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.

Like decode/1, but returns the payload directly and raises Arrow.DecodeError on failure.

Encodes a schema, an optional dictionaries registry, and zero or more record batches into a complete Arrow IPC file as a binary.

Types

payload()

@type payload() :: %{
  schema: Arrow.Schema.t(),
  dictionaries: %{optional(non_neg_integer()) => Arrow.Array.t()},
  batches: [Arrow.RecordBatch.t()]
}

Everything a decoded file carries: schema, dictionary registry, batches.

Functions

decode(binary)

@spec decode(binary()) :: {:ok, payload()} | {:error, Arrow.DecodeError.t()}

Parses an Arrow IPC file binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.

decode!(binary)

@spec decode!(binary()) :: payload()

Like decode/1, but returns the payload directly and raises Arrow.DecodeError on failure.

encode(schema, batches, dictionaries \\ %{})

@spec encode(
  Arrow.Schema.t(),
  [Arrow.RecordBatch.t()],
  %{optional(non_neg_integer()) => Arrow.Array.t()}
) :: binary()

Encodes a schema, an optional dictionaries registry, and zero or more record batches into a complete Arrow IPC file as a binary.