Arrow. Ipc. File
(Arrow v0.1.0)
Copy Markdown
Encodes and decodes the Arrow IPC file format.
The file format wraps the streaming format with magic bytes and a Footer FlatBuffers table that supports random access — consumers can jump straight to the footer and read individual record batches by offset, without scanning the whole stream linearly.
On-disk layout
"ARROW1\0\0" (8 bytes magic prefix)
<schema message> (stream-format framing)
<batch 0 message>
<batch 1 message>
...
<EOS> (continuation + zero length; 8 bytes)
<Footer FlatBuffer> (variable; contains schema + Block descriptors)
<footer length> (4 bytes, i32 little-endian)
"ARROW1" (6 bytes magic suffix)Each Block in the Footer is {offset, metaDataLength, bodyLength},
pointing at one record batch message within the file:
offset— absolute byte offset where the message's continuation marker starts.metaDataLength— the framing-inclusive metadata size:8 + padded_metadata_bytes. Per the upstream spec the recorded value includes the 8-byte prefix.bodyLength— the raw body byte count after the metadata.
Notes
- Dictionaries are written as DictionaryBatch messages between the
Schema and the record batches, with Block descriptors recorded in
the Footer's
dictionarieslist; decode reads them back through those Blocks (delta dictionaries are rejected). - The Schema appears once in the inline stream prefix and again in the Footer. Decode uses only the Footer's copy — the inline stream prefix is never parsed — and does not cross-check the two.
Errors
decode/1 returns {:ok, payload} or {:error, %Arrow.DecodeError{}}
with kind :unsupported (the input uses a feature this library
deliberately rejects) or :malformed (the input is corrupt, truncated,
or internally inconsistent). decode!/1 raises the same error.
Summary
Functions
Parses an Arrow IPC file binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.
Like decode/1, but returns the payload directly and raises
Arrow.DecodeError on failure.
Encodes a schema, an optional dictionaries registry, and zero or more record batches into a complete Arrow IPC file as a binary.
Types
@type payload() :: %{ schema: Arrow.Schema.t(), dictionaries: %{optional(non_neg_integer()) => Arrow.Array.t()}, batches: [Arrow.RecordBatch.t()] }
Everything a decoded file carries: schema, dictionary registry, batches.
Functions
@spec decode(binary()) :: {:ok, payload()} | {:error, Arrow.DecodeError.t()}
Parses an Arrow IPC file binary into {:ok, %{schema:, dictionaries:, batches:}} or {:error, %Arrow.DecodeError{}}.
Like decode/1, but returns the payload directly and raises
Arrow.DecodeError on failure.
@spec encode( Arrow.Schema.t(), [Arrow.RecordBatch.t()], %{optional(non_neg_integer()) => Arrow.Array.t()} ) :: binary()
Encodes a schema, an optional dictionaries registry, and zero or more record batches into a complete Arrow IPC file as a binary.