ExArrow.IPC.File (ex_arrow v0.8.0)

IPC file format reader: random access to schema, batch count, and batches by index.

Use from_file/1 or from_binary/1 to open, then schema/1, batch_count/1, and get_batch/2 for random access. Data remains in native memory until you use ExArrow.RecordBatch / ExArrow.Schema accessors.

Summary

Types

t()

Functions

batch_count(file)

Returns the number of record batches in the file.

from_binary(binary)

Opens an IPC file format from in-memory binary (e.g. for tests). Random access as with from_file.

from_file(path)

Opens an IPC file (file format) from a path. Enables random access to batches.

get_batch(file, index)

Returns the record batch at the given 0-based index as an ExArrow.RecordBatch handle.

schema(file)

Returns the schema of the IPC file as an ExArrow.Schema handle. Returns {:error, message} if the file handle is invalid (e.g. poisoned lock).

write(path, schema, batches)

Write schema and batches to an Arrow IPC file-format path (magic + footer, random-access compatible).

Types

t()

@opaque t()

Functions

batch_count(file)

@spec batch_count(t()) :: non_neg_integer()

Returns the number of record batches in the file.

from_binary(binary)

@spec from_binary(binary()) :: {:ok, t()} | {:error, String.t()}

Opens an IPC file format from in-memory binary (e.g. for tests). Random access as with from_file.

from_file(path)

@spec from_file(Path.t()) :: {:ok, t()} | {:error, String.t()}

Opens an IPC file (file format) from a path. Enables random access to batches.

get_batch(file, index)

@spec get_batch(t(), non_neg_integer()) ::
  {:ok, ExArrow.RecordBatch.t()} | {:error, String.t()}

Returns the record batch at the given 0-based index as an ExArrow.RecordBatch handle.

schema(file)

@spec schema(t()) :: {:ok, ExArrow.Schema.t()} | {:error, String.t()}

Returns the schema of the IPC file as an ExArrow.Schema handle. Returns {:error, message} if the file handle is invalid (e.g. poisoned lock).

write(path, schema, batches)

@spec write(Path.t(), ExArrow.Schema.t(), [ExArrow.RecordBatch.t()]) ::
  :ok | {:error, String.t()}

Write schema and batches to an Arrow IPC file-format path (magic + footer, random-access compatible).

This is distinct from ExArrow.IPC.Writer.to_file/3, which writes the IPC stream format.