Arrow.Json (Arrow v0.1.0)

Copy Markdown

Arrow integration test JSON format.

Arrow defines a JSON form used by apache/arrow:dev/archery to drive cross-language conformance tests. Every primitive type has an unambiguous JSON representation, validity is encoded as 0/1 arrays, and offsets are explicit.

The reader is in Arrow.Json.Reader; the writer in Arrow.Json.Writer. This module exposes thin shortcuts that round-trip via Jason.

Errors

decode/1 returns {:ok, payload} or {:error, %Arrow.DecodeError{}} with kind :unsupported (the input uses a feature this library deliberately rejects) or :malformed (the input is corrupt, truncated, or internally inconsistent — including invalid gzip or JSON). decode!/1 raises the same error.

Summary

Types

payload()

Everything a decoded document carries: schema, dictionary registry, batches.

Functions

decode(gzipped)

Reads a JSON document (or in-memory map) into a schema, dictionaries registry, and list of record batches.

decode!(input)

Like decode/1, but returns the payload directly and raises Arrow.DecodeError on failure.

encode(schema, batches, dictionaries \\ %{})

Encodes a schema plus record batches (and optional dictionaries registry) into the Arrow integration JSON form.

Types

payload()

@type payload() :: %{
  schema: Arrow.Schema.t(),
  dictionaries: %{optional(non_neg_integer()) => Arrow.Array.t()},
  batches: [Arrow.RecordBatch.t()]
}

Everything a decoded document carries: schema, dictionary registry, batches.

Functions

decode(gzipped)

@spec decode(binary() | map()) :: {:ok, payload()} | {:error, Arrow.DecodeError.t()}

Reads a JSON document (or in-memory map) into a schema, dictionaries registry, and list of record batches.

Gzipped input (the arrow-testing fixtures ship as .json.gz) is detected via the gzip magic bytes and decompressed transparently; the magic can never start a valid JSON document.

decode!(input)

@spec decode!(binary() | map()) :: payload()

Like decode/1, but returns the payload directly and raises Arrow.DecodeError on failure.

encode(schema, batches, dictionaries \\ %{})

@spec encode(
  Arrow.Schema.t(),
  [Arrow.RecordBatch.t()],
  %{optional(non_neg_integer()) => Arrow.Array.t()}
) :: iodata()

Encodes a schema plus record batches (and optional dictionaries registry) into the Arrow integration JSON form.