SimdJson (SimdJson v1.0.0)

View Source

Decodes complete JSON values, opens binary or native file-backed documents, selects scalar values, and lazily streams projected rows using SIMD-accelerated parsing.

select/2 extracts several named scalar paths from either a JSON binary or a caller-owned document. Results use the exact atom or binary keys supplied by the caller. Object and array leaves are deliberately not materialized.

A document belongs to the process that opened it. Owner close/1 is idempotent and waits for native cleanup; another process receives a stable :not_owner error even when the document has already been closed. Projection is one-shot: after native cursor access begins, success or failure consumes the document. Use another document, or call select/2 with the binary again, for another projection.

Projection grammar errors are tagged :invalid_projection. Values that are not binaries or genuine document resources raise ArgumentError. Other parsing, path, ownership, lifecycle, allocation, and cancellation failures return {:error, %SimdJson.Error{}} without a partial result. Selected strings are fresh binaries independent of their source.

Decode accepts binaries and an empty option list only. Its bang wrapper is Elixir-only and raises the same structured error returned by decode/2. Eager decode allocates the complete value; prefer projection or streaming for large inputs when only a subset is needed.

For large files, open_file/1, select_file/2, and stream_file/2 pass only the path through the BEAM boundary. File streaming delegates the memory map, fixed parser windows, document iteration, and projection batching to simdjson; do not wrap these calls in File.read/1.

The API intentionally has no projection bang variant, JSONPath, wildcard, default-field policy, public compiled plan, raw cursor/batch operation, ownership transfer, or native-handle operation.

All native execution uses a fixed bounded worker pool. Queue saturation returns a redacted :busy error. The qualified target is Ubuntu 24.04 x86-64.

Examples

iex> SimdJson.decode(~s({"ready":true}))
{:ok, %{"ready" => true}}
iex> SimdJson.decode("[1,2,3]", [])
{:ok, [1, 2, 3]}
iex> SimdJson.decode!(~s("hello"))
"hello"
iex> SimdJson.decode!("null", [])
nil

iex> {:ok, document} = SimdJson.open(~s({"ready": true}))
iex> inspect(document)
"#SimdJson.Document<opaque>"
iex> SimdJson.close(document)
:ok
iex> SimdJson.close(document)
:ok

iex> json = ~s({"customer":{"id":7,"name":"Acme"},"orders":[{"sku":"A-1"}]})
iex> projection = [{:id, ["customer", "id"]}, {"sku", ["orders", 0, "sku"]}]
iex> SimdJson.select(json, projection)
{:ok, %{"sku" => "A-1", id: 7}}

iex> {:ok, document} = SimdJson.open(~s({"value":42}))
iex> SimdJson.select(document, value: ["value"])
{:ok, %{value: 42}}
iex> {:error, consumed} = SimdJson.select(document, value: ["value"])
iex> consumed.reason
:cursor_consumed
iex> SimdJson.close(document)
:ok

iex> {:error, missing} = SimdJson.select(~s({"ready":true}), value: ["value"])
iex> {missing.reason, missing.path}
{:no_such_field, ["value"]}
iex> inspect(missing) =~ "value"
false

iex> {:error, type_error} = SimdJson.select(~s({"items":[]}), items: ["items"])
iex> type_error.reason
:incorrect_type

iex> {:error, invalid} = SimdJson.select("not inspected", [])
iex> {invalid.reason, invalid.byte_offset, invalid.path}
{:invalid_projection, nil, nil}

iex> source = ~s({"selected":"small","ignored":"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"})
iex> {:ok, %{selected: selected}} = SimdJson.select(source, selected: ["selected"])
iex> source = nil
iex> :erlang.garbage_collect(self())
iex> {source, selected}
{nil, "small"}

iex> function_exported?(SimdJson, :select!, 2)
false
iex> Code.ensure_loaded?(SimdJson.CompiledProjection)
false

iex> {:error, error} = SimdJson.open("[1,")
iex> {error.reason, error.message}
{:unexpected_eof, "unexpected end of JSON input"}
iex> inspect(error) =~ "[1,"
false

iex> SimdJson.open(:not_a_binary)
** (ArgumentError) expected JSON input to be a binary

iex> {:ok, document} = SimdJson.open("null")
iex> task = Task.async(fn -> SimdJson.close(document) end)
iex> {:error, non_owner_error} = Task.await(task)
iex> non_owner_error.reason
:not_owner
iex> SimdJson.close(document)
:ok

Summary

Types

A JSON array index in the unsigned 64-bit domain.

An explicit top-level file format supported by native batching.

Options for native file-backed document streaming.

A UTF-8 JSON object-key segment.

An exact caller-supplied result key. No atom is created from a binary key.

A non-empty path to one scalar JSON value.

One object-key or array-index segment in a projection path.

A non-empty proper list of projection entries with unique output keys.

One output key paired with its scalar path.

The transactional map returned after every selected path succeeds.

A selected scalar converted to an independent BEAM term.

The exact-key projection applied independently to every array row.

Options for bounded array streaming.

One scalar-only projected row.

A path locating the array to stream; the empty path selects a root array.

A target segment locating the array to stream.

Functions

Closes an opaque document owned by the calling process.

Decodes one complete JSON binary into Elixir maps, lists, binaries, numbers, booleans, and nil.

Decodes one complete JSON binary, returning its value or raising the same SimdJson.Error returned by decode/2.

Opens one JSON binary as an opaque document owned by the calling process.

Opens one immutable regular-file JSON source through simdjson's native memory-map owner.

Selects several scalar paths from a JSON binary or caller-owned document.

Selects scalar paths directly from one immutable regular-file JSON source.

Constructs a lazy, owner-bound Enumerable over projected JSON array rows.

Constructs a lazy, owner-bound stream over top-level documents in a file.

Types

array_index()

@type array_index() :: 0..18_446_744_073_709_551_615

A JSON array index in the unsigned 64-bit domain.

file_stream_format()

@type file_stream_format() ::
  :json_array | :ndjson | :json_sequence | :comma_delimited_json

An explicit top-level file format supported by native batching.

file_stream_option()

@type file_stream_option() ::
  {:format, file_stream_format()}
  | {:fields, stream_fields()}
  | {:batch_size, 1..10000}
  | {:max_batch_bytes, 1..67_108_864}

Options for native file-backed document streaming.

object_segment()

@type object_segment() :: binary()

A UTF-8 JSON object-key segment.

output_key()

@type output_key() :: atom() | binary()

An exact caller-supplied result key. No atom is created from a binary key.

path()

@type path() :: [path_segment(), ...]

A non-empty path to one scalar JSON value.

path_segment()

@type path_segment() :: object_segment() | array_index()

One object-key or array-index segment in a projection path.

projection()

@type projection() :: [projection_entry(), ...]

A non-empty proper list of projection entries with unique output keys.

projection_entry()

@type projection_entry() :: {output_key(), path()}

One output key paired with its scalar path.

projection_result()

@type projection_result() :: %{optional(output_key()) => scalar_result()}

The transactional map returned after every selected path succeeds.

scalar_result()

@type scalar_result() :: binary() | integer() | float() | boolean() | nil

A selected scalar converted to an independent BEAM term.

stream_fields()

@type stream_fields() :: projection()

The exact-key projection applied independently to every array row.

stream_option()

@type stream_option() ::
  {:path, stream_target_path()}
  | {:fields, stream_fields()}
  | {:batch_size, 1..10000}
  | {:max_batch_bytes, 1..67_108_864}

Options for bounded array streaming.

stream_row()

@type stream_row() :: %{optional(output_key()) => scalar_result()}

One scalar-only projected row.

stream_target_path()

@type stream_target_path() :: [stream_target_segment()]

A path locating the array to stream; the empty path selects a root array.

stream_target_segment()

@type stream_target_segment() :: path_segment()

A target segment locating the array to stream.

Functions

close(document)

@spec close(SimdJson.Document.t()) :: :ok | {:error, SimdJson.Error.t()}

Closes an opaque document owned by the calling process.

Owner close is idempotent and returns only after native cleanup completes. A non-owner receives {:error, %SimdJson.Error{reason: :not_owner}} without learning the document lifecycle. A non-document argument raises ArgumentError before cleanup is submitted.

decode(input)

@spec decode(binary()) :: {:ok, term()} | {:error, SimdJson.Error.t()}

Decodes one complete JSON binary into Elixir maps, lists, binaries, numbers, booleans, and nil.

The first compatibility release accepts only an empty option list. Decode executes through the bounded native worker pool and returns copied binary keys and strings.

decode(input, options)

@spec decode(
  binary(),
  keyword()
) :: {:ok, term()} | {:error, SimdJson.Error.t()}

decode!(input)

@spec decode!(binary()) :: term()

Decodes one complete JSON binary, returning its value or raising the same SimdJson.Error returned by decode/2.

decode!(input, options)

@spec decode!(
  binary(),
  keyword()
) :: term()

open(input)

@spec open(binary()) :: {:ok, SimdJson.Document.t()} | {:error, SimdJson.Error.t()}

Opens one JSON binary as an opaque document owned by the calling process.

Malformed input returns a structured error. A non-binary argument raises ArgumentError before native work is submitted.

open_file(path)

@spec open_file(binary()) ::
  {:ok, SimdJson.Document.t()} | {:error, SimdJson.Error.t()}

Opens one immutable regular-file JSON source through simdjson's native memory-map owner.

The path crosses the BEAM boundary, but the JSON bytes do not become an Elixir binary and are not copied into a padded native source allocation. The file must remain unchanged until the returned document is closed.

select(source, projection)

@spec select(binary() | SimdJson.Document.t(), projection()) ::
  {:ok, projection_result()} | {:error, SimdJson.Error.t()}

Selects several scalar paths from a JSON binary or caller-owned document.

The projection is completely validated before parsing or document reservation. It must be a non-empty proper list of {output_key, path} pairs. Output keys are existing atoms or binaries; paths are non-empty proper lists of UTF-8 binary object keys and unsigned 64-bit array indexes.

Success returns one map under the exact supplied keys. Strings are copied into fresh binaries. A document is a forward-only, single-owner, one-shot source: once cursor access starts, either success or operational failure consumes it. Invalid projection, invalid source, non-owner, closed, and proven pre-worker submission rejection do not consume a fresh document.

Source misuse raises ArgumentError. Projection, parse, path, lifecycle, allocation, and cancellation failures return a structured tagged error; no partial map escapes.

select_file(path, projection)

@spec select_file(binary(), projection()) ::
  {:ok, projection_result()} | {:error, SimdJson.Error.t()}

Selects scalar paths directly from one immutable regular-file JSON source.

The projection is validated before file access. Native code maps and parses the file without constructing a complete BEAM source binary or padded native source copy, copies only selected scalar results, and closes the mapping before this function returns. The source must remain unchanged throughout the operation.

stream(source, options)

Constructs a lazy, owner-bound Enumerable over projected JSON array rows.

:path and :fields are required. Options are fully validated immediately, but parsing, document admission, and native allocation wait until reduction. Binary streams may be constructed again; a document stream is one-shot. Batches default to 1,000 rows and 8 MiB, with maxima of 10,000 rows and 64 MiB. Returned string values are copied and each row contains scalars only.

Invalid sources or options raise ArgumentError. Runtime failures raise a redacted SimdJson.Error during enumeration. A stream captures its creating process and applies demand one batch at a time without prefetch.

stream_file(path, options)

@spec stream_file(binary(), [file_stream_option()]) :: SimdJson.Stream.t()

Constructs a lazy, owner-bound stream over top-level documents in a file.

:format and :fields are required. Supported formats are :json_array, :ndjson, :json_sequence, and :comma_delimited_json. The JSON bytes remain in a native memory map; simdjson parses bounded windows and returns only the copied projected scalars for each demanded batch.