Parquex.Store (parquex v0.3.0)

View Source

A reusable local or S3-compatible object-storage namespace.

A store owns backend configuration and one reusable native handle while operations address objects with relative keys.

Examples

iex> {:ok, store} = Parquex.Store.open(:local, root: System.tmp_dir!())
iex> Parquex.Store.backend(store)
:local

iex> {:ok, store} = Parquex.Store.open(:s3, bucket: "events", prefix: "archive")
iex> Parquex.Store.prefix(store)
"archive/"

Summary

Types

backend()
t()

Functions

backend(store)

Returns the configured backend.

bucket(store)

Returns the configured S3 bucket, or nil for local storage.

cancel(writer)

Cancels an open writer and discards its incomplete data.

delete(store, key)

Deletes one relative object key.

head(store, key)

Returns metadata for one relative object key.

identity(store)

Returns the stable identity of this native store handle.

list(store, prefix \\ "")

Lists metadata for keys beginning with the relative prefix.

open(backend, options \\ [])

Creates a validated reusable storage namespace.

open_writer(store, key)

Opens a writer for a relative key.

prefix(store)

Returns the normalized optional base prefix.

publish(writer)

Completes a writer, replacing the key when it already exists.

put(store, key, chunks)

Consumes bounded chunks and writes one object.

read(store, key)

Materializes one finite object using bounded native range reads.

read_range(store, key, offset, length)

Reads at most length bytes from one object without exceeding the configured bound.

root(store)

Returns the configured local root, or nil for S3.

write(writer, data)

Writes one iodata chunk to an open store writer.

Types

backend()

@type backend() :: :local | :s3

t()

@opaque t()

Functions

backend(store)

@spec backend(t()) :: backend()

Returns the configured backend.

bucket(store)

@spec bucket(t()) :: String.t() | nil

Returns the configured S3 bucket, or nil for local storage.

cancel(writer)

@spec cancel(Parquex.Store.Writer.t()) :: :ok | {:error, Parquex.Error.t()}

Cancels an open writer and discards its incomplete data.

delete(store, key)

@spec delete(t(), String.t()) :: :ok | {:error, Parquex.Error.t()}

Deletes one relative object key.

head(store, key)

@spec head(t(), String.t()) ::
  {:ok, Parquex.Store.Metadata.t()} | {:error, Parquex.Error.t()}

Returns metadata for one relative object key.

identity(store)

@spec identity(t()) :: {:ok, pos_integer()} | {:error, Parquex.Error.t()}

Returns the stable identity of this native store handle.

list(store, prefix \\ "")

@spec list(t(), String.t()) ::
  {:ok, [Parquex.Store.Metadata.t()]} | {:error, Parquex.Error.t()}

Lists metadata for keys beginning with the relative prefix.

open(backend, options \\ [])

@spec open(
  backend(),
  keyword()
) :: {:ok, t()} | {:error, Parquex.Error.t()}

Creates a validated reusable storage namespace.

open_writer(store, key)

@spec open_writer(t(), String.t()) ::
  {:ok, Parquex.Store.Writer.t()} | {:error, Parquex.Error.t()}

Opens a writer for a relative key.

prefix(store)

@spec prefix(t()) :: String.t()

Returns the normalized optional base prefix.

publish(writer)

@spec publish(Parquex.Store.Writer.t()) ::
  {:ok, Parquex.Store.Metadata.t()} | {:error, Parquex.Error.t()}

Completes a writer, replacing the key when it already exists.

put(store, key, chunks)

@spec put(t(), String.t(), Enumerable.t()) ::
  {:ok, Parquex.Store.Metadata.t()} | {:error, Parquex.Error.t()}

Consumes bounded chunks and writes one object.

read(store, key)

@spec read(t(), String.t()) :: {:ok, binary()} | {:error, Parquex.Error.t()}

Materializes one finite object using bounded native range reads.

read_range(store, key, offset, length)

@spec read_range(t(), String.t(), non_neg_integer(), non_neg_integer()) ::
  {:ok, binary()} | {:error, Parquex.Error.t()}

Reads at most length bytes from one object without exceeding the configured bound.

root(store)

@spec root(t()) :: Path.t() | nil

Returns the configured local root, or nil for S3.

write(writer, data)

@spec write(Parquex.Store.Writer.t(), iodata()) :: :ok | {:error, Parquex.Error.t()}

Writes one iodata chunk to an open store writer.