ExDataSketch.Storage behaviour (ExDataSketch v0.10.1)

Behaviour, registry, and dispatching facade for ExDataSketch's persistence backends.

Each backend module (ExDataSketch.Storage.ETS, DETS, CubDB, Mnesia, Ecto) implements a consistent storage contract:

save/3 -- persist a sketch under a key
load/3 -- retrieve and deserialize a sketch by key
merge/3 -- atomically (or read-modify-write) merge a sketch into the persisted value
delete/2 -- remove a sketch by key

All backends serialize sketches using sketch_module.serialize/1 / sketch_module.deserialize/1. No backend stores raw sketch state; every stored value is a complete EXSK v2 frame with CRC32C checksum.

Available Backends

Backend	Module	Ref type	Distribution	Durability
ETS	`ExDataSketch.Storage.ETS`	`atom()` (table)	Per-node	Process lifetime
DETS	`ExDataSketch.Storage.DETS`	`atom()` (table)	Per-node	Disk
CubDB	`ExDataSketch.Storage.CubDB`	`pid() \| atom()`	Per-node	Disk
Mnesia	`ExDataSketch.Storage.Mnesia`	`atom()` (table)	Multi-node	Disk+RAM
Ecto	`ExDataSketch.Storage.Ecto`	`module()` (repo)	Multi-node	Database

ExDataSketch.Storage.Ecto is compiled only when the optional :ecto_sql dependency is available; the other four backends are always compiled (ExDataSketch.Storage.CubDB's functions raise a clear error if the optional :cubdb dependency is missing, but the module itself always exists).

Backend Ref Resolution

save/3, load/3, merge/3, and delete/2 accept a backend_ref/0, which is either:

an explicit {backend_module, ref} pair, for example {ExDataSketch.Storage.ETS, :my_table}; or
a bare ref (for example, a plain table name), resolved against the module configured under config :ex_data_sketch, :storage, backend: ... below. Raises ExDataSketch.Errors.InvalidOptionError if no default backend is configured.

A value is treated as an explicit pair only when it is a 2-tuple whose first element is an atom; none of the five shipped backends' own ref types (atom(), pid() | atom(), module()) are themselves 2-tuples, so this is unambiguous today.

Configuration

Backends can be enabled or disabled via application config:

config :ex_data_sketch, :persistence_backends,
  ets: [enabled: true],
  dets: [enabled: true],
  cubdb: [enabled: true],
  mnesia: [enabled: true],
  ecto: [enabled: true]

When not explicitly configured, a backend defaults to enabled if its runtime dependency is available.

A default backend module for the dispatching facade (save/3, load/3, merge/3, delete/2 below) can be configured separately:

config :ex_data_sketch, :storage, backend: ExDataSketch.Storage.ETS

`@behaviour ExDataSketch.Storage`

defmodule MyApp.Storage.Redis do
  @behaviour ExDataSketch.Storage

  @impl true
  def save(sketch, conn, key), do: ...

  @impl true
  def load(sketch_module, conn, key), do: ...

  @impl true
  def merge(sketch, conn, key), do: ...

  @impl true
  def delete(conn, key), do: ...
end

Summary

Types

backend_ref()

Either an explicit {backend_module, ref} pair, or a bare ref to be resolved against the default backend configured under config :ex_data_sketch, :storage, backend: .... See save/3 for the dispatching rules.

backend_type()

A registry key identifying one of the 5 backend modules. See backends/0.

key()

A key under which a sketch is stored. Backend-defined; typically a string or atom.

ref()

A backend-specific reference identifying where to store or find data -- an ETS/DETS/Mnesia table name (atom()), a CubDB process (pid() | atom()), or an Ecto repo module (module()).

Callbacks

child_spec(keyword)

Returns a supervisor child spec for backends whose ref must be started and supervised (for example, a CubDB process).

delete(ref, key)

Removes the value stored at key, if any.

load(sketch_module, ref, key)

Retrieves and deserializes a sketch by key.

merge(sketch, ref, key)

Merges a sketch into the value persisted at key.

save(sketch, ref, key)

Persists a sketch under key via the backend's storage mechanism.

Functions

backends()

Returns the registry mapping every backend's atom key to its module.

delete(backend_ref, key)

Removes the value stored at key via backend_ref, dispatching to the resolved backend module's delete/2.

load(sketch_module, backend_ref, key)

Retrieves and deserializes a sketch by key via backend_ref, dispatching to the resolved backend module's load/3.

merge(sketch, backend_ref, key)

Merges a sketch into the value persisted at key via backend_ref, dispatching to the resolved backend module's merge/3.

save(sketch, backend_ref, key)

Persists a sketch under key via backend_ref, dispatching to the resolved backend module's save/3.

Types

backend_ref()

@type backend_ref() :: {module(), ref()} | ref()

backend_type()

@type backend_type() :: :ets | :dets | :cubdb | :mnesia | :ecto

A registry key identifying one of the 5 backend modules. See backends/0.

key()

@type key() :: String.t() | atom() | term()

A key under which a sketch is stored. Backend-defined; typically a string or atom.

ref()

@type ref() :: term()

A backend-specific reference identifying where to store or find data -- an ETS/DETS/Mnesia table name (atom()), a CubDB process (pid() | atom()), or an Ecto repo module (module()).

Callbacks

child_spec(keyword)

(optional)

@callback child_spec(keyword()) :: Supervisor.child_spec() | {module(), term()} | module()

Returns a supervisor child spec for backends whose ref must be started and supervised (for example, a CubDB process).

This optional callback exists for custom @behaviour ExDataSketch.Storage implementations to adopt; none of the five shipped backends implement it today. ExDataSketch.Storage.CubDB -- the one shipped backend with an actual process to supervise -- instead documents starting and supervising CubDB.start_link/1 directly in the host application's own supervision tree (see its moduledoc); ETS, DETS, and Mnesia have no process of their own, and Ecto's repo is supervised by the host application the ordinary Ecto.Repo way, outside this behaviour entirely.

delete(ref, key)

@callback delete(ref(), key()) :: :ok | {:error, term()}

Removes the value stored at key, if any.

Succeeds (:ok) even if key does not exist.

load(sketch_module, ref, key)

@callback load(sketch_module :: module(), ref(), key()) ::
  {:ok, struct()} | {:error, :not_found | term()}

Retrieves and deserializes a sketch by key.

Returns {:error, :not_found} if no value is stored at key, or {:error, reason} for other lookup or deserialization failures. Takes the target sketch_module (rather than inferring it) because the backend cannot otherwise know which module's deserialize/1 to call.

merge(sketch, ref, key)

@callback merge(sketch :: struct(), ref(), key()) :: :ok | {:error, term()}

Merges a sketch into the value persisted at key.

If no value exists at key, this is equivalent to save/3. Backends that support atomic merge (ExDataSketch.Storage.Mnesia, ExDataSketch.Storage.Ecto, ExDataSketch.Storage.CubDB, the last via CubDB.transaction/2) do so via a transaction; others (ExDataSketch.Storage.ETS, ExDataSketch.Storage.DETS) perform a non-atomic read-modify-write cycle -- see each backend's own merge/3 documentation for its concurrency guarantees.

save(sketch, ref, key)

@callback save(sketch :: struct(), ref(), key()) :: :ok | {:error, term()}

Persists a sketch under key via the backend's storage mechanism.

The sketch is serialized to an EXSK v2 binary frame before storage.

Functions

backends()

@spec backends() :: %{required(backend_type()) => module()}

Returns the registry mapping every backend's atom key to its module.

ExDataSketch.Storage.Ecto is only compiled when the optional :ecto_sql dependency is available; its entry is still present in this map (module names are atoms, so referencing one costs nothing), but calling any function on it without :ecto_sql installed raises UndefinedFunctionError.

Examples

iex> ExDataSketch.Storage.backends()[:ets]
ExDataSketch.Storage.ETS

iex> map_size(ExDataSketch.Storage.backends())
5

delete(backend_ref, key)

@spec delete(backend_ref(), key()) :: :ok | {:error, term()}

Removes the value stored at key via backend_ref, dispatching to the resolved backend module's delete/2.

See the "Backend Ref Resolution" section above for how backend_ref is resolved.

Examples

iex> :ets.new(:storage_facade_doctest_delete, [:set, :public, :named_table])
iex> sketch = ExDataSketch.HLL.new(p: 10)
iex> ExDataSketch.Storage.save(sketch, {ExDataSketch.Storage.ETS, :storage_facade_doctest_delete}, "key")
iex> ExDataSketch.Storage.delete({ExDataSketch.Storage.ETS, :storage_facade_doctest_delete}, "key")
:ok
iex> ExDataSketch.Storage.load(ExDataSketch.HLL, {ExDataSketch.Storage.ETS, :storage_facade_doctest_delete}, "key")
{:error, :not_found}
iex> :ets.delete(:storage_facade_doctest_delete)
true

load(sketch_module, backend_ref, key)

@spec load(module(), backend_ref(), key()) ::
  {:ok, struct()} | {:error, :not_found | term()}

Retrieves and deserializes a sketch by key via backend_ref, dispatching to the resolved backend module's load/3.

See the "Backend Ref Resolution" section above for how backend_ref is resolved.

Examples

iex> :ets.new(:storage_facade_doctest_load, [:set, :public, :named_table])
iex> sketch = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.save(sketch, {ExDataSketch.Storage.ETS, :storage_facade_doctest_load}, "key")
iex> {:ok, loaded} = ExDataSketch.Storage.load(ExDataSketch.HLL, {ExDataSketch.Storage.ETS, :storage_facade_doctest_load}, "key")
iex> ExDataSketch.HLL.estimate(loaded) > 0.0
true
iex> ExDataSketch.Storage.load(ExDataSketch.HLL, {ExDataSketch.Storage.ETS, :storage_facade_doctest_load}, "nonexistent")
{:error, :not_found}
iex> :ets.delete(:storage_facade_doctest_load)
true

merge(sketch, backend_ref, key)

@spec merge(struct(), backend_ref(), key()) :: :ok | {:error, term()}

Merges a sketch into the value persisted at key via backend_ref, dispatching to the resolved backend module's merge/3.

See the "Backend Ref Resolution" section above for how backend_ref is resolved.

Examples

iex> :ets.new(:storage_facade_doctest_merge, [:set, :public, :named_table])
iex> a = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.save(a, {ExDataSketch.Storage.ETS, :storage_facade_doctest_merge}, "key")
iex> b = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("b")
iex> ExDataSketch.Storage.merge(b, {ExDataSketch.Storage.ETS, :storage_facade_doctest_merge}, "key")
:ok
iex> {:ok, merged} = ExDataSketch.Storage.load(ExDataSketch.HLL, {ExDataSketch.Storage.ETS, :storage_facade_doctest_merge}, "key")
iex> ExDataSketch.HLL.estimate(merged) >= 1.9
true
iex> :ets.delete(:storage_facade_doctest_merge)
true

save(sketch, backend_ref, key)

@spec save(struct(), backend_ref(), key()) :: :ok | {:error, term()}

Persists a sketch under key via backend_ref, dispatching to the resolved backend module's save/3.

See the "Backend Ref Resolution" section above for how backend_ref is resolved.

Examples

iex> :ets.new(:storage_facade_doctest_save, [:set, :public, :named_table])
iex> sketch = ExDataSketch.HLL.new(p: 10) |> ExDataSketch.HLL.update("a")
iex> ExDataSketch.Storage.save(sketch, {ExDataSketch.Storage.ETS, :storage_facade_doctest_save}, "key")
:ok
iex> :ets.delete(:storage_facade_doctest_save)
true