Pockets (Pockets v1.6.0)

View Source

Pockets is a wrapper around Erlang :ets and :dets, built-in options for memory- and disk-based term storage. It offers simple key/value storage using an interface similar to the Map or Keyword modules. This can be a useful persistent cache for many use cases.

Note that this package and the libraries that underpin it may have limitations or specific behaviors that may affect its suitability for various use-cases. For example, the limited support for concurrency provided by the :ets(3) module is not yet provided by :dets.

See also

Summary

Types

A table alias is used to refer to a Pockets table: it is usually an atom, but in some cases it may be a reference (e.g. in an :ets table with named_table: false).

Functions

Closes the given table (i.e. the opposite of open/3). Only processes that have opened a table are allowed to close it. For :ets in-memory tables, this is the same as destroy/1. For :dets disk-based tables, this will save any outstanding changes to the file.

Deletes the entry in table for a specific key. Because this returns the table alias, you can pipe multiple operations together.

Destroys the given table. For disk-based (:dets) tables, this will delete the backing file. For memory-based (:ets) tables, this destroys the table and its contents.

Checks if the given table is empty. This will return true if the given table does not exist.

Checks if the table exists in the Pockets registry.

Akin to Enum.filter/2, this function will pare down the contents of the given table preserving only entries for which fun returns a truthy value. Similar to using Enum.filter/2 on maps, the fun receives a tuple representing the key and the value stored at that location.

Gets the value for a specific key in the table.

Checks if the table has the given key.

Increments the value at the given key by the given step (default 1). If the table does not exist or the value at the given key is not a number, no action is taken.

Gets info about the given table, returned as either a %Pockets.EtsInfo{} or a %Pockets.DetsInfo{} struct. An :error tuple is returned if the table does not exist.

Gets info about the given item in the table. The available items depend on the type of table. An error tuple is returned if the table does not exist.

Gets a list of keys in the given table. For larger tables, consider using keys_stream/1. An error tuple is returned if the table does not exist.

Gets a list of keys in the given table as a stream. An error tuple is returned if the table does not exist.

This is a powerful function that lets you merge input into an open table. All data in the input will be added to the table: the keys in the input have precedence over pre-existing keys in the table.

Creates a new table either in memory (default) or on disk.

Open a table for use. The behavior and available options of this function vary depending on the storage mechanism of the table (i.e.memory or disk).

Puts the given value under key in given table.

Akin to Enum.reject/2, this function will pare down the contents of the given table deleting entries for which fun returns a truthy value. Similar to using Enum.filter/2 on maps, the fun receives a tuple representing the key and the value stored at that location.

Both :ets and :dets files can be saved to disk. You can use this function to persist an in-memory :ets file to disk for later use, or you can use it to make a copy of an existing :dets table.

Show all registered Pockets tables.

Returns the size of the given table, measured by the number of entries. Zero is returned if the table does not exist.

Outputs the contents of the given table to a list. If the table does not exist, an empty list is returned. Although this is useful for debugging purposes, for larger data sets consider using to_stream/1 instead.

Outputs the contents of the table to a map. If the table does not exist, an empty map is returned. Although this is useful for debugging purposes, for larger data sets consider using to_stream/1 instead.

Outputs the contents of the table to a stream for lazy evaluation. Returns an error tuple if the table does not exist.

Truncates the given table; this removes all entries from the table while leaving its options intact.

Types

alias()

@type alias() :: atom() | reference()

A table alias is used to refer to a Pockets table: it is usually an atom, but in some cases it may be a reference (e.g. in an :ets table with named_table: false).

Functions

close(table_alias)

@spec close(table_alias :: alias()) :: :ok | {:error, any()}

Closes the given table (i.e. the opposite of open/3). Only processes that have opened a table are allowed to close it. For :ets in-memory tables, this is the same as destroy/1. For :dets disk-based tables, this will save any outstanding changes to the file.

delete(table_alias, key)

@spec delete(table_alias :: alias(), any()) :: alias() | {:error, any()}

Deletes the entry in table for a specific key. Because this returns the table alias, you can pipe multiple operations together.

Examples

iex> Pockets.new(:my_cache)
{:ok, :my_cache}
iex> Pockets.merge(:my_cache, %{a: "apple", b: "boy", c: "cat"})
:my_cache
iex> Pockets.to_map(:my_cache)
%{a: "apple", b: "boy", c: "cat"}
iex> Pockets.delete(:my_cache, :b)
:my_cache
iex> Pockets.to_map(:my_cache)
%{a: "apple", c: "cat"}

destroy(table_alias)

@spec destroy(table_alias :: alias()) :: :ok | {:error, any()}

Destroys the given table. For disk-based (:dets) tables, this will delete the backing file. For memory-based (:ets) tables, this destroys the table and its contents.

Examples

iex> Pockets.new(:my_cache, "/tmp/cache.dets")
{:ok, :my_cache}
iex> Pockets.destroy(:my_cache)
:ok

empty?(table_alias)

@spec empty?(table_alias :: alias()) :: boolean()

Checks if the given table is empty. This will return true if the given table does not exist.

Examples

iex> {:ok, tid} = Pockets.new(:my_cache)
{:ok, :my_cache}
iex> Pockets.empty?(tid)
true

exists?(table_alias)

(since 1.1.0)
@spec exists?(table_alias :: alias()) :: boolean()

Checks if the table exists in the Pockets registry.

Note: this function does not check whether an ETS or DETS table was created outside of Pockets (e.g. via ETS or DETS directly). For those cases, see :ets.whereis/1 or :dets.info/1.

filter(table_alias, fun)

(since 1.3.0)
@spec filter(table_alias :: alias(), (any() -> as_boolean(term()))) ::
  {:ok, non_neg_integer()} | {:error, String.t()}

Akin to Enum.filter/2, this function will pare down the contents of the given table preserving only entries for which fun returns a truthy value. Similar to using Enum.filter/2 on maps, the fun receives a tuple representing the key and the value stored at that location.

The result of a successful operation is an :ok tuple including the count of rows removed.

This Updates the Table In Place!

This operation updates the table in place, so it has the potential to delete every item in the table. Use it carefully!

If a safer alternative is needed, use Pockets.to_stream/1 to manipulate values and feed the results into a new table.

Examples

iex> Pockets.new(:ex)
iex> Pockets.merge(:ex, %{a: 12, b: 7, c: 22, d: 8})
iex> Pockets.filter(:ex, fn {_, v} -> v > 10 end)
{:ok, 2}
iex> Pockets.to_map(:ex)
%{a: 12, c: 22}

See Also

get(table_alias, key, default \\ nil)

@spec get(table_alias :: alias(), any(), any()) :: any() | :error

Gets the value for a specific key in the table.

For tables of type :set, this function behaves like Map.get/3: the value is returned (if present), otherwise the default is returned.

However, for tables of type :bag and :duplicate_bag, this function will always return a list and the default is ignored. This has to do with keyword lists being the underlying data structure powering all of this.

The default value will be returned if the table does not exist.

Examples

iex> Pockets.new(:t1, :memory, type: :set)
{:ok, :t1}
iex> Pockets.get(:t1, :x, "Default Value")
"Default Value"

iex> Pockets.new(:t2, :memory, type: :bag)
{:ok, :t2}
iex> Pockets.get(:t2, :x, "Ignored Default Value")
[]

# Bag
iex> Pockets.new(:t_bag, :memory, type: :bag)
{:ok, :t_bag}
iex> Pockets.put(:t_bag, :c, "Cream")
:t_bag
iex> Pockets.put(:t_bag, :c, "Sugar")
:t_bag
iex> Pockets.put(:t_bag, :c, "Sugar")
:t_bag
iex> Pockets.get(:t_bag, :c)
["Cream", "Sugar"]

# Duplicate Bag
iex> Pockets.new(:t_dupe_bag, :memory, type: :duplicate_bag)
{:ok, :t_dupe_bag}
iex> Pockets.put(:t_dupe_bag, :c, "Cream")
:t_bag
iex> Pockets.put(:t_dupe_bag, :c, "Sugar")
:t_bag
iex> Pockets.put(:t_dupe_bag, :c, "Sugar")
:t_bag
iex> Pockets.get(:t_bag, :c)
["Cream", "Sugar", "Sugar"]

has_key?(table_alias, key)

@spec has_key?(table_alias :: alias(), any()) :: boolean()

Checks if the table has the given key.

false is returned if the table does not exist.

incr(table_alias, key, step \\ 1, initial_value \\ 0)

(since 1.2.0)
@spec incr(alias(), key :: any(), step :: integer(), initial_value :: number()) ::
  alias() | {:error, String.t()}

Increments the value at the given key by the given step (default 1). If the table does not exist or the value at the given key is not a number, no action is taken.

Examples

iex> Pockets.new(:inc_table)
  :inc_table
  |> Pockets.incr(:x)
  |> Pockets.incr(:x)
  |> Pockets.get(:x)
2

Decrement a value:

iex> Pockets.new(:decr_table)
  :decr_table
  |> Pockets.incr(:x, -1, 10)
  |> Pockets.incr(:x, -1)
  |> Pockets.get(:x)
8

info(table_alias)

@spec info(table_alias :: alias()) ::
  Pockets.EtsInfo.t() | Pockets.DetsInfo.t() | {:error, String.t()}

Gets info about the given table, returned as either a %Pockets.EtsInfo{} or a %Pockets.DetsInfo{} struct. An :error tuple is returned if the table does not exist.

info(table_alias, item)

@spec info(table_alias :: alias(), atom()) :: any() | {:error, String.t()}

Gets info about the given item in the table. The available items depend on the type of table. An error tuple is returned if the table does not exist.

is_alias(value)

(macro)

keys(table_alias)

@spec keys(table_alias :: alias()) :: list() | {:error, String.t()}

Gets a list of keys in the given table. For larger tables, consider using keys_stream/1. An error tuple is returned if the table does not exist.

keys_stream(table_alias)

Gets a list of keys in the given table as a stream. An error tuple is returned if the table does not exist.

merge(table_alias1, table_alias2)

@spec merge(alias(), input :: alias() | list() | map()) :: alias()

This is a powerful function that lets you merge input into an open table. All data in the input will be added to the table: the keys in the input have precedence over pre-existing keys in the table.

When the input to be merged is...

  • an alias for another table, the contents from that table are added to the given table_alias
  • a map, the contents from the map are added into the given table_alias
  • a keyword list, the contents from the list are added into the given table_alias

Examples

# Merging a map into a table:
iex> Pockets.new(:my_cache)
{:ok, :my_cache}
iex> Pockets.merge(:my_cache, %{a: "apple", b: "boy", c: "cat"})
:my_cache
iex> Pockets.to_map(:my_cache)
%{a: "apple", b: "boy", c: "cat"}

# Merging a list into a table
iex> Pockets.new(:my_cache)
{:ok, :my_cache}
iex> Pockets.merge(:my_cache, [a: "apple", b: "boy", c: "cat"])
:my_cache
iex> Pockets.to_map(:my_cache)
%{a: "apple", b: "boy", c: "cat"}


# Merging two tables:
iex> Pockets.new(:my_first)
{:ok, :my_first}
iex> Pockets.merge(:my_first, %{a: "apple", b: "boy", c: "cat"})
:my_first
iex> Pockets.new(:my_second)
{:ok, :my_second}
iex> Pockets.merge(:my_second, %{x: "xray", y: "yellow", z: "zebra"})
:my_second
iex> Pockets.merge(:my_first, :my_second)
:my_first
iex> Pockets.to_map(:my_first)
%{a: "apple", b: "boy", c: "cat", x: "xray", y: "yellow", z: "zebra"}

new(table_alias, storage \\ :memory, opts \\ [type: :set])

@spec new(table_alias :: alias(), :memory | (file :: String.t()), opts :: keyword()) ::
  {:ok, alias()} | {:error, any()}

Creates a new table either in memory (default) or on disk.

The second argument specifies the storage mechanism for the table, either a path to a file (as a string) for disk-backed tables (:dets), or in :memory for memory-backed tables (:ets).

The available opts are specific to the storage engine being used (i.e. to ets or dets).

In Memory

When creating a new memory-based table (i.e. the second argument is :memory or is omitted), the following options are supported:

  • :type One of [:bag, :duplicate_bag, :set] Default: set
  • :access One of :public | :protected | :private. Default: :public

  • :named_table boolean affects how the table is referenced. If false, the table alias will be a reference (not an atom). Default: true
  • :keypos integer indicating the position of the element of each object to be used as key. Default: 1
  • :read_concurrency boolean. Default: true
  • :write_concurrency boolean. Default: true
  • :decentralized_counters boolean. Default: false
  • :compressed boolean. Default: false

The full list of default options for memory-based tables are:

[type: :set, access: :public, named_table: true, keypos: 1, read_concurrency: true, write_concurrency: true, decentralized_counters: false, compressed: false]

The structure of the options is changed from the original to make them more compatible with Elixir conventions. See the original documentation for specifics about each option.

File-based Tables

When creating a new disk-based table (i.e. the second argument is a string path to a file), the following options are supported:

  • :type One of [:bag, :duplicate_bag, :set] Default: set
  • :access One of :read | :read_write. Default: :read_write

  • :auto_save integer or :infinity specifying the autosave interval (where :infinity disables the feature). Default: 180000 (3 minutes)
  • :keypos integer indicating the position of the element of each object to be used as key. Default: 1
  • :file The name of the file to be opened. Defaults to the table name. (Use of this confusing option is not recommended)
  • :max_no_slots The maximum number of slots to be used. Default: 32 M (million?)
  • :min_no_slots Application performance can be enhanced with this flag by specifying the estimated number of different keys to be stored in the table. Default: 256 (minimum)
  • :ram_file boolean. Whether the table is to be kept in RAM. Keeping the table in RAM can sound like an anomaly, but it can enhance the performance of applications that open a table, insert a set of objects, and then close the table. When the table is closed, its contents are written to the disk file. Default: false
  • :repair boolean or :force. The flag specifies if the :dets server invokes the automatic file reparation algorithm. Default: true

The default options for disk-based tables are:

[type: :set]

Examples

iex> Pockets.new(:ram_cache, :memory)
{:ok, :ram_cache}

iex> Pockets.new(:disk_cache, "/tmp/my.dets")
{:ok, :disk_cache}

open(table_alias, storage \\ :memory, opts \\ [type: :set])

@spec open(alias(), :memory | binary(), opts :: keyword()) :: any()

Open a table for use. The behavior and available options of this function vary depending on the storage mechanism of the table (i.e.memory or disk).

Memory-based Tables

For memory-based tables open/3 is synonymous with new/3.

Disk-based Tables

Because a file is involved, the :create? option provides a bit more control over how to handle cases when the file does not exist.

Options

  • :create? Indicates whether a new table should be created if the one being requested does not already exist. Default: false

If :create? is true and the filepath indicated by the second argument does not exist, open/3 is synonymous with new/3.

Examples

iex> Pockets.open(:my_cache)
{:ok, :my_cache}
iex> Pockets.open(:my_cache, :memory)
{:ok, :my_cache}
iex> Pockets.open(:disk_cache, "/tmp/cache.dets")
{:ok, :disk_cache}
iex> Pockets.open(:boo, "/tmp/does_not_exist_yet.dets", create?: true)
{:ok, :boo}

put(table_alias, key, value)

@spec put(table_alias :: alias(), any(), any()) :: alias() | {:error, String.t()}

Puts the given value under key in given table.

For tables of type :set, this will behave like Map.put/3.

Other table types aren't as straightforward. See the examples under get/3

Examples

# Set
iex> Pockets.new(:t_set)
{:ok, :t_set}
iex> Pockets.put(:t_set, :z, "Zulu")
:t_set
iex> Pockets.get(:t_set, :z)
"Zulu"

reject(table_alias, fun)

(since 1.3.0)
@spec reject(table_alias :: alias(), (any() -> as_boolean(term()))) ::
  {:ok, non_neg_integer()} | {:error, String.t()}

Akin to Enum.reject/2, this function will pare down the contents of the given table deleting entries for which fun returns a truthy value. Similar to using Enum.filter/2 on maps, the fun receives a tuple representing the key and the value stored at that location.

The result of a successful operation is an :ok tuple including the count of rows removed.

This Updates the Table In Place!

This operation updates the table in place, so it has the potential to delete every item in the table. Use it carefully!

If a safer alternative is needed, use Pockets.to_stream/1 to manipulate values and feed the results into a new table.

Examples

iex> Pockets.new(:ex)
iex> Pockets.merge(:ex, %{a: 12, b: 7, c: 22, d: 8})
iex> Pockets.reject(:ex, fn {_, v} -> v > 10 end)
{:ok, 2}
iex> Pockets.to_map(:ex)
%{b: 7, d: 8}

See Also

save_as(table_alias, target_file, opts \\ [])

@spec save_as(table_alias :: alias(), binary(), keyword()) :: :ok | {:error, any()}

Both :ets and :dets files can be saved to disk. You can use this function to persist an in-memory :ets file to disk for later use, or you can use it to make a copy of an existing :dets table.

The target file must not be in use by another table; if the target file exists this will return an error unless the :overwrite? option is set to true.

Options:

  • overwrite? default: false

show_tables(opts \\ [])

Show all registered Pockets tables.

Examples

iex> Pockets.show_tables()
[%Pockets.Table{alias: :my_cache, library: :ets, tid: :my_cache, type: :set}]

size(table_alias)

@spec size(table_alias :: alias()) :: integer()

Returns the size of the given table, measured by the number of entries. Zero is returned if the table does not exist.

to_list(table_alias)

@spec to_list(table_alias :: alias()) :: list()

Outputs the contents of the given table to a list. If the table does not exist, an empty list is returned. Although this is useful for debugging purposes, for larger data sets consider using to_stream/1 instead.

to_map(table_alias)

@spec to_map(table_alias :: alias()) :: map()

Outputs the contents of the table to a map. If the table does not exist, an empty map is returned. Although this is useful for debugging purposes, for larger data sets consider using to_stream/1 instead.

to_stream(table_alias)

Outputs the contents of the table to a stream for lazy evaluation. Returns an error tuple if the table does not exist.

truncate(table_alias)

@spec truncate(table_alias :: alias()) :: alias() | {:error, any()}

Truncates the given table; this removes all entries from the table while leaving its options intact.

Examples

iex> Pockets.put(:my_cache, :a, "Apple") |> Pockets.put(:b, "boy") |> Pockets.put(:c, "Charlie")
:my_cache
iex> Pockets.truncate(:my_cache)
:my_cache
iex> Pockets.to_map(:my_cache)
%{}