Settings View Source ActiveMemory

the-typed-attribute-queryable-in-memory-store-for-ets-and-mnesia

The typed, attribute-queryable in-memory store for ETS and Mnesia

overview

Overview

A key/value cache answers one question: "what is the value for this key?" ActiveMemory answers the questions a cache cannot:

# every active session for a user
SessionStore.select(%{user_id: user_id, active?: true})

# admins who have not logged in since the cutoff
StaffStore.select(match(:role == "admin" and :last_login < cutoff))

# atomically claim a one-time token — exactly one concurrent caller wins
TokenStore.withdraw(%{value: submitted_token})

Define a Table with named attributes and you get typed structs you can query by any combination of fields — no cache keys to design, no ETS match specs to hand-write, no database round-trip. The same simple ORM interface runs on ETS or Mnesia, with built-in record expiry (TTL), crash resilience, and atomic take-once reads.

ActiveMemory abstracts the ETS and Mnesia specifics behind a common interface called a Store, or an ActiveRepo when you need multiple tables.

when-to-reach-for-activememory

When to reach for ActiveMemory

You need	Reach for
To cache computed values by key, with eviction policies and hit/miss stats	A cache: Cachex, Nebulex
Durable, relational data	A database with Ecto
Structured records in memory, queried by their attributes	ActiveMemory

The sweet spot is any small-to-medium dataset you would be tempted to put in a database table but want at memory speed: one-time tokens and 2FA codes, sessions, feature flags and config, API keys, reference data. See Potential Use Cases.

example-setup

Example setup

1. Define a Table with attributes.
2. Define a Store or an ActiveRepo with configuration settings or accept the defaults (most applications should be fine with defaults).
3. Add the Store or ActiveRepo to your application supervision tree.

Your app is ready!

Example Table:

defmodule MyApp.People.Person do
  use ActiveMemory.Table,
    options: [index: [:last, :cylon?]]

  attributes do
    field(:email)
    field(:first)
    field(:last)
    field(:hair_color)
    field(:age)
    field(:cylon?)
  end
end

There is also optional auto-generation of uuid

  attributes auto_generate_uuid: true do
    field(:email)
    field(:first)
    field(:last)
    field(:hair_color)
    field(:age)
    field(:cylon?)
  end

Example Mnesia Store (default):

defmodule MyApp.People.Store do
  use ActiveMemory.Store,
    table: MyApp.People.Person
end

Example ETS Store:

defmodule MyApp.People.Store do
  use ActiveMemory.Store,
    table: MyApp.People.Person,
    type: :ets
end

Add the Store to your application supervision tree:

defmodule MyApp.Application do
  # code..
  def start(_type, _args) do
    children = [
      # other children
      MyApp.People.Store,
      # other children
    ]
    # code..
  end
end

Now you have the default Store methods available!

store-api

Store API

• Store.all/0 Get all records stored
• Store.delete/1 Delete the record provided
• Store.delete_all/0 Delete all records stored
• Store.one/1 Get one record matching either an attributes search or match query
• Store.select/1 Get all records matching either an attributes search or match query
• Store.withdraw/1 Atomically get one record matching either an attributes search or match query, delete the record and return it. The find-and-delete is a single atomic operation (:ets.select_delete/2 for ETS, a :mnesia.transaction/1 for Mnesia), so under concurrent access exactly one caller receives {:ok, record} for a given record and any others receive {:error, :not_found}. This makes withdraw/1 safe for take-once workloads such as one time use tokens.
• Store.write/1 Write a record into the memmory table

concurrency

Concurrency

Both a Store and an ActiveRepo are GenServers, but the data functions (all, one, select, write, delete, delete_all, withdraw) are not routed through that process and are not serialized by it. They are ordinary module functions that run in the caller's process and delegate straight to the table's adapter, so reads and writes execute with :ets/:mnesia concurrency — many processes operate in parallel and the single GenServer is not a bottleneck. Only lifecycle and metadata operations (init, state, reload_seeds) actually use the GenServer.

These functions live on the GenServer module purely for organization: it is the single place responsible for how the application talks to its table(s), following the Single Responsibility Principle. See the S.T.O.N.E principles for the broader design philosophy.

query-interface

Query interface

This is where ActiveMemory earns its keep: records are found by their attributes, not by a key you had to design up front. Any field — or any combination of fields — is queryable. There are two query styles.

the-attribute-query-syntax

The Attribute query syntax

Attribute matching allows you to provide a map of any subset of the table's fields to search by.

Store.one(%{uuid: "a users uuid"})
Store.select(%{department: "accounting", admin?: false, active: true})

the-match-query-syntax

The match query syntax

When equality isn't enough, the match macro adds comparisons and boolean logic.

query = match(:department == "sales" or :department == "marketing" and :start_date > last_month)
Store.select(query)

Expressing either of these with a key/value cache would mean maintaining your own secondary indexes by hand; here the table's schema makes every field queryable for free.

seeding

Seeding

When starting a Store there is an option to provide a valid seed file and have the Store auto load seeds contained in the file.

defmodule MyApp.People.Store do
  use ActiveMemory.Store,
    table: MyApp.People.Person,
    seed_file: Path.expand("person_seeds.exs", __DIR__)
end

before-init

Before init

All stores are GenServers and have init functions. While those are abstracted you can still specify methods to run during the init phase of the GenServer startup. Use the before_init keyword and add the methods as tuples with the arguments.

defmodule MyApp.People.Store do
  use ActiveMemory.Store,
    table: MyApp.People.Person,
    before_init: [{:run_me, ["arg1", "arg2", ...]}, {:run_me_too, []}]
end

⚠️ before_init and table recovery

For ETS stores the table is preserved across a store crash/restart by the table heir (see Resilience). On such a recovery seed files are not re-run, but before_init methods always run, including on recovery. If a before_init method writes records with unique or generated keys (for example a uuid), running it again on recovery can create duplicates.

How to handle this is left to the implementer. One option is to make any before_init write follow a "find or create" pattern — check with one/1 before calling write/1 — so the method is idempotent across restarts:

def run_me(args) do
  record = build_record(args)

  case one(%{key: record.key}) do
    {:ok, existing} -> {:ok, existing}
    {:error, :not_found} -> write(record)
  end
end

initial-state

Initial State

All stores are GenServers and thus have a state. The default state is an array as such:

%{started_at: "date time when first started", table_name: MyApp.People.Store}

This default state can be overwritten with a new state structure or values by supplying a method and arguments as a tuple to the keyword initial_state.

defmodule MyApp.People.Store do
  use ActiveMemory.Store,
    table: MyApp.People.Person,
    initial_state: {:initial_state_method, ["arg1", "arg2", ...]}
end

resilience

Resilience

An ETS table is owned by the process that creates it, so if a Store were to crash the table — and all of its data — would normally be destroyed and recreated empty when the supervisor restarts the Store.

ActiveMemory guards against this automatically. The library starts a small, stable process, ActiveMemory.TableHeir, and registers it as the ETS :heir for every table a Store creates. When a Store process terminates, ETS transfers the table to the heir instead of destroying it. When the supervisor restarts the Store, it reclaims the table from the heir with the data intact.

This requires no configuration and no API changes: the heir is started as part of the :active_memory application, and the Store functions behave exactly as before. When the heir is not running, stores fall back to creating a fresh table.

# A store crashes...               the table survives (held by the heir)
# ...the supervisor restarts it...  the store reclaims the table, data intact

A few things to be aware of:

• Seeds are skipped on recovery. A recovered table already holds its data, so a configured seed_file is not re-run. before_init methods, however, always run — see the warning in Before init.
• Mnesia stores are unaffected. Mnesia tables are owned by the Mnesia subsystem rather than the Store process, so they already survive a Store crash; the heir is purely an ETS concern.
• Scope is process crashes, not node restarts. The heir protects against Store crashes and supervisor restarts. It does not protect against a full node/BEAM restart, which clears all ETS regardless. For data that must survive a restart, use a Mnesia store with disc_copies.

expiry-ttl

Expiry (TTL)

Give a Table a ttl (time-to-live, in milliseconds) and its records expire automatically — ideal for the one-time tokens, 2FA codes, magic links and short-lived API keys in Potential Use Cases.

defmodule MyApp.Tokens.Token do
  use ActiveMemory.Table,
    type: :ets,
    ttl: :timer.hours(1)

  attributes do
    field(:token)
    field(:user_id)
  end
end

A ttl adds an expires_at field (appended last, so it never becomes the table key) and stamps it on each write as now + ttl. Expiry is then enforced in two complementary ways:

• Lazy filter on read — one, select, all and withdraw never return an expired record. This is immediate and exact: a record is unreadable the instant it expires.
• Periodic sweep — the owning Store/ActiveRepo deletes expired records on a timer to reclaim memory. The cadence defaults to one minute and is configurable per process:

use ActiveMemory.Store, table: MyApp.Tokens.Token, sweep_interval: :timer.seconds(30)

Notes:

• The sweep only runs when a table declares a ttl; non-TTL tables are untouched and incur zero overhead.
• expires_at is plain data, so it survives Resilience recovery — TTL keeps working after a crash.
• Works the same for :ets and :mnesia, and for both a Store and an ActiveRepo (where each table can have its own ttl).

multiple-tables-with-an-activerepo

Multiple tables with an ActiveRepo

A Store manages a single Table. When you want one supervised entry point over several tables, use an ActiveMemory.ActiveRepo — the multi-table counterpart to a Store. (It is named ActiveRepo rather than Repo so it does not collide with an application's Ecto.Repo.)

defmodule MyApp.ActiveRepo do
  use ActiveMemory.ActiveRepo,
    tables: [
      MyApp.People.Person,
      {MyApp.Dogs.Dog, seed_file: Path.expand("dog_seeds.exs", __DIR__), before_init: [{:warm, []}]}
      ..other Table
    ]
end

Add it to your supervision tree like any other process (children = [MyApp.ActiveRepo]). Tables may freely mix :ets and :mnesia; each call dispatches to the adapter configured on the given table.

activerepo-api

ActiveRepo API

Reads and withdraw take the table module as the first argument; writes and deletes infer the table from the struct:

MyApp.ActiveRepo.write(%Person{...})          # table inferred from the struct
MyApp.ActiveRepo.withdraw(Dog, query)         # reads take the table explicitly
MyApp.ActiveRepo.all(Person)
MyApp.ActiveRepo.one(Dog, %{name: "gem"})
MyApp.ActiveRepo.select(Person, query)
MyApp.ActiveRepo.delete(%Dog{} = dog)
MyApp.ActiveRepo.delete_all(Person)

• ActiveRepo.all/1, ActiveRepo.delete/1, ActiveRepo.delete_all/1, ActiveRepo.one/2, ActiveRepo.select/2, ActiveRepo.withdraw/2, ActiveRepo.write/1
• An operation for a struct or table that is not part of the ActiveRepo returns {:error, :unknown_table}.

per-table-options

Per-table options

Each tables: entry is a table module or a {table, opts} tuple. Per-table seed_file and before_init work exactly as they do for a Store; initial_state is an ActiveRepo-level option (one process, one state). Seeding, the query interface and Resilience all behave the same as for a Store — including the before_init recovery caveat.

installation

Installation

The package can be installed by adding active_memory to your list of dependencies in mix.exs:

def deps do
  [
    {:active_memory, "~> 0.7.0"}
  ]
end

Check out the (documentation)

potential-use-cases

Potential Use Cases

There are many reasons to be leveraging the power of in memory store and the awesome tools of Mnesia and ETS in your Elixir applications.

storing-config-settings-and-application-secrets

Storing config settings and Application secrets

Instead of having hard coded secrets and application settings crowding your config files store them in an in memory table. Provide your application a small UI to support the secrets and settings and you can update while the application is running in a matter of seconds.

one-time-use-tokens

One Time Use Tokens

Perfect for short lived tokens such as password reset tokens, 2FA tokens, magic links (password less login) etc. Store the tokens along with any other needed data into an ActiveMemory.Store to reduce the burden of your database and provide your users a better experience with faster responses. Use Store.withdraw/1 to redeem a token: it atomically fetches and deletes the record, so even under concurrent requests a token can only be redeemed once.

api-keys-for-clients

API Keys for clients

For applications which have a fixed set of API Keys or a relativly small set of API keys (less than a few thousand). Store the keys along with any relevent information into an ActiveMemory.Store to reduce the burden of your database and provide your users a better experience with faster responses.

jwt-encryption-keys

JWT Encryption Keys

Applications using JWT's can store the keys in an ActiveMemory.Store and provide fast access for encrypting JWT's and fast access for publishing the public keys on an endpoint for token verification by consuming clients.

admin-user-management

Admin User Management

Create an ActiveMemory.Store to manage your admins easily and safely.

and many many many more...

demo-application

Demo Application

The following Repo is a demo application using ActiveMemory and MnesiaManager concept.

• BeamDemo
• MnesiaManager

planned-enhancements

Planned Enhancements

• Allow pass through :ets and mnesia options for table creation
• Allow pass through :ets and mnesia syntax for searches
• Mnesia co-ordination with Docker instance for backup and disk persistance
• Enhance match query syntax
  • Select option for certain fields
  • Group results

Any suggestions appreciated.