defmodule ActiveMemory.Table do @moduledoc """ Define your table attributes and options. Example Table (without auto generated uuid): ```elixir defmodule Test.Support.People.Person do use ActiveMemory.Table, options: [index: [:last, :cylon?]] attributes do field :email field :first field :last field :hair_color field :age field :cylon? end end ``` ### Auto Generated UUID A table can have an auto generated UUID. Specify the option `auto_generate_uuid: true` in the attributes as an option. Example Table with auto generated uuid: ```elixir defmodule Test.Support.People.Person do use ActiveMemory.Table, options: [index: [:last, :cylon?]] attributes auto_generate_uuid: true do field :email field :first field :last field :hair_color field :age field :cylon? end end ``` ## Field types and Ecto Changesets Fields accept an optional [Ecto type](https://hexdocs.pm/ecto/Ecto.Schema.html#module-types-and-casting) as the second argument (defaulting to `:any` when omitted): ```elixir defmodule MyApp.Planet do use ActiveMemory.Table, type: :ets attributes auto_generate_uuid: true do field :name, :string field :gravity, :float field :moons, :integer, default: 0 end end ``` Types are not enforced on `write/1` — ETS and Mnesia store any term — they exist to power `Ecto.Changeset` casting and validation, which works directly on the table struct. `write/1` accepts the changeset itself, the way `c:Ecto.Repo.insert/2` does: ```elixir {:ok, planet} = %MyApp.Planet{} |> Ecto.Changeset.cast(params, [:name, :gravity, :moons]) |> Ecto.Changeset.validate_required([:name]) |> MyApp.Planet.Store.write() ``` An invalid changeset returns `{:error, changeset}` with its `action` set to `:insert`, so a Phoenix form renders the errors without any extra work. The declared types are available as `__attributes__(:types)`. ## Using an Ecto schema instead of attributes A table can skip the `attributes` block entirely and define an Ecto `embedded_schema`. All table metadata is derived from the schema, and the module is a real Ecto schema, so every changeset function works out of the box: ```elixir defmodule MyApp.Comet do use ActiveMemory.Table, type: :ets use Ecto.Schema embedded_schema do field :name, :string field :orbit_years, :integer end end ``` Notes for Ecto schema tables: - Autogenerated fields are honored: `write/1` fills any field the schema declares as autogenerated when it is still `nil`, leaving a value you set yourself alone. That covers the default `embedded_schema` primary key (`{:id, :binary_id, autogenerate: true}`), an explicit uuid key such as `@primary_key {:uuid, Ecto.UUID, autogenerate: true}`, and `timestamps()`. - An autogenerated **integer** primary key (`{:id, :id, autogenerate: true}`) cannot be generated in memory and raises when the table is created. Use `Ecto.UUID`/`:binary_id`, or `@primary_key false` and assign the key yourself. - With `@primary_key false` the first declared field is the table key. - Virtual fields are never stored; they reset to their defaults on read. - `timestamps()` are stamped on the first write that leaves them `nil`, but they are not refreshed on a later write — `write/1` is an upsert, not an update. - A table with a `ttl` must declare its own expiry field: `field :expires_at, :integer` (milliseconds since epoch, stamped on write). ## Options when creating tables `ActiveMemory.Table` support almost all of the same options as `:ets` and `:mneisia`. Please be aware that the options are different for `:ets` and `:mneisia`. Further reading can be found with [ETS docs](https://www.erlang.org/doc/man/ets.html) and [Mnesia docs](https://www.erlang.org/doc/man/mnesia.html). All options should be structured as a [Keyword list](https://hexdocs.pm/elixir/1.12/Keyword.html). Example: ```elixir use ActiveMemory.Table, type: :ets, options: [compressed: true, read_concurrency: true, type: :protected] ``` ### Record Expiry (`ttl`) Pass a `ttl` (time-to-live, in milliseconds) to give every record in the table a lifetime. Expiry is tracked in an `expires_at` field holding milliseconds since the epoch, stamped on each write as `now + ttl`. ```elixir use ActiveMemory.Table, type: :ets, ttl: :timer.hours(1) ``` In an `attributes` block the field is added for you, appended last so it never becomes the table key. **An Ecto schema table must declare it explicitly**, because ActiveMemory does not add fields to a schema you wrote — the schema stays the single description of the record, exactly as Ecto treats it: ```elixir defmodule MyApp.Token do use ActiveMemory.Table, type: :ets, ttl: :timer.minutes(15) use Ecto.Schema @primary_key {:uuid, Ecto.UUID, autogenerate: true} embedded_schema do field :value, :string field :expires_at, :integer end end ``` Declare it last, so it does not take the key position, and type it `:integer`. A `ttl` table without an `expires_at` field would silently never expire, so the table raises when it is created instead. Expiry is enforced in two complementary ways (see `ActiveMemory.Store` and `ActiveMemory.ActiveRepo`): reads never return an expired record, and the owning `Store`/`ActiveRepo` periodically sweeps expired records to reclaim memory. This makes a `ttl` table well suited to one time use tokens, 2FA codes, magic links and similar short-lived data. ### Mnesia Options #### Table Read and Write Access Mnesia tables can be set to `read_only` or `read_write`. The default is `read_write`. Read only tables updates cannot be performed. if you need to change the access use the following syntax: `[access_mode: :read_only]` #### Table Types Tables can be either a `:set`, `:ordered_set`, or a `:bag`. The default is `:set` if you need to change the type use the following syntax: `[type: :bag]` #### Disk Copies A list of nodes can be specified to maintain disk copies of the table. Nodes specified will recieve a replica of the table. Disk copy talbes still maintain a ram copy of the table as well. By default all tables are `ram_copies` and no `disc_copies` are specified. if you need to specify nodes use following syntax: `[disc_copies: [node1, node2, node3, ...]]` #### Disk Only Copies A list of nodes can be specified to maintain only disk copies. A disc only table replica is kept on disc only and unlike the other replica types, the contents of the replica do not reside in RAM. These replicas are considerably slower than replicas held in RAM. if you need to specify nodes use following syntax: `[disc_only_copies: [node1, node2, node3, ...]]` #### Ram Copies A list of nodes can be specified to maintain ram copies of the table. Nodes specified will recieve a replica of the table. By default all tables are set to ram_copies: `[ram_copies: [node()]]` if you need to specify nodes use following syntax: `[ram_copies: [node1, node2, node3, ...]]` #### Indexes If Indexes are desired specify an atom attribute list for which Mnesia is to build and maintain an extra index table. The qlc query compiler may be able to optimize queries if there are indexes available. To specify Indexes use the following syntax: `[index: [:age, :hair_color, :cylon?]]` #### Table Load Order The load order priority is by default 0 (zero) but can be set to any integer. The tables with the highest load order priority are loaded first at startup. If you need to change the load order use the following syntax: `[load_order: 2]` #### Majority (quorum writes, and surviving a network partition) `[majority: true]` requires a **majority of a table's replicas to be reachable before any transactional write commits**. A write on the minority side of a network partition is aborted rather than accepted, which is the single most effective thing you can do about Mnesia's partition behavior. It defaults to `false`. ```elixir use ActiveMemory.Table, options: [ majority: true, ram_copies: [:"node1@host", :"node2@host", :"node3@host"] ] ``` ##### Why it matters With the default `majority: false`, each side of a partition keeps accepting writes against its own replicas. Mnesia does not merge conflicting histories, and it does not stop you from creating them. When the nodes reconnect and each has logged the other as down, Mnesia emits an `{inconsistent_database, running_partitioned_network, node}` system event — and the default handler **logs an error and carries on**, serving whichever replica a given node reads from. That is not carelessness: there is no correct automatic merge without knowing what the data means. But it does mean divergence is not loud, and recovery is operator work — choose an authoritative replica with `:mnesia.set_master_nodes/1,2` and restart the nodes that should resynchronise from it, or restore from a backup. It is the reason many teams give up on Mnesia. With `majority: true` the minority side refuses writes for that table, so there is much less to reconcile — the same trade CP systems make, and roughly what Mnesia's `pause_minority` strategy does at the node level. ##### What it costs - Writes on the minority side fail. Availability is traded for consistency. - It needs an odd number of replicas to be useful; with two replicas neither side of a split holds a majority, so writes stop on both. - It gates **updates**, not reads, and dirty operations bypass it entirely. ActiveMemory's Mnesia reads run inside a transaction but commit nothing, so they still succeed on the minority side and return that replica's contents, which may be behind the majority's. - It is per table, so a table can opt in without changing the rest. ##### If you need more than this Quorum writes reduce divergence; they do not make Mnesia a partition tolerant database. If your data genuinely cannot tolerate a partition, the options are to keep the system of record in a database and treat the ActiveMemory table as derived, or to reach for a consensus backed store such as [Khepri](https://hexdocs.pm/khepri) when the data model suits a leader and quorum. Khepri is not a drop-in for arbitrary Mnesia tables — it is a tree structured store built for strongly consistent state, and the RabbitMQ team adopted it for their metadata rather than as a general replacement. For a single node application none of this applies: the only replica is always a majority, so `majority: true` adds no availability constraint. ### ETS Options #### Table Access Access options are: `:public` `:protected` or `:private`. The default access is `:public` if you need to change the access use the following syntax: `[access: :private]` #### Table Types Tables can be either a `:set`, `:ordered_set`, `:bag`, or a `:duplicate_bag`. The default is `:set` if you need to change the type use the following syntax: `[type: :bag]` #### Compression Compression can be used to help shrink the size of the memory the data consumes, however this does mean the access is slower. The default is `false` where no compression happens. if you need to change the compression use the following syntax: `[Compression: true]` #### Read Concurrency From ETS documentation: Performance tuning. Defaults to `false`. When set to true, the table is optimized for concurrent read operations. When this option is enabled read operations become much cheaper; especially on systems with multiple physical processors. However, switching between read and write operations becomes more expensive. You typically want to enable this option when concurrent read operations are much more frequent than write operations, or when concurrent reads and writes comes in large read and write bursts (that is, many reads not interrupted by writes, and many writes not interrupted by reads). You typically do not want to enable this option when the common access pattern is a few read operations interleaved with a few write operations repeatedly. In this case, you would get a performance degradation by enabling this option. Option read_concurrency can be combined with option write_concurrency. You typically want to combine these when large concurrent read bursts and large concurrent write bursts are common. if you need to change the read_concurrency use the following syntax: `[read_concurrency: true]` #### Write Concurrency From ETS documentation: Performance tuning. Defaults to `false`, in which case an operation that mutates (writes to) the table obtains exclusive access, blocking any concurrent access of the same table until finished. If set to true, the table is optimized for concurrent write access. Different objects of the same table can be mutated (and read) by concurrent processes. This is achieved to some degree at the expense of memory consumption and the performance of sequential access and concurrent reading. The auto alternative for the write_concurrency option is similar to the true option but automatically adjusts the synchronization granularity during runtime depending on how the table is used. This is the recommended write_concurrency option when using Erlang/OTP 25 and above as it performs well in most scenarios. The write_concurrency option can be combined with the options read_concurrency and decentralized_counters. You typically want to combine write_concurrency with read_concurrency when large concurrent read bursts and large concurrent write bursts are common; for more information, see option read_concurrency. It is almost always a good idea to combine the write_concurrency option with the decentralized_counters option. Notice that this option does not change any guarantees about atomicity and isolation. Functions that makes such promises over many objects (like insert/2) gain less (or nothing) from this option. The memory consumption inflicted by both write_concurrency and read_concurrency is a constant overhead per table for set, bag and duplicate_bag when the true alternative for the write_concurrency option is not used. For all tables with the auto alternative and ordered_set tables with true alternative the memory overhead depends on the amount of actual detected concurrency during runtime. The memory overhead can be especially large when both write_concurrency and read_concurrency are combined. if you need to change the write_concurrency use the following syntax: `[write_concurrency: true]` or `[write_concurrency: :auto]` #### Decentralized Counters From ETS documentation: Performance tuning. Defaults to true for all tables with the write_concurrency option set to auto. For tables of type ordered_set the option also defaults to true when the write_concurrency option is set to true. The option defaults to false for all other configurations. This option has no effect if the write_concurrency option is set to false. When this option is set to true, the table is optimized for frequent concurrent calls to operations that modify the tables size and/or its memory consumption (e.g., insert/2 and delete/2). The drawback is that calls to info/1 and info/2 with size or memory as the second argument can get much slower when the decentralized_counters option is turned on. When this option is enabled the counters for the table size and memory consumption are distributed over several cache lines and the scheduling threads are mapped to one of those cache lines. The erl option +dcg can be used to control the number of cache lines that the counters are distributed over. if you need to change the decentralized_counters use the following syntax: `[decentralized_counters: true]` """ alias ActiveMemory.Adapters.Helpers defmacro __using__(opts) do quote do import ActiveMemory.Table, only: [attributes: 1, attributes: 2] @before_compile ActiveMemory.Table Module.register_attribute(__MODULE__, :active_memory_fields, accumulate: true) Module.register_attribute(__MODULE__, :active_memory_query_fields, accumulate: true) Module.register_attribute(__MODULE__, :active_memory_field_sources, accumulate: true) Module.register_attribute(__MODULE__, :active_memory_types, accumulate: true) opts = unquote(Macro.expand(opts, __CALLER__)) table_type = Keyword.get(opts, :type, :mnesia) table_options = Keyword.get(opts, :options, :defaults) Module.put_attribute(__MODULE__, :adapter, Helpers.set_adapter(table_type)) Module.put_attribute( __MODULE__, :table_options, Helpers.build_options(table_options, table_type) ) Module.put_attribute(__MODULE__, :ttl, Keyword.get(opts, :ttl, nil)) end end defmacro attributes(opts \\ [], do: block) do define_attributes(opts, block) end defmacro field(name, type \\ :any, opts \\ []) do quote do ActiveMemory.Table.__field__( __MODULE__, unquote(name), unquote(type), unquote(opts) ) end end @doc false def __after_compile__(%{module: _module}, _) do :ok end @doc false defmacro __before_compile__(env) do cond do Module.defines?(env.module, {:__attributes__, 1}) -> :ok Module.defines?(env.module, {:__schema__, 1}) -> define_ecto_schema_attributes(env.module) true -> raise ArgumentError, "#{inspect(env.module)} uses ActiveMemory.Table but defines no fields. " <> "Define them with an `attributes do ... end` block or with an Ecto schema " <> "(`embedded_schema do ... end`)." end end @doc false def __attributes__(fields, field_sources) do load = for name <- fields do if alias = field_sources[name] do {name, {:source, alias}} else name end end dump = for name <- fields do {name, field_sources[name] || name} end field_sources_quoted = for name <- fields do {[:field_source, name], field_sources[name] || name} end single_arg = [ {[:dump], dump |> Map.new() |> Macro.escape()}, {[:load], load |> Macro.escape()} ] catch_all = [ {[:field_source, quote(do: _)], nil} ] [ single_arg, field_sources_quoted, catch_all ] end @doc false # Ecto splits autogeneration in two: `:autogenerate_id` holds an `:id`/`:binary_id` # primary key, while `:autogenerate` holds every other autogenerating field (a # custom type such as `Ecto.UUID`, and `timestamps()`). Both are honored, as the # specs Ecto itself uses: `{fields, {module, function, args}}`. # ETS and Mnesia key a record on its first field, so that field is the primary # key regardless of what an Ecto schema declares. A declared key anywhere else # would silently read the wrong field, and a composite key cannot be expressed # at all, so both are rejected. def __primary_key__(module, declared, fields) do first = hd(fields) case declared do [] -> first [^first] -> first [elsewhere] -> raise ArgumentError, "#{inspect(module)} declares #{inspect(elsewhere)} as its primary key but " <> "#{inspect(first)} is the first field, and the first field is the table key. " <> "Declare the primary key first, or use `@primary_key false`." [_ | _] = composite -> raise ArgumentError, "#{inspect(module)} declares a composite primary key #{inspect(composite)}, " <> "which an in memory table cannot express. Use a single key field." end end @doc false def __autogenerate__(nil, autogenerate), do: autogenerate def __autogenerate__({field, _source, type}, autogenerate) when type in [:binary_id, Ecto.UUID] do [{[field], {Ecto.UUID, :autogenerate, []}} | autogenerate] end def __autogenerate__({field, _source, type}, _autogenerate) do raise ArgumentError, "cannot autogenerate the #{inspect(field)} primary key of type #{inspect(type)} " <> "in memory. Use `Ecto.UUID` or `:binary_id` for a generated key, or set " <> "`@primary_key false` and assign the value yourself." end @doc false def __field__(mod, name, type, opts) do {type, opts} = normalize_field_args(type, opts) validate_type!(name, type) define_field(mod, name, type, opts) end defp define_field(mod, name, type, opts) do put_struct_field(mod, name, Keyword.get(opts, :default)) Module.put_attribute(mod, :active_memory_types, {name, type}) Module.put_attribute(mod, :active_memory_query_fields, name) Module.put_attribute(mod, :active_memory_fields, name) end # `field :name, default: "x"` predates types: a keyword list in the type # position is the options list of an untyped field. defp normalize_field_args(opts, []) when is_list(opts), do: {:any, opts} defp normalize_field_args(type, opts), do: {type, opts} defp validate_type!(name, type) when not is_atom(type) and not is_tuple(type) do raise ArgumentError, "invalid type #{inspect(type)} for field #{inspect(name)}. " <> "Use an Ecto type such as :string, :integer, {:array, :string} or a custom type module." end defp validate_type!(_name, _type), do: :ok defp define_ecto_schema_attributes(module) do adapter = Module.get_attribute(module, :adapter) table_options = Module.get_attribute(module, :table_options) ttl = Module.get_attribute(module, :ttl) quote do def __attributes__(:adapter), do: unquote(adapter) def __attributes__(:auto_generate_uuid), do: false # Mirrors `auto_generate_uuid` for Ecto schema tables: every field the schema # declares as autogenerated is populated on write when it is still `nil`. def __attributes__(:autogenerate) do ActiveMemory.Table.__autogenerate__( __schema__(:autogenerate_id), __schema__(:autogenerate) ) end def __attributes__(:match_head) do ActiveMemory.Adapters.Helpers.build_match_head( __attributes__(:query_map), __MODULE__, unquote(adapter) ) end # The table key is the first stored field, so a schema whose declared primary # key sits elsewhere would make `get/1` read the wrong field. def __attributes__(:primary_key), do: ActiveMemory.Table.__primary_key__( __MODULE__, __schema__(:primary_key), __schema__(:fields) ) def __attributes__(:query_fields), do: __schema__(:fields) def __attributes__(:query_map), do: ActiveMemory.Adapters.Helpers.build_query_map(__schema__(:fields)) def __attributes__(:table_options), do: unquote(Macro.escape(table_options)) def __attributes__(:ttl), do: unquote(ttl) def __attributes__(:types), do: Map.new(__schema__(:fields), fn field -> {field, __schema__(:type, field)} end) end end defp define_attributes(options, block) do prelude = quote do opts = unquote(options) @after_compile ActiveMemory.Table auto_generate_uuid = Keyword.get(opts, :auto_generate_uuid, false) Module.put_attribute(__MODULE__, :auto_generate_uuid, auto_generate_uuid) Module.register_attribute(__MODULE__, :active_memory_struct_fields, accumulate: true) if auto_generate_uuid do ActiveMemory.Table.__field__( __MODULE__, :uuid, Ecto.UUID, primary_key: true, autogenerate: true ) end try do import ActiveMemory.Table unquote(block) after :ok end # When a `ttl` is configured the `expires_at` field is appended last so it # never displaces the table key (the first query field, or the uuid). if Module.get_attribute(__MODULE__, :ttl) do ActiveMemory.Table.__field__(__MODULE__, :expires_at, :integer, []) end end postlude = quote unquote: false do fields = @active_memory_fields |> Enum.reverse() active_memory_query_fields = @active_memory_query_fields |> Enum.reverse() field_sources = @active_memory_field_sources |> Enum.reverse() query_fields = Enum.map(active_memory_query_fields, & &1) query_map = Helpers.build_query_map(query_fields) types = Map.new(@active_memory_types) # A `uuid` attribute is generated on write, whether it came from # `auto_generate_uuid: true` or was declared by hand. autogenerate = if Enum.member?(fields, :uuid), do: [{[:uuid], {Ecto.UUID, :autogenerate, []}}], else: [] defstruct Enum.reverse(@active_memory_struct_fields) def __attributes__(:adapter), do: unquote(Macro.escape(@adapter)) def __attributes__(:auto_generate_uuid), do: unquote(Macro.escape(@auto_generate_uuid)) def __attributes__(:autogenerate), do: unquote(Macro.escape(autogenerate)) # The first attribute is the table key: `auto_generate_uuid: true` puts # `:uuid` there, otherwise it is the first field declared. def __attributes__(:primary_key), do: unquote(hd(query_fields)) def __attributes__(:match_head), do: Helpers.build_match_head( unquote(query_map), unquote(__MODULE__), unquote(Macro.escape(@adapter)) ) def __attributes__(:query_fields), do: unquote(query_fields) def __attributes__(:query_map), do: unquote(query_map) def __attributes__(:table_options), do: unquote(Macro.escape(@table_options)) def __attributes__(:ttl), do: unquote(Macro.escape(@ttl)) def __attributes__(:types), do: unquote(Macro.escape(types)) # The reflection `Ecto.Changeset.cast/4` looks up when given a struct, # so table structs can be cast and validated like any Ecto schema. @doc false def __changeset__, do: unquote(Macro.escape(types)) for clauses <- ActiveMemory.Table.__attributes__( fields, field_sources ), {args, body} <- clauses do def __attributes__(unquote_splicing(args)), do: unquote(body) end end quote do unquote(prelude) unquote(postlude) end end defp put_struct_field(mod, name, assoc) do fields = Module.get_attribute(mod, :active_memory_struct_fields) if List.keyfind(fields, name, 0) do raise ArgumentError, "field/association #{inspect(name)} already exists on attributes, you must either remove the duplication or choose a different name" end Module.put_attribute(mod, :active_memory_struct_fields, {name, assoc}) end end