Magneto

An Elixir DSL for Amazon DynamoDB

Etymology: the name hints to the device that like the dynamo, generates electric current - not to Professor Xavier’s archenemy.

A word of caution: the library is still very much a work in progress; you can toy with it, help me extend it, but you shouldn’t (yet!) use it for actual work but at your own peril.

Installation

  1. Add magneto to your list of dependencies in mix.exs:

    Using it as a hex dependency, you’d add the customary:

    def deps do
      [{:magneto, "~> 0.1.2"}]
    end
  2. Ensure magneto is started before your application:

    def application do
      [applications: [:magneto]]
    end
  3. Change the AWS configuration as needed, with keys and region specifications to suit your DynamoDB usage.

  4. [optional] Install the run the local DynamoDB simulator (more info here)


Usage

The rationale behind this library is to provide a declarative and close to natural language way to work with DynamoDB, and conveniently abstract away the heavily-verbose JSON interaction with the data store. To some extent, this is already accomplished by the ExAws, upon which this DSL relies. But the aim is to take the matters further and bring forth the underlying concepts of DynamoDB and make them explicit and first-class citizens.

The list of features and concepts in DynamoDB is pretty terse (compared to a classical RDBMS, for example); therefore the resulting DSL syntax is largely fixed, though it strives to be as natural as possible.

DDL

Describing a model (mapped to one DynamoDB table) should be straight-forward:

defmodule GrandPrixModel do
  use Magneto.Model
  hash grand_prix: :string #like "Silverstone", "Suzuka" or "Monaco"
  range year: :number

  attributes winner_driver: :string, best_lap_driver: :string,
      pole_position_driver: :string
  attribute laps: :number
  attribute all_time_winner: :string #most wins on this circuit

  index local: GPWinner, range: :all_time_winner, projection: :keys

end

Model Definition

Simply define a module and use Magneto.Model.

Primary keys

Since the keys specification for DynamoDB is to have either one (hash) key or a composite of two keys (hash and range), declare them as such: hash key_name: :key_type and (optionally) range key_name: :key_type. Since the documentation uses the terms hash and partition key interchangeably, as well as range and sort key respectively, one can use those synonyms: partition_key key_name: :key_type and sort_key key_name: :key_type.

Attributes

The attributes declaration follows the same pattern observed above att_name: :type. One can specify attributes one by one (attribute att_name...) or as a list (attributes att_1: :type1, att_2: :type2 ...).

All the attributes and keys end up forming a struct so they can be accessed using the dot notation model.att_name.

Data Types

From the storage perspective, the values of the attributes are all booleans, numbers, strings, Base64-encoded binaries or sets of them. We wanted to extend this list of native types with some common custom ones: dates, timestamps, UUIDs (commonly-used as artificial PKs), as well as to make transparent the use of embedded models.

Therefore, the list of types allowed for keys and attributes alike is :boolean,:number,:string,:binary,:date,:timestamp, :uuid EmbeddedModel and their list-form ([:string], [:date], etc).

The library is responsible for the conversion to and from the custom data types.

Indexes

There are global and local indexes:

index local: IndexName, range: :range_attribute, projection: [:attributes, :to, :include]

The range attribute is mandatory and must be one of the attributes defined above, as per the DynamoDB specification. The projected attributes must be also already defined. One could use the shortcut to only project the index keys ... projection: :keys or all the attributes ... projection: :all.

The global index mandates that both hash and range keys be specified:

index global: GlobalIndexName, hash: :att_hash, range: :att_range, projection: [:attributes, :to, :include]

DDL Operations

Creating and deleting tables (permanent DynamoDB operations) is straight-forward:

...
use Magneto
alias My.Magneto.Model

create Model
description = describe Model
destroy Model

DML

The CRUD operations are simple because the storage model is simple:

Create and Update

There is really no distinction between creating a new item or updating an existing one (eventual consistency topics aside). The map of values to be inserted needs to include the keys too.

...
use Magneto
alias My.Magneto.Model

put %{hash_key: uid, attribute1: att1_value}, into: Model

Reading

One needs to be aware of the one-key or composite-key structure of the model, because reading an item must specify all the keys for the item:

...
use Magneto
alias My.Magneto.SingleKeyModel
alias My.Magneto.CompositeKeyModel

v1 = get SingleKeyModel, for: uid
uid = v1.hash
IO.puts "attribute 1: #{v1.attribute1}"

v2 = get CompositeKeyModel, for: hash_value, and: range_value
hv = v2.hash
rv = v2.range
IO.puts "Retrieved vales for #{inspect v2.keys}: #{inspect v2}"

The for: keyword may be substituted with hash: or with:, depending on which phrasing sounds better in English.

Queries and Scans

This is largely work-in-progress, but the gist of usage should be:

...
use Magneto
alias My.Formula1.Stats.GrandPrixModel

triple_crowns = scan GrandPrixModel, where: winner_driver == pole_position_driver
    and winner_driver == fastest_lap_driver, limit: 10

all_years_on_Monza = query GrandPrixModel, for: "Monza"

Other concepts

Namespaces

The concept of namespace is artificial - it basically uses a prefix for the names of the tables and indexes. It can be useful when one wants to delimit tables into sub-systems or when one has many deployment environments (such as development, test and production) using the same AWS account and wants to keep their data separate (obviously).

The root namespace is declared globally in the configuration:

config :magneto,
  namespace: "local"

Inside a model, one can extend it:

defmodule GrandPrixModel do
  use Magneto.Model
  namespace "analytics" # full name of the table: local.analytics.GrandPrixModel
  ...

or even override it altogether namespace "global", :override

Throughput

The read and write capacity is specified at the model level as throughput read: <number>, write: <number>

Also, the global indexes support a similar keyword: ... projection: :keys, throughput: [<number>, <number>]


Roadmap

Unfinished features:

  • pretty much all of them, for the moment :-S

Features to be added:

  • Embedded models
  • Two-phase commit emulation using static tables in DynamoDB

Attribution

Some code inspired by the amazing Ecto library and the whole scaffolding gratefully relies on the wonderful ExAws.

License

Copyright (c) 2016 Lucian Feroiu

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.