Magneto
An Elixir DSL for Amazon DynamoDB
Etymology: the name hints to the device that like the dynamo, generates electric current - not to Professor Xavier’s archenemy.
A word of caution: the library is still very much a work in progress; you can toy with it, help me extend it, but you shouldn’t (yet!) use it for actual work but at your own peril.
Installation
Add
magneto
to your list of dependencies inmix.exs
:Using it as a hex dependency, you’d add the customary:
def deps do [{:magneto, "~> 0.1.2"}] end
Ensure
magneto
is started before your application:def application do [applications: [:magneto]] end
Change the AWS configuration as needed, with keys and region specifications to suit your DynamoDB usage.
[optional] Install the run the local DynamoDB simulator (more info here)
Usage
The rationale behind this library is to provide a declarative and close to natural language way to work with DynamoDB, and conveniently abstract away the heavily-verbose JSON interaction with the data store. To some extent, this is already accomplished by the ExAws, upon which this DSL relies. But the aim is to take the matters further and bring forth the underlying concepts of DynamoDB and make them explicit and first-class citizens.
The list of features and concepts in DynamoDB is pretty terse (compared to a classical RDBMS, for example); therefore the resulting DSL syntax is largely fixed, though it strives to be as natural as possible.
DDL
Describing a model (mapped to one DynamoDB table) should be straight-forward:
defmodule GrandPrixModel do
use Magneto.Model
hash grand_prix: :string #like "Silverstone", "Suzuka" or "Monaco"
range year: :number
attributes winner_driver: :string, best_lap_driver: :string,
pole_position_driver: :string
attribute laps: :number
attribute all_time_winner: :string #most wins on this circuit
index local: GPWinner, range: :all_time_winner, projection: :keys
end
Model Definition
Simply define a module and use Magneto.Model
.
Primary keys
Since the keys specification for DynamoDB is to have either one (hash) key or a composite of two keys (hash and range), declare them as such: hash key_name: :key_type
and (optionally) range key_name: :key_type
.
Since the documentation uses the terms hash and partition key interchangeably,
as well as range and sort key respectively, one can use those synonyms: partition_key key_name: :key_type
and sort_key key_name: :key_type
.
Attributes
The attributes declaration follows the same pattern observed above att_name: :type
. One can specify attributes one by one (attribute att_name...
) or as a list (attributes att_1: :type1, att_2: :type2 ...
).
All the attributes and keys end up forming a struct
so they can be accessed using the
dot notation model.att_name
.
Data Types
From the storage perspective, the values of the attributes are all booleans, numbers, strings, Base64-encoded binaries or sets of them. We wanted to extend this list of native types with some common custom ones: dates, timestamps, UUIDs (commonly-used as artificial PKs), as well as to make transparent the use of embedded models.
Therefore, the list of types allowed for keys and attributes alike is :boolean
,:number
,:string
,:binary
,:date
,:timestamp
, :uuid
EmbeddedModel
and their list-form ([:string]
, [:date]
, etc).
The library is responsible for the conversion to and from the custom data types.
Indexes
There are global and local indexes:
index local: IndexName, range: :range_attribute, projection: [:attributes, :to, :include]
The range attribute is mandatory and must be one of the attributes defined above, as per the DynamoDB specification. The projected attributes must be also already defined. One could use the shortcut
to only project the index keys ... projection: :keys
or all the attributes ... projection: :all
.
The global index mandates that both hash and range keys be specified:
index global: GlobalIndexName, hash: :att_hash, range: :att_range,
projection: [:attributes, :to, :include]
DDL Operations
Creating and deleting tables (permanent DynamoDB operations) is straight-forward:
...
use Magneto
alias My.Magneto.Model
create Model
description = describe Model
destroy Model
DML
The CRUD operations are simple because the storage model is simple:
Create and Update
There is really no distinction between creating a new item or updating an existing one (eventual consistency topics aside). The map of values to be inserted needs to include the keys too.
...
use Magneto
alias My.Magneto.Model
put %{hash_key: uid, attribute1: att1_value}, into: Model
Reading
One needs to be aware of the one-key or composite-key structure of the model, because reading an item must specify all the keys for the item:
...
use Magneto
alias My.Magneto.SingleKeyModel
alias My.Magneto.CompositeKeyModel
v1 = get SingleKeyModel, for: uid
uid = v1.hash
IO.puts "attribute 1: #{v1.attribute1}"
v2 = get CompositeKeyModel, for: hash_value, and: range_value
hv = v2.hash
rv = v2.range
IO.puts "Retrieved vales for #{inspect v2.keys}: #{inspect v2}"
The for:
keyword may be substituted with hash:
or with:
, depending on which phrasing sounds
better in English.
Queries and Scans
This is largely work-in-progress, but the gist of usage should be:
...
use Magneto
alias My.Formula1.Stats.GrandPrixModel
triple_crowns = scan GrandPrixModel, where: winner_driver == pole_position_driver
and winner_driver == fastest_lap_driver, limit: 10
all_years_on_Monza = query GrandPrixModel, for: "Monza"
Other concepts
Namespaces
The concept of namespace is artificial - it basically uses a prefix for the names of the tables and indexes. It can be useful when one wants to delimit tables into sub-systems or when one has many deployment environments (such as development, test and production) using the same AWS account and wants to keep their data separate (obviously).
The root namespace is declared globally in the configuration:
config :magneto,
namespace: "local"
Inside a model, one can extend it:
defmodule GrandPrixModel do
use Magneto.Model
namespace "analytics" # full name of the table: local.analytics.GrandPrixModel
...
or even override it altogether namespace "global", :override
Throughput
The read and write capacity is specified at the model level as
throughput read: <number>, write: <number>
Also, the global indexes support a similar keyword: ... projection: :keys, throughput: [<number>, <number>]
Roadmap
Unfinished features:
- pretty much all of them, for the moment :-S
Features to be added:
- Embedded models
- Two-phase commit emulation using static tables in DynamoDB
Attribution
Some code inspired by the amazing Ecto library and the whole scaffolding gratefully relies on the wonderful ExAws.
License
Copyright (c) 2016 Lucian Feroiu
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.