yggdrasil v2.1.0 Yggdrasil

Yggdrasil is an immense mythical tree that connects the nine worlds in Norse cosmology.

Yggdrasil manages subscriptions to channels/queues in several brokers with the possibility to add more. Simple Redis, RabbitMQ and PostgreSQL adapters are implemented. Message passing is done through YProcess. YProcess allows to use Phoenix.PubPub as a pub/sub to distribute messages between processes.

Example using Redis

iex(1)> channel = %Yggdrasil.Channel{channel: "redis_channel", decoder: Yggdrasil.Decoder.Default.Redis}
iex(2)> Yggdrasil.subscribe(channel)

By default, the Redis adapter connects to "redis://localhost:6379".

Then in redis:

127.0.0.1:6379> PUBLISH "redis_channel" "hello"
(integer) (1)

And finally if you flush in your iex you’ll see the message received by the Elixir shell:

iex(3> flush()
{:Y_CAST_EVENT, "redis_channel", "hello"}
:ok

Things to note:

  • Every message coming from a broker (Redis, RabbitMQ, PostgreSQL) will be like:
  {:Y_CAST_EVENT, channel, message}

Example using RabbitMQ

First you must have the RabbitMQ exchange created. Otherwise the client won’t connect. Channels for the RabbitMQ adapter use a tupple instead of a string:

{"exchange_name", "routing_key"}

where the exchange should be of type :topic. This would allow you to connect to a channel using a routing key like "*.error" where messages with routing keys like "miami.error" and "barcelona.error" would match the routing key of the Yggdrasil channel.

Let’s say you want to connect to the exchange "amq.topic" (created by default) with the previous routing key ("*.error") where you’ll receive errors from all the servers:

iex(1)> channel = %Yggdrasil.Channel{channel: {"amq.topic", "*.error"}, decoder: Yggdrasil.Decoder.Default.RabbitMQ}
iex(2)> Yggdrasil.subscribe(channel)

By default, the RabbitMQ adapter connects to "amqp://guest:guest@localhost:5672/"

Then using AMQP library, publish some messages in RabbitMQ:

iex(3)> options = Application.get_env(:yggdrasil, :rabbitmq, [])
iex(4)> {:ok, conn} = AMQP.Connection.open(options)
iex(5)> {:ok, chan} = AMQP.Channel.open(conn)
iex(6)> AMQP.Basic.publish(chan, "amq.topic", "miami.error", "Error from Miami")
iex(7)> AMQP.Basic.publish(chan, "amq.topic", "barcelona.error", "Error from Barcelona")

And finally if you flush in your iex you’ll see the message received by the Elixir shell:

iex(8> flush()
{:Y_CAST_EVENT, {"amq.topic", "*.error"}, "Error from Miami"}
{:Y_CAST_EVENT, {"amq.topic", "*.error"}, "Error from Barcelona"}
:ok

Example using PostgreSQL

For this example, it’s necessary to provide a valid configuration for the PostgreSQL adapter i.e:

use Mix.Config

config :yggdrasil,
  postgres: [hostname: "localhost",
            port: 5432,
            username: "yggdrasil_test",
            password: "yggdrasil_test",
            database: "yggdrasil_test"]

This will connect the adapter to the database yggdrasil_test with the user yggdrasil_test and the password yggdrasil_test on localhost:5432.

iex(1)> channel = %Yggdrasil.Channel{channel: "postgres_channel", decoder: Yggdrasil.Decoder.Default.Postgres}
iex(2)> Yggdrasil.subscribe(channel)

Then in PostgreSQL:

yggdrasil_test=> NOTIFY postgres_channel, 'hello'
NOTIFY

And finally if you flush in iex you’ll see the message received by the Elixir shell:

iex(8> flush()
{:Y_CAST_EVENT, "postgres_channel", "hello"}
:ok

Example using GenServer

Any of the previous examples can be wrapped inside a GenServer, in this case it is Redis:

defmodule Subscriber do
  use GenServer

  ###################
  # Client functions.

  def start_link(channel, opts \ []) do
    GenServer.start_link(__MODULE__, channel, opts)
  end

  def stop(subscriber, reason \ :normal) do
    GenServer.stop(subscriber, reason)
  end

  ######################
  # GenServer callbacks.

  def init(channel) do
    Yggdrasil.subscribe(channel)
    {:ok, channel}
  end

  def handle_info({:Y_CAST_EVENT, channel, message}, state) do
    IO.inspect %{channel: channel, message: message}
    {:noreply, state}
  end

  def terminate(_reason, channel) do
    Yggdrasil.unsubscribe(channel)
    :ok
  end
end

So in iex:

iex(1)> channel = %Yggdrasil.Channel{channel: "redis_channel", decoder: Yggdrasil.Decoder.Default.Redis}
iex(2)> {:ok, subscriber} = Subscriber.start_link(channel)
iex(3)>

Again in Redis:

127.0.0.1:6379> PUBLISH "redis_channel" "hello"
(integer) (1)

And finally you’ll see in your iex the following:

%{channel: "redis_channel", message: "hello"}
iex(3)>

Example using YProcess

YProcess is a GenServer wrapper with pubsub capabilities and it has great sinergy with Yggdrasil. The previous example implemented with YProcess would be:

defmodule YSubscriber do
  use YProcess, backend: Yggdrasil.Backend

  ###################
  # Client functions.

  def start_link(channel, opts \ []) do
    YProcess.start_link(__MODULE__, channel, opts)
  end

  def stop(subscriber, reason \ :normal) do
    YProcess.stop(subscriber, reason)
  end

  #####################
  # YProcess callbacks.

  def init(channel) do
    {:join, [channel], channel}
  end

  def handle_event(channel, message, state) do
    IO.inspect %{channel: channel, message: message}
    {:noreply, state}
  end
end

So in iex:

iex(1)> channel = %Yggdrasil.Channel{channel: "redis_channel", decoder: Yggdrasil.Decoder.Default.Redis}
iex(2)> {:ok, y_subscriber} = YSubscriber.start_link(channel)
iex(3)>

Again in Redis:

127.0.0.1:6379> PUBLISH "redis_channel" "hello"
(integer) (1)

And finally you’ll see in your iex the following:

%{channel: "redis_channel", message: "hello"}
iex(3)>

Yggdrasil Channels

Yggdrasil channels have the name of the channel in the broker and the name of the module of the message decoder. A decoder module also defines which adapter should be used to connect to the channel.

%Yggdrasil.Channel{channel: "channel", decoder: Yggdrasil.Decoder.Default.Redis}

The previous example will tell Yggdrasil to subscribe to the channel "channel". The decoder module Yggdrasil.Decoder.Default.Redis defined Redis as the broker and does not change the message coming from Redis before sending it to the subscribers.

Decoders

The current Yggdrasil version has the following decoder modules:

For more information about adapters, see the Adapters section.

To implement a decoder is necessary to implement the decode/2 callback for Yggdrasil.Decoder behaviour, i.e. subscribe to a Redis channel "test" that publishes JSON. The subscribers must receive a map instead of a string with the JSON.

defmodule CustomDecoder do
  use Yggdrasil.Decoder, adapter: Yggdrasil.Adapter.Redis

  def decode(_channel, message) do
    Poison.decode!(message)
  end
end

Important: The channel received by the decode/2 function might be different than the channel the client is subscribed. For example, with the RabbitMQ adapter you can subscribe to the channel {"amq.topic", "*.error"}, but if the routing key of the received message is "barcelona.error", then the channel received by this function will be {"amq.topic", "barcelona.error"} instead of {"amq.topic", "*.error"}. It is a good idea to include this channel to the decoded message in order to know its real procedence.

To subscribe to this channel, clients must use the following Yggdrasil channel:

%Yggdrasil.Channel{channel: "test", decoder: CustomDecoder}

Adapters

The current Yggdrasil version has the following adapters:

Also the function Yggdrasil.publish/2 is used to simulate published messages by any of the brokers.

To implement a new adapter is necessary to use a GenServer or any wrapper over GenServer. For more information, see the source code of any of the implemented adapters.

Installation

Yggdrasil is available as a Hex package. To install, add it to your dependencies in your mix.exs file:

def deps do
  [{:amqp_client, git: "https://github.com/jbrisbin/amqp_client.git", override: true},
  {:yggdrasil, "~> 2.0"}]
end

Overriding :amqp_client dependency is necessary in order to use Yggdrasil with Erlang 19.

and ensure Yggdrasil is started before your application:

def application do
  [applications: [:yggdrasil]]
end

Configuration

Yggdrasil uses YProcess as a means to distribute the messages. So it is necessary to provide a configuration for YProcess if you want to use, for example, Phoenix.PubSub as your pubsub, i.e:

use Mix.Config

config :y_process,
  backend: YProcess.Backend.PhoenixPubSub,
  name: Yggdrasil.PubSub,
  adapter: Phoenix.PubSub.PG2,
  options: [pool_size: 10]

by default, YProcess will use YProcess.Backend.PG2 as default backend.

For Yggdrasil, there’s only one general configuration parameter which is the process name registry. By default, it uses ExReg, which is a simple but rich process name registry. It is possible to use another one like :gproc.

use Mix.Config

config :yggdrasil,
  registry: :gproc

Specific configuration parameters are as follows:

  use Mix.Config

  config :yggdrasil,
    redis: [host: "localhost",
            port: 6379,
            password: "my password"]

The default Redis adapter uses Redix, so the configuration parameters have the same name as the ones in Redix. By default connects to redis://localhost.

  use Mix.Config

  config :yggdrasil,
    rabbitmq: [host: "localhost",
              port: 5672,
              username: "guest",
              password: "guest",
              virtual_host: "/"]

The default RabbitMQ adapter uses AMQP, so the configuration parameters have the same name as the ones in AMQP. By default connects to amqp://guest:guest@localhost/

  use Mix.Config

  config :yggdrasil,
    postgres: [hostname: "localhost",
              port: 5432,
              username: "postgres",
              password: "postgres",
              database: "yggdrasil"]

The default PostgreSQL adapter uses Postgrex, so the configuration parameters have the same name as the ones in Postgrex.

Summary

Functions

Emits a message in a channel. Bypasses the adapter

Subscribes to a channel

Unsubscribe from a channel

Functions

publish(channel, message)

Emits a message in a channel. Bypasses the adapter.

subscribe(channel)

Subscribes to a channel.

unsubscribe(channel)

Unsubscribe from a channel.