Errol

An opinionated framework to run and orchestrate RabbitMQ consumers.

Index

Installation

The package can be installed by adding errol to your list of dependencies in mix.exs:

def deps do
  [{:errol, "~> 0.1.0"}]
end

You should also update your application list to include :errol:

def application do
  [applications: [:errol]]
end

Documentation can be found at https://hexdocs.pm/errol.

Usage

To bind consumers to queue, you can use the Errol.Wiring module:

defmodule Sample.Wiring do
  use Wiring

  connection "amqp://guest:guest@localhost"

  @exchange "/users"
  @exchange_type :topic

  # You can pass a reference to a function with arity of 1
  consume("account_created", "users.account.created", &UsersConsumer.account_created/1)

  # or even an anonymous function
  consume("account_updated", "users.account.updated", fn message -> ... end)
end

For more complex setups, you can add middleware and group different consumers for more granularity.

defmodule Sample.Wiring do
  use Wiring

  connection "amqp://guest:guest@localhost"

  @exchange "/users"
  @exchange_type :topic

  # Use pipe_before/1, pipe_after/1 or pipe_error/1 to run middleware functions
  # middlewares declared outside of a group will run for every consumer
  pipe_after(&Sample.StatisticsMiddleware.track/2)
  
  # Use the `group` macro to group consumers with specific middleware
  group :account do
    # This middleware will run only for consumers in the :account group
    pipe_before(&Errol.Middleware.Json.parse/2)

    consume("account_created", "users.account.created", &UsersConsumer.account_created/1)
    consume("account_updated", "users.account.updated", fn message -> ... end)
  end

  group :photos do
    pipe_before(&Sample.ImagesMiddleware.uncompress/2)

    consume("profile_photo_uploaded", "users.profile.photo.uploaded", fn message -> ... end)
  end
end

At this point, the only thing left is to run Sample.Wiring as a supervisor in your application.ex file:

defmodule Sample.Application do
  use Application

  def start(_type, _args) do
    import Supervisor.Spec

    children = [
      supervisor(Sample.Wiring, []),
      ...
    ]

    opts = [strategy: :one_for_one, name: Sample.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Voilà! This will spin up the following supervision tree:


                                      --------------------
                                     | Sample.Application |
                                      --------------------
                                               |
                                               |
                                        ---------------
                                       | Sample.Wiring |
                                        ---------------
                                               |
                  _____________________________|_____________________________
                 |                             |                             |
                 |                             |                             |
   ---------------------------    ---------------------------    -------------------------
  | :account_created_consumer |  | :account_updated_consumer |  | :profile_photo_uploaded |
   ---------------------------    ---------------------------    -------------------------
       .     .     .     .           .    .    .    .    .          .     .     .     .
       .     .     .     .           .    .    .    .    .          .     .     .     .
       .     .     .     .           .    .    .    .    .          .     .     .     .
       .     .     .     .           .    .    .    .    .          .     .     .     .
       .     .     .     .           .    .    .    .    .          .     .     .     .

                       New monitored process per each message received

Compatibility with other AMQP implementations exists but is not guaranteed (at least for now 😁).

Running locally

Clone the repository

git clone git@github.com:uesteibar/errol.git

Install dependencies

cd errol
mix deps.get

To run the tests (you will need docker installed)

./scripts/test_prepare.sh
mix test

Roadmap

  • [ ] Allowing to specify the number of retries. For now, when a message fails to be processed it will be redelivered only once. After that the message will be just rejected.
  • [ ] Adding a middleware to ensure duplicated messages are not consumed more than once.
  • [ ] Allowing to specify number of workers per consumer. Poolboy would come handy here.
  • [ ] Handle RabbitMQ outages, following the great explanation in the amqp hex documentation.
  • [ ] Publishing messages.

Contributing

Pull requests are always welcome =)

The project uses standard-changelog to update the Changelog with each commit message and upgrade the package version. For that reason every contribution must have a title and body that follows the conventional commits standard conventions (e.g. feat(consumer): Consume it all).

To make this process easier, you can do the following:

Install commitizen and cz-conventional-changelog globally

npm i -g commitizen cz-conventional-changelog

Save cz-conventional-changelog as default

echo '{ "path": "cz-conventional-changelog" }' > ~/.czrc

Instead of git commit, you can now run

git cz

and follow the instructions to generate the commit message.

Credit

🎉 Special thanks to @pma for the amazing work on the amqp hex.