View Source MongoQueue (MongoQueue v0.0.3)

A queue implementation using MongoDB.

Inspired by the NPM mongodb-queue package.

Getting Started

config

Config

To use MongoQueue, you must first create a queue configuration. This configuration requires a connection to a MongoDB database, and the name of the collection to use for the queue.

To connect to Mongo, consult the mongodb_driver docs.

For example:

{:ok, conn} = Mongo.start_link(url: "mongodb://localhost:27017/my-database")

Then, create a queue configuration:

config = MongoQueue.Config.new(conn, "my_queue")

Configuration options are described in detail in the MongoQueue.Config module.

creating-indexes

Creating Indexes

To ensure high performance for the queue, you create indexes on the queue collection. This only needs to be run once per queue.

:ok = MongoQueue.create_indexes(config)

adding-messages

Adding Messages

To add a message to the queue, use the add/2 function:

{:ok, message_id} = MongoQueue.add(config, %{foo: "bar"})

receiving-messages

Receiving Messages

To receive a message from the queue, use the get/1 function:

{:ok, message} = MongoQueue.get(config)

By default, messages are claimed for 30 seconds. If they are not acknowledged within that time, they will become available to be received again.

To customize this timeout, use the visibility_timeout option:

{:ok, message} = MongoQueue.get(config, visibility_timeout: 60)

acknowledging-messages

Acknowledging Messages

When the message has finished processing, call the ack/2 function:

:ok = MongoQueue.ack(config, message.ack)

Working with multiple messages at once

adding-messages-1

Adding Messages

To add multiple messages to the queue, use the add_many/2 function:

{:ok, message_ids} = MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}])

receiving-messages-1

Receiving Messages

To receive multiple messages from the queue, use the get_many/2 function:

{:ok, messages} = MongoQueue.get_many(config, 2)

Note: This function performs a multi-document transaction. See the warning on the get_many/2 function for more information.

acknowledging-messages-1

Acknowledging Messages

To acknowledge multiple messages, use the ack/2 function, with a list of ack IDs:

:ok = MongoQueue.ack(config, [message1.ack, message2.ack])

Statistics

A few methods are provided to help you observe the state of the queue.

total-messages

Total Messages

The total/1 function returns the number of messages that have been added to the queue, regardless of their status. This will not include messages that have been deleted by clean/1.

size

Size

The size/1 function returns the number of messages that are currently enqueued and visible.

in-flight

In Flight

The in_flight/1 function returns the number of messages that have been received from the queue but not yet acknowledged.

done

Done

The done/1 function returns the number of messages that have been acknowledged but not yet deleted.

Cleaning Up

From time to time, it may be necessary to clean up the queue collection. To do this, run the clean/1 function.

This function deletes all messages that have been acknowledged.

Link to this section Summary

Functions

Marks a message as acknowledged, making it unavailable to be received from the queue in the future.

Adds a message to the queue.

Adds multiple messages to the queue.

Removes acknowledged documents from the queue collection.

Creates indexes to ensure high performance for the queue.

Returns the number of messages that have been acknowledged but not yet deleted.

Gets a payload from the queue.

Gets multiple messages from the queue.

Returns the number of messages that have been received from the queue but not yet acknowledged.

Marks a message as not acknowledged, making it immediately available to be received from the queue again.

Extends the visibility timeout of a received message.

Returns the number of enqueued, visible messages in the queue’s collection.

Returns the total number of messages in the queue’s collection—regardless of their status.

Link to this section Types

@type message() :: %{id: BSON.ObjectId.t(), ack: BSON.ObjectId.t(), payload: any()}

Link to this section Functions

Link to this function

ack(config, ack_or_acks, opts \\ [])

View Source
@spec ack(MongoQueue.Config.t(), BSON.ObjectId.t() | [BSON.ObjectId.t()], Keyword.t()) ::
  :ok | {:error, any()}

Marks a message as acknowledged, making it unavailable to be received from the queue in the future.

Acknowledged messages may eventually be deleted, using the clean/1 function.

parameters

Parameters

  • config - A queue configuration.
  • ack_or_acks - The ack ID of the message to acknowledge, or a list of ack IDs to acknowledge.
  • opts - Additional options, to override those found in the config parameter.

examples

Examples

iex> MongoQueue.ack(config, ack)
:ok

iex> MongoQueue.ack(config, [ack1, ack2])
:ok
Link to this function

add(config, payload, opts \\ [])

View Source
@spec add(MongoQueue.Config.t(), any(), Keyword.t()) ::
  {:ok, BSON.ObjectId.t()} | {:error, any()}

Adds a message to the queue.

examples

Examples

iex> MongoQueue.add(config, %{foo: "bar"})
{:ok, #BSON.ObjectId<5f0e1e1b0000000000000000>}

iex> MongoQueue.add(config, %{foo: "bar"}, delay: 60)
{:ok, #BSON.ObjectId<5f0e1e1b0000000000000000>}
Link to this function

add_many(config, payloads, opts \\ [])

View Source
@spec add_many(MongoQueue.Config.t(), Enumerable.t(any()), Keyword.t()) ::
  {:ok, [BSON.ObjectId.t()]} | {:error, any()}

Adds multiple messages to the queue.

examples

Examples

iex> MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}])
{:ok, [#BSON.ObjectId<5f0e1e1b0000000000000000>, #BSON.ObjectId<5f0e1e1b0000000000000001>]}

iex> MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}], delay: 60)
{:ok, [#BSON.ObjectId<5f0e1e1b0000000000000000>, #BSON.ObjectId<5f0e1e1b0000000000000001>]}

Removes acknowledged documents from the queue collection.

Messages remain in the MongoDB collection until they are deleted by this function.

Creates indexes to ensure high performance for the queue.

This only needs to be run once per queue.

Returns the number of messages that have been acknowledged but not yet deleted.

@spec get(MongoQueue.Config.t(), Keyword.t()) ::
  {:ok, message()} | {:ok, nil} | {:error, any()}

Gets a payload from the queue.

examples

Examples

iex> MongoQueue.get(config)
{:ok, %{
  id: #BSON.ObjectId<5f0e1e1b0000000000000000>,
  ack: #BSON.ObjectId<5f0e1e1b0000000000000001>,
  payload: %{foo: "bar"}
}}

iex> MongoQueue.get(config)
{:ok, nil}
Link to this function

get_many(config, count, opts \\ [])

View Source
@spec get_many(MongoQueue.Config.t(), non_neg_integer(), Keyword.t()) ::
  {:ok, [message()]} | {:error, any()}

Gets multiple messages from the queue.

Warning

This operation performs a multi-document transaction. In MongoDB, this requires a replica set. To learn more, check out MongoDB’s documentation.

parameters

Parameters

  • config - A queue configuration.
  • count - The maximum number of messages to retrieve. If fewer than this number of messages are available, a smaller amount will be received.
  • opts - Additional options, to override those found in the config parameter.

examples

Examples

iex> MongoQueue.get_many(config, 2)
{:ok, [
  %{
    id: #BSON.ObjectId<5f0e1e1b0000000000000000>,
    ack: #BSON.ObjectId<5f0e1e1b0000000000000001>,
    payload: %{foo: "bar"}
  },
  %{
    id: #BSON.ObjectId<5f0e1e1b0000000000000002>,
    ack: #BSON.ObjectId<5f0e1e1b0000000000000003>,
    payload: %{foo: "baz"}
  }
]}

iex> MongoQueue.get_many(config, 2)
{:ok, []}

Returns the number of messages that have been received from the queue but not yet acknowledged.

Link to this function

nack(config, ack, opts \\ [])

View Source
@spec nack(MongoQueue.Config.t(), BSON.ObjectId.t(), Keyword.t()) ::
  :ok | {:error, any()}

Marks a message as not acknowledged, making it immediately available to be received from the queue again.

examples

Examples

iex> MongoQueue.nack(config, ack)
:ok
Link to this function

ping(config, ack, opts \\ [])

View Source

Extends the visibility timeout of a received message.

This allows long-running tasks to operate on a message without the risk of it being received by another process.

examples

Examples

iex> MongoQueue.ping(config, ack)
:ok

Returns the number of enqueued, visible messages in the queue’s collection.

Returns the total number of messages in the queue’s collection—regardless of their status.