View Source MongoQueue (MongoQueue v0.0.4)
A queue implementation using MongoDB.
Inspired by the NPM mongodb-queue package.
getting-started
Getting Started
config
Config
To use MongoQueue, you must first create a queue configuration. This configuration requires a connection to a MongoDB database, and the name of the collection to use for the queue.
To connect to Mongo, consult the mongodb_driver
docs.
For example:
{:ok, conn} = Mongo.start_link(url: "mongodb://localhost:27017/my-database")
Then, create a queue configuration:
config = MongoQueue.Config.new(conn, "my_queue")
Configuration options are described in detail in the MongoQueue.Config
module.
creating-indexes
Creating Indexes
To ensure high performance for the queue, you create indexes on the queue collection. This only needs to be run once per queue.
:ok = MongoQueue.create_indexes(config)
adding-messages
Adding Messages
To add a message to the queue, use the add/2
function:
{:ok, message_id} = MongoQueue.add(config, %{foo: "bar"})
receiving-messages
Receiving Messages
To receive a message from the queue, use the get/1
function:
{:ok, message} = MongoQueue.get(config)
By default, messages are claimed for 30 seconds. If they are not acknowledged within that time, they will become available to be received again.
To customize this timeout, use the visibility_timeout
option:
{:ok, message} = MongoQueue.get(config, visibility_timeout: 60)
acknowledging-messages
Acknowledging Messages
When the message has finished processing, call the ack/2
function:
:ok = MongoQueue.ack(config, message.ack)
working-in-bulk
Working in bulk
adding-messages-1
Adding Messages
To add multiple messages to the queue, use the add_many/2
function:
{:ok, message_ids} = MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}])
receiving-messages-1
Receiving Messages
To receive multiple messages from the queue, use the get_many/2
function:
{:ok, messages} = MongoQueue.get_many(config, 2)
Note: This function performs a multi-document transaction. See the warning on the get_many/2
function for more information.
acknowledging-messages-1
Acknowledging Messages
To acknowledge multiple messages, use the ack/2
function, with a list of ack IDs:
:ok = MongoQueue.ack(config, [message1.ack, message2.ack])
Statistics
A few methods are provided to help you observe the state of the queue.
total-messages
Total Messages
The total/1
function returns the number of messages that have been added to the queue, regardless of their status.
This will not include messages that have been deleted by clean/1
.
size
Size
The size/1
function returns the number of messages that are currently enqueued and visible.
in-flight
In Flight
The in_flight/1
function returns the number of messages that have been received from the queue but not yet acknowledged.
done
Done
The done/1
function returns the number of messages that have been acknowledged but not yet deleted.
cleaning-up
Cleaning Up
From time to time, it may be necessary to clean up the queue collection.
To do this, run the clean/1
function.
This function deletes all messages that have been acknowledged.
Link to this section Summary
Functions
Marks a message as acknowledged, making it unavailable to be received from the queue in the future.
Adds a message to the queue.
Adds multiple messages to the queue.
Removes acknowledged documents from the queue collection.
Creates indexes to ensure high performance for the queue.
Returns the number of messages that have been acknowledged but not yet deleted.
Gets a payload from the queue.
Gets multiple messages from the queue.
Returns the number of messages that have been received from the queue but not yet acknowledged.
Marks a message as not acknowledged, making it immediately available to be received from the queue again.
Extends the visibility timeout of a received message.
Returns the number of enqueued, visible messages in the queue’s collection.
Returns the total number of messages in the queue’s collection—regardless of their status.
Link to this section Types
@type message() :: %{id: BSON.ObjectId.t(), ack: BSON.ObjectId.t(), payload: any()}
Link to this section Functions
@spec ack(MongoQueue.Config.t(), BSON.ObjectId.t() | [BSON.ObjectId.t()], Keyword.t()) :: :ok | {:error, any()}
Marks a message as acknowledged, making it unavailable to be received from the queue in the future.
Acknowledged messages may eventually be deleted, using the clean/1
function.
parameters
Parameters
config
- A queue configuration.ack_or_acks
- The ack ID of the message to acknowledge, or a list of ack IDs to acknowledge.opts
- Additional options, to override those found in theconfig
parameter.
examples
Examples
iex> MongoQueue.ack(config, ack)
:ok
iex> MongoQueue.ack(config, [ack1, ack2])
:ok
@spec add(MongoQueue.Config.t(), any(), Keyword.t()) :: {:ok, BSON.ObjectId.t()} | {:error, any()}
Adds a message to the queue.
examples
Examples
iex> MongoQueue.add(config, %{foo: "bar"})
{:ok, #BSON.ObjectId<5f0e1e1b0000000000000000>}
iex> MongoQueue.add(config, %{foo: "bar"}, delay: 60)
{:ok, #BSON.ObjectId<5f0e1e1b0000000000000000>}
@spec add_many(MongoQueue.Config.t(), Enumerable.t(any()), Keyword.t()) :: {:ok, [BSON.ObjectId.t()]} | {:error, any()}
Adds multiple messages to the queue.
examples
Examples
iex> MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}])
{:ok, [#BSON.ObjectId<5f0e1e1b0000000000000000>, #BSON.ObjectId<5f0e1e1b0000000000000001>]}
iex> MongoQueue.add_many(config, [%{foo: "bar"}, %{foo: "baz"}], delay: 60)
{:ok, [#BSON.ObjectId<5f0e1e1b0000000000000000>, #BSON.ObjectId<5f0e1e1b0000000000000001>]}
Removes acknowledged documents from the queue collection.
Messages remain in the MongoDB collection until they are deleted by this function.
Creates indexes to ensure high performance for the queue.
This only needs to be run once per queue.
Returns the number of messages that have been acknowledged but not yet deleted.
@spec get(MongoQueue.Config.t(), Keyword.t()) :: {:ok, message()} | {:ok, nil} | {:error, any()}
Gets a payload from the queue.
examples
Examples
iex> MongoQueue.get(config)
{:ok, %{
id: #BSON.ObjectId<5f0e1e1b0000000000000000>,
ack: #BSON.ObjectId<5f0e1e1b0000000000000001>,
payload: %{foo: "bar"}
}}
iex> MongoQueue.get(config)
{:ok, nil}
@spec get_many(MongoQueue.Config.t(), non_neg_integer(), Keyword.t()) :: {:ok, [message()]} | {:error, any()}
Gets multiple messages from the queue.
Warning
This operation performs a multi-document transaction. In MongoDB, this requires a replica set. To learn more, check out MongoDB’s documentation.
parameters
Parameters
config
- A queue configuration.count
- The maximum number of messages to retrieve. If fewer than this number of messages are available, a smaller amount will be received.opts
- Additional options, to override those found in theconfig
parameter.
examples
Examples
iex> MongoQueue.get_many(config, 2)
{:ok, [
%{
id: #BSON.ObjectId<5f0e1e1b0000000000000000>,
ack: #BSON.ObjectId<5f0e1e1b0000000000000001>,
payload: %{foo: "bar"}
},
%{
id: #BSON.ObjectId<5f0e1e1b0000000000000002>,
ack: #BSON.ObjectId<5f0e1e1b0000000000000003>,
payload: %{foo: "baz"}
}
]}
iex> MongoQueue.get_many(config, 2)
{:ok, []}
Returns the number of messages that have been received from the queue but not yet acknowledged.
@spec nack(MongoQueue.Config.t(), BSON.ObjectId.t(), Keyword.t()) :: :ok | {:error, any()}
Marks a message as not acknowledged, making it immediately available to be received from the queue again.
examples
Examples
iex> MongoQueue.nack(config, ack)
:ok
Extends the visibility timeout of a received message.
This allows long-running tasks to operate on a message without the risk of it being received by another process.
examples
Examples
iex> MongoQueue.ping(config, ack)
:ok
Returns the number of enqueued, visible messages in the queue’s collection.
Returns the total number of messages in the queue’s collection—regardless of their status.