honeydew v1.1.0 Honeydew View Source

A pluggable job queue + worker pool for Elixir.

Link to this section Summary

Functions

Runs a task asynchronously

Cancels a job

Cancels the job associated with the first argument

Filters the jobs currently on the queue

Moves a job to another queue

Creates a supervision spec for a queue

Resumes job processing for a queue

Suspends job processing for a queue

Creates a supervision spec for workers

Wait for a job to complete and return result

Link to this section Types

Link to this type async_opt() View Source
async_opt() :: [{:reply, true}]
Link to this type mod_or_mod_args() View Source
mod_or_mod_args() :: module() | {module(), args :: term()}
Link to this type queue_name() View Source
queue_name() :: String.t() | atom() | {:global, String.t() | atom()}
Link to this type queue_spec_opt() View Source
queue_spec_opt() ::
  {:queue, mod_or_mod_args()}
  | {:dispatcher, mod_or_mod_args()}
  | {:failure_mode, mod_or_mod_args() | nil}
  | {:success_mode, mod_or_mod_args() | nil}
  | {:supervisor_opts, supervisor_opts()}
  | {:suspended, boolean()}

Result of a Honeydew.Job

Link to this type supervisor_opts() View Source
supervisor_opts() :: Keyword.t()
Link to this type task() View Source
task() :: {atom(), [arg :: term()]}
Link to this type worker_spec_opt() View Source
worker_spec_opt() ::
  {:num, non_neg_integer()}
  | {:init_retry, non_neg_integer()}
  | {:supervisor_opts, supervisor_opts()}
  | {:nodes, [node()]}

Link to this section Functions

Link to this function async(task, queue, opts \\ []) View Source
async(task(), queue_name(), [async_opt()]) :: Honeydew.Job.t() | no_return()

Runs a task asynchronously.

Raises a RuntimeError if queue process is not available.

Examples

To run a task asynchronously.

Honeydew.async({:ping, ["127.0.0.1"]}, :my_queue)

To run a task asynchronously and wait for result.

# Without pipes
job = Honeydew.async({:ping, ["127.0.0.1"]}, :my_queue, reply: true)
Honeydew.yield(job)

# With pipes
result =
  {:ping, ["127.0.0.1"]}
  |> Honeydew.async(:my_queue, reply: true)
  |> Honeydew.yield()
Link to this function cancel(job) View Source
cancel(Honeydew.Job.t()) :: :ok | {:error, :in_progress} | {:error, :not_found}

Cancels a job.

The return value depends on the status of the job.

  • :ok - Job had not been started and was able to be cancelled.
  • {:error, :in_progress} - Job was in progress and unable to be cancelled.
  • {:error, :not_found} - Job was not found on the queue (or already processed) and was unable to be cancelled.
Link to this function cancel(private, queue) View Source
cancel(Honeydew.Job.private(), queue_name()) ::
  :ok | {:error, :in_progress} | {:error, :not_found}

Cancels the job associated with the first argument.

For example, for the Ecto Poll Queue, the first argument is the value of an ID from your schema.

The return value depends on the status of the job.

  • :ok - Job had not been started and was able to be cancelled.
  • {:error, :in_progress} - Job was in progress and unable to be cancelled, the Ecto Poll Queue does not support this return.
  • {:error, :not_found} - Job was not found on the queue (or already processed) and was unable to be cancelled.
Link to this function filter(queue, function) View Source
filter(queue_name(), (Honeydew.Job.t() -> boolean())) :: [Honeydew.Job.t()]

Filters the jobs currently on the queue.

Please Note — This function returns a List, not a Stream, so calling it can be memory intensive when invoked on a large queue.

Examples

Filter jobs with a specific task.

Honeydew.filter(:my_queue, &match?(%Honeydew.Job{task: {:ping, _}}, &1))

Return all jobs.

Honeydew.filter(:my_queue, fn _ -> true end)

Moves a job to another queue.

Raises a RuntimeError if to_queue is not available.

This function first enqueues the job on to_queue, and then tries to cancel it on its current queue. This means there’s a possiblity a job could be processed on both queues. This behavior is consistent with Honeydew’s at-least-once execution goal.

This function is most helpful on a queue where there a no workers (like a dead letter queue), because the job won’t be processed out from under the queue.

Link to this function queue_spec(name, opts \\ []) View Source

Creates a supervision spec for a queue.

name is how you’ll refer to the queue to add a task.

You can provide any of the following opts:

  • queue: is the module that queue will use. Defaults to Honeydew.Queue.ErlangQueue. You may also provide args to the queue’s Honeydew.Queue.init/2 callback using the following format: {module, args}.

  • dispatcher: the job dispatching strategy, {module, init_args}.

  • failure_mode: the way that failed jobs should be handled. You can pass either a module, or {module, args}. The module must implement the Honeydew.FailureMode behaviour. Defaults to {Honeydew.FailureMode.Abandon, []}.

  • success_mode: a callback that runs when a job successfully completes. You can pass either a module, or {module, args}. The module must implement the Honeydew.SuccessMode behaviour. Defaults to nil.

  • supervisor_opts: options accepted by Supervisor.Spec.supervisor/3.

  • suspended: Start queue in suspended state. Defaults to false.

For example:

  • Honeydew.queue_spec("my_awesome_queue")

  • Honeydew.queue_spec("my_awesome_queue", queue: {MyQueueModule, [ip: "localhost"]}, dispatcher: {Honeydew.Dispatcher.MRU, []})

Link to this function resume(queue) View Source
resume(queue_name()) :: :ok

Resumes job processing for a queue.

Link to this function suspend(queue) View Source
suspend(queue_name()) :: :ok

Suspends job processing for a queue.

Link to this function worker_spec(queue, module_and_args, opts \\ []) View Source

Creates a supervision spec for workers.

queue is the name of the queue that the workers pull jobs from.

module is the module that the workers in your queue will use. You may also provide Honeydew.Worker.init/1 args with {module, args}.

You can provide any of the following opts:

  • num: the number of workers to start. Defaults to 10.

  • init_retry: the amount of time, in seconds, to wait before respawning a worker whose Honeydew.Worker.init/1 function failed. Defaults to 5.

  • shutdown: if a worker is in the middle of a job, the amount of time, in milliseconds, to wait before brutally killing it. Defaults to 10_000.

  • supervisor_opts options accepted by Supervisor.Spec.supervisor/3.

  • nodes: for :global queues, you can provide a list of nodes to stay connected to (your queue node and enqueuing nodes). Defaults to [].

For example:

  • Honeydew.worker_spec("my_awesome_queue", MyJobModule)

  • Honeydew.worker_spec("my_awesome_queue", {MyJobModule, [key: "secret key"]}, num: 3)

  • Honeydew.worker_spec({:global, "my_awesome_queue"}, MyJobModule, nodes: [:clientfacing@dax, :queue@dax])

Link to this function yield(job, timeout \\ 5000) View Source
yield(Honeydew.Job.t(), timeout()) :: {:ok, result()} | nil | no_return()

Wait for a job to complete and return result.

Returns the result of a job, or nil on timeout. Raises an ArgumentError if the job was not created with reply: true and in the current process.

Example

Calling yield/2 with different timeouts.

iex> job = Honeydew.async({:ping, ["127.0.0.1"]}, :my_queue, reply: true)
iex> Honeydew.yield(job, 500) # Wait half a second
nil
# Result comes in at 1 second
iex> Honeydew.yield(job, 1000) # Wait up to a second
{:ok, :pong}
iex> Honeydew.yield(job, 0)
nil # <- because the message has already arrived and been handled

The only time yield/2 would ever return the result more than once is if the job executes more than once (as Honeydew aims for at-least-once execution).