View Source OddJob (OddJob v0.3.1)

Job pools for Elixir OTP applications, written in Elixir.

Use OddJob when you want to limit concurrency of background processing in your Elixir app. One possible use case is forcing backpressure on calls to external APIs.

Installation

The package can be installed by adding odd_job to your list of dependencies in mix.exs:

def deps do
[
  {:odd_job, "~> 0.3.0"}
]
end

Getting started

OddJob automatically starts up a supervised job pool of 5 workers out of the box with no extra configuration. The default name of this job pool is :job, and it can be sent work in the following way:

OddJob.perform(:job, fn -> do_some_work() end)

The default pool can be customized in your config if you want to change the name or pool size:

config :odd_job,
  default_pool: :work,
  pool_size: 10 # this changes the size of all pools in your application, defaults to 5

You can also add extra pools to be supervised by the OddJob application supervision tree:

config :odd_job,
  extra_pools: [:email, :external_app]

If you don't want OddJob to supervise any pools for you (including the default :job pool) then pass false to the :default_pool config key:

config :odd_job, default_pool: false

Supervising job pools

To supervise your own job pools you can add a tuple in the form of {OddJob, name} (where name is an atom) directly to the top level of your application's supervision tree or any other list of child specs for a supervisor:

defmodule MyApp.Application do
  use Application

  def start(_type, _args) do

    children = [
      {OddJob, :email},
      {OddJob, :external_app}
    ]

    opts = [strategy: :one_for_one, name: MyApp.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

The tuple {OddJob, :email} will return a child spec for a supervisor that will start and supervise the :email pool. The second element of the tuple can be any atom that you want to use as a unique name for the pool. You can supervise as many pools as you want, as long as they have unique names.

All of the aforementioned config options can be combined. You can have a default pool (with an optional custom name), extra pools in the OddJob supervision tree, and pools to be supervised by your own application.

Basic usage and async/await

A job pool can be sent jobs by passing its unique name and an anonymous function to one of the OddJob module's perform functions:

job = OddJob.async_perform(:external_app, fn -> get_data(user) end)
# do something else
data = OddJob.await(job)
OddJob.perform(:email, fn -> send_email(user, data) end)

If a worker in the pool is available then the job will be performed right away. If all of the workers are already assigned to other jobs then the new job will be added to a FIFO queue. Jobs in the queue are performed as workers become available.

Use perform/2 for immediate fire and forget jobs where you don't care about the results or if it succeeds. async_perform/2 and await/1 follow the async/await pattern in the Task module, and are useful when you need to retrieve the results and you care about success or failure. Similarly to Task.async/1, async jobs will be linked and monitored by the caller (in this case, through a proxy). If either the caller or the job crash or exit, the other will crash or exit with the same reason.

Scheduled jobs

Jobs can be scheduled for later execution with perform_after/3 and perform_at/3:

OddJob.perform_after(1_000_000, :job, fn -> clean_database() end) # accepts a timer in milliseconds

time = ~T[03:00:00.000000]
OddJob.perform_at(time, :job, fn -> verify_work_is_done() end) # accepts a valid Time or DateTime struct

The scheduling functions return a unique timer reference which can be read with Process.read_timer/1 and cancelled with OddJob.cancel_timer/1, which will cancel execution of the job itself and cause the scheduler process to exit. When the timer is up the job will be sent to the pool and can no longer be aborted.

ref = OddJob.perform_after(5000, :job, fn -> :will_be_canceled end)

# somewhere else in your code
if some_condition() do
  OddJob.cancel_timer(ref)
end

Note that there is no guarantee that a scheduled job will be performed right away when the timer runs out. Like all jobs it is sent to the pool and if all workers are busy at that time then the job enters the queue to be performed when a worker is available.

For more usage, explore the documentation.

License

MIT - Copyright (c) 2022 M. Simon Borg

Link to this section Summary

Functions

Performs an async job that can be awaited on for the result.

Awaits on an async job and returns the results.

Awaits replies form multiple async jobs and returns them in a list.

Cancels a scheduled job.

Performs a fire and forget job.

Sends a job to the pool after the given timer has elapsed.

Sends a job to the pool at the given time.

A macro for creating jobs with an expressive DSL.

Returns the pid and state of the job pool.

Returns the ID of the job pool.

Returns the pid of the job pool's supervisor.

Returns the ID of the job pool's supervisor.

Returns a list of pids for the specified worker pool.

Link to this section Types

Specs

child_spec() :: %{
  id: atom(),
  start: {OddJob.Supervisor, :start_link, [atom()]},
  type: :supervisor
}

Specs

job() :: OddJob.Job.t()

Specs

pool() :: OddJob.Pool.t()

Link to this section Functions

Link to this function

async_perform(pool, fun)

View Source

Specs

async_perform(atom(), function()) :: job()

Performs an async job that can be awaited on for the result.

Functions like Task.async/1 and Task.await/2.

Examples

iex> job = OddJob.async_perform(:job, fn -> :math.exp(100) end)
iex> OddJob.await(job)
2.6881171418161356e43
Link to this function

await(job, timeout \\ 5000)

View Source

Specs

await(job(), timeout()) :: any()

Awaits on an async job and returns the results.

Examples

iex> OddJob.async_perform(:job, fn -> :math.log(2.6881171418161356e43) end)
...> |> OddJob.await()
100.0
Link to this function

await_many(jobs, timeout \\ 5000)

View Source

Specs

await_many([job()], timeout()) :: [any()]

Awaits replies form multiple async jobs and returns them in a list.

This function receives a list of jobs and waits for their replies in the given time interval. It returns a list of the results, in the same order as the jobs supplied in the jobs input argument.

If any of the job worker processes dies, the caller process will exit with the same reason as that worker.

A timeout, in milliseconds or :infinity, can be given with a default value of 5000. If the timeout is exceeded, then the caller process will exit. Any worker processes that are linked to the caller process (which is the case when a job is started with async_perform/2) will also exit.

This function assumes the jobs' monitors are still active or the monitor's :DOWN message is in the message queue. If any jobs have been demonitored, or the message already received, this function will wait for the duration of the timeout.

Examples

iex> job1 = OddJob.async_perform(:job, fn -> 2 ** 2 end)
iex> job2 = OddJob.async_perform(:job, fn -> 3 ** 2 end)
iex> [job1, job2] |> OddJob.await_many()
[4, 9]

Specs

cancel_timer(reference()) :: non_neg_integer() | false

Cancels a scheduled job.

timer_ref is the unique reference returned by perform_at/3 or perform_after/3. This function returns the number of milliseconds left in the timer when cancelled, or false if the timer already expired. If the return is false you can assume that the job has already been sent to the pool for execution.

NOTE: Cancelling the timer with this function ensures that the job is never executed and that the scheduler process is exited and not left "hanging". Using Process.cancel_timer/1 will also cancel execution, but may leave hanging processes. A hanging scheduler process will eventually timeout, but not until one second after the expiration of the original timer.

Examples

iex> ref = OddJob.perform_after(500, :work, fn -> :never end)
iex> time = OddJob.cancel_timer(ref)
iex> is_integer(time)
true

iex> ref = OddJob.perform_after(10, :work, fn -> :never end)
iex> Process.sleep(11)
iex> OddJob.cancel_timer(ref)
false

Specs

perform(atom(), function()) :: :ok

Performs a fire and forget job.

Examples

iex> parent = self()
iex> :ok = OddJob.perform(:job, fn -> send(parent, :hello) end)
iex> receive do
...>   msg -> msg
...> end
:hello
Link to this function

perform_after(timer, pool, fun)

View Source

Specs

perform_after(integer(), atom(), function()) :: reference()

Sends a job to the pool after the given timer has elapsed.

timer is an integer that indicates the number of milliseconds that should elapse before the job is sent to the pool. The timed message is executed under a separate supervised process, so if the caller crashes the job will still be performed. A timer reference is returned, which can be read with Process.read_timer/1 or cancelled with OddJob.cancel_timer/1.

Examples

timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end)
Process.read_timer(timer_ref)
#=> 2836 # time remaining before job is sent to the pool
OddJob.cancel_timer(timer_ref)
#=> 1175 # job has been cancelled

timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end)
Process.sleep(6000)
OddJob.cancel_timer(timer_ref)
#=> false # too much time has passed to cancel the job
Link to this function

perform_at(time, pool, fun)

View Source

Specs

perform_at(Time.t() | DateTime.t(), atom(), function()) :: reference()

Sends a job to the pool at the given time.

time can be a Time or a DateTime struct. If a Time struct is received, then the job will be done the next time the clock strikes the given time. The timer is executed under a separate supervised process, so if the caller crashes the job will still be performed. A timer reference is returned, which can be read with Process.read_timer/1 or canceled with OddJob.cancel_timer/1.

Examples

time = Time.utc_now() |> Time.add(600, :second)
OddJob.perform_at(time, :job, fn -> scheduled_job() end)
Link to this macro

perform_this(pool, list)

View Source (macro)

A macro for creating jobs with an expressive DSL.

perform_this accepts a do block instead of an anonymous function. You can pass a few different arguments and options to receive different results:

Arguments

  • :async - Passing the atom :async as the second argument before the do block is equivalent to calling async_perform/2
  • options - Pass keyword options as the second argument to access scheduling features. at: time is like perform_at/3, while after: timer is like perform_after/3.

Examples

You must import or require OddJob to use macros:

import OddJob

perform_this :work do
  some_work()
  some_other_work()
end

time = ~T[03:00:00.000000]
perform_this :work, at: time do
  scheduled_work()
end

perform_this :work, after: 5000, do: something_important()

perform_this :work, :async do
  get_data()
end
|> await()
Link to this macro

perform_this(pool, arg2, arg3)

View Source (macro)

Specs

pool(atom()) :: {pid(), pool()}

Returns the pid and state of the job pool.

Examples

iex> {pid, %OddJob.Pool{id: id}} = OddJob.pool(:job)
iex> is_pid(pid)
true
iex> id
:job_pool

Specs

pool_id(atom()) :: atom()

Returns the ID of the job pool.

Examples

iex> OddJob.pool_id(:job)
:job_pool

Specs

supervisor(atom()) :: pid()

Returns the pid of the job pool's supervisor.

There is no guarantee that the process will still be alive after the results are returned, as it could exit or be killed or restarted at any time. Use supervisor_id/1 to obtain the persistent ID of the supervisor.

Examples

OddJob.supervisor(:job)
#=> #PID<0.239.0>

Specs

supervisor_id(atom()) :: atom()

Returns the ID of the job pool's supervisor.

Examples

iex> OddJob.supervisor_id(:job)
:job_pool_sup

Specs

workers(atom()) :: [pid()]

Returns a list of pids for the specified worker pool.

There is no guarantee that the processes will still be alive after the results are returned, as they could exit or be killed at any time.

Examples

OddJob.workers(:job)
#=> [#PID<0.105.0>, #PID<0.106.0>, #PID<0.107.0>, #PID<0.108.0>, #PID<0.109.0>]