View Source OddJob (OddJob v0.3.2)
Job pools for Elixir OTP applications, written in Elixir.
Use OddJob when you want to limit concurrency of background processing in your Elixir app. One possible use case is forcing backpressure on calls to external APIs.
Installation
The package can be installed by adding odd_job
to your list of dependencies in mix.exs
:
def deps do
[
{:odd_job, "~> 0.3.2"}
]
end
Getting started
OddJob automatically starts up a supervised job pool of 5 workers out of the box with no extra
configuration. The default name of this job pool is :job
, and it can be sent work in the following way:
OddJob.perform(:job, fn -> do_some_work() end)
The default pool can be customized in your config if you want to change the name or pool size:
config :odd_job,
default_pool: :work,
pool_size: 10 # this changes the size of all pools in your application, defaults to 5
You can also add extra pools to be supervised by the OddJob application supervision tree:
config :odd_job,
extra_pools: [:email, :external_app]
If you don't want OddJob to supervise any pools for you (including the default :job
pool) then
pass false
to the :default_pool
config key:
config :odd_job, default_pool: false
Supervising job pools
To supervise your own job pools you can add a tuple in the form of {OddJob, name}
(where name
is an atom)
directly to the top level of your application's supervision tree or any other list of child specs for a supervisor:
defmodule MyApp.Application do
use Application
def start(_type, _args) do
children = [
{OddJob, :email},
{OddJob, :external_app}
]
opts = [strategy: :one_for_one, name: MyApp.Supervisor]
Supervisor.start_link(children, opts)
end
end
The tuple {OddJob, :email}
will return a child spec for a supervisor that will start and supervise
the :email
pool. The second element of the tuple can be any atom that you want to use as a unique
name for the pool. You can supervise as many pools as you want, as long as they have unique name
s.
All of the aforementioned config options can be combined. You can have a default pool (with an optional custom name), extra pools in the OddJob supervision tree, and pools to be supervised by your own application.
Basic usage and async/await
A job pool can be sent jobs by passing its unique name and an anonymous function to one of the OddJob
module's perform
functions:
job = OddJob.async_perform(:external_app, fn -> get_data(user) end)
# do something else
data = OddJob.await(job)
OddJob.perform(:email, fn -> send_email(user, data) end)
If a worker in the pool is available then the job will be performed right away. If all of the workers are already assigned to other jobs then the new job will be added to a FIFO queue. Jobs in the queue are performed as workers become available.
Use perform/2
for immediate fire and forget jobs where you don't care about the results or if it succeeds.
async_perform/2
and await/1
follow the async/await pattern in the Task
module, and are useful when
you need to retrieve the results and you care about success or failure. Similarly to Task.async/1
, async jobs
will be linked and monitored by the caller (in this case, through a proxy). If either the caller or the job
crash or exit, the other will crash or exit with the same reason.
Scheduled jobs
Jobs can be scheduled for later execution with perform_after/3
and perform_at/3
:
OddJob.perform_after(1_000_000, :job, fn -> clean_database() end) # accepts a timer in milliseconds
time = ~T[03:00:00.000000]
OddJob.perform_at(time, :job, fn -> verify_work_is_done() end) # accepts a valid Time or DateTime struct
The scheduling functions return a unique timer reference which can be read with Process.read_timer/1
and
cancelled with OddJob.cancel_timer/1
, which will cancel execution of the job itself and cause the scheduler process to exit. When the timer is up the job will be sent to the pool and can no longer be aborted.
ref = OddJob.perform_after(5000, :job, fn -> :will_be_canceled end)
# somewhere else in your code
if some_condition() do
OddJob.cancel_timer(ref)
end
Note that there is no guarantee that a scheduled job will be performed right away when the timer runs out. Like all jobs it is sent to the pool and if all workers are busy at that time then the job enters the queue to be performed when a worker is available.
License
Link to this section Summary
Functions
Performs an async job that can be awaited on for the result.
Awaits on an async job and returns the results.
Awaits replies form multiple async jobs and returns them in a list.
Cancels a scheduled job.
Performs a fire and forget job.
Sends a job to the pool
after the given timer
has elapsed.
Sends a job to the pool
at the given time
.
A macro for creating jobs with an expressive DSL.
A macro for creating jobs with an expressive DSL.
Returns the pid and state of the job pool
.
Returns the ID of the job pool
.
Returns the pid of the job pool
's supervisor.
Returns the ID of the job pool
's supervisor.
Returns a list of pid
s for the specified worker pool.
Link to this section Types
Link to this section Functions
Specs
Performs an async job that can be awaited on for the result.
Functions like Task.async/1
and Task.await/2
.
Examples
iex> job = OddJob.async_perform(:job, fn -> :math.exp(100) end)
iex> OddJob.await(job)
2.6881171418161356e43
Specs
Awaits on an async job and returns the results.
Examples
iex> OddJob.async_perform(:job, fn -> :math.log(2.6881171418161356e43) end)
...> |> OddJob.await()
100.0
Specs
Awaits replies form multiple async jobs and returns them in a list.
This function receives a list of jobs and waits for their replies in the given time interval.
It returns a list of the results, in the same order as the jobs supplied in the jobs
input argument.
If any of the job worker processes dies, the caller process will exit with the same reason as that worker.
A timeout, in milliseconds or :infinity
, can be given with a default value of 5000
. If the timeout
is exceeded, then the caller process will exit. Any worker processes that are linked to the caller process
(which is the case when a job is started with async_perform/2
) will also exit.
This function assumes the jobs' monitors are still active or the monitor's :DOWN message is in the message queue. If any jobs have been demonitored, or the message already received, this function will wait for the duration of the timeout.
Examples
iex> job1 = OddJob.async_perform(:job, fn -> 2 ** 2 end)
iex> job2 = OddJob.async_perform(:job, fn -> 3 ** 2 end)
iex> [job1, job2] |> OddJob.await_many()
[4, 9]
Specs
cancel_timer(reference()) :: non_neg_integer() | false
Cancels a scheduled job.
timer_ref
is the unique reference returned by perform_at/3
or perform_after/3
. This function
returns the number of milliseconds left in the timer when cancelled, or false
if the timer already
expired. If the return is false
you can assume that the job has already been sent to the pool
for execution.
NOTE: Cancelling the timer with this function ensures that the job is never executed and that
the scheduler process is exited and not left "hanging". Using Process.cancel_timer/1
will also
cancel execution, but may leave hanging processes. A hanging scheduler process will eventually
timeout, but not until one second after the expiration of the original timer.
Examples
iex> ref = OddJob.perform_after(500, :work, fn -> :never end)
iex> time = OddJob.cancel_timer(ref)
iex> is_integer(time)
true
iex> ref = OddJob.perform_after(10, :work, fn -> :never end)
iex> Process.sleep(11)
iex> OddJob.cancel_timer(ref)
false
Specs
Performs a fire and forget job.
Examples
iex> parent = self()
iex> :ok = OddJob.perform(:job, fn -> send(parent, :hello) end)
iex> receive do
...> msg -> msg
...> end
:hello
Specs
Sends a job to the pool
after the given timer
has elapsed.
timer
is an integer that indicates the number of milliseconds that should elapse before
the job is sent to the pool. The timed message is executed under a separate supervised process,
so if the caller crashes the job will still be performed. A timer reference is returned,
which can be read with Process.read_timer/1
or cancelled with OddJob.cancel_timer/1
.
Examples
timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end)
Process.read_timer(timer_ref)
#=> 2836 # time remaining before job is sent to the pool
OddJob.cancel_timer(timer_ref)
#=> 1175 # job has been cancelled
timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end)
Process.sleep(6000)
OddJob.cancel_timer(timer_ref)
#=> false # too much time has passed to cancel the job
Specs
perform_at(Time.t() | DateTime.t(), atom(), function()) :: reference()
Sends a job to the pool
at the given time
.
time
can be a Time
or a DateTime
struct. If a Time
struct is received, then
the job will be done the next time the clock strikes the given time. The timer is executed
under a separate supervised process, so if the caller crashes the job will still be performed.
A timer reference is returned, which can be read with Process.read_timer/1
or canceled with
OddJob.cancel_timer/1
.
Examples
time = Time.utc_now() |> Time.add(600, :second)
OddJob.perform_at(time, :job, fn -> scheduled_job() end)
A macro for creating jobs with an expressive DSL.
perform_this/2
works like perform/2
except it accepts a do
block instead of an anonymous function.
Examples
You must import
or require
OddJob
to use macros:
import OddJob
perform_this :work do
some_work()
some_other_work()
end
perform_this :work, do: something_hard()
A macro for creating jobs with an expressive DSL.
perform_this/3
accepts a single configuration option as the second argument that will control execution of
the job. The available options prvode the functionality of async_perform/2
, perform_at/3
,
and perform_after/3
.
Options
:async
- Passing the atom:async
as the second argument before thedo
block creates an async job that can be awaited on. Seeasync_perform/2
.at: time
- Use this option to schedule the job for a specifictime
in the future.time
must be a validTime
orDateTime
struct. Seeperform_at/3
.after: timer
- Use this option to schedule the job to perform after the giventimer
has elapsed.timer
must be in milliseconds. Seeperform_after/3
.
Examples
import OddJob
time = ~T[03:00:00.000000]
perform_this :work, at: time do
scheduled_work()
end
perform_this :work, after: 5000, do: something_important()
perform_this :work, :async do
get_data()
end
|> await()
iex> (perform_this :work, :async, do: 10 ** 2) |> await()
100
Specs
Returns the pid and state of the job pool
.
Examples
iex> {pid, %OddJob.Pool{id: id}} = OddJob.pool(:job)
iex> is_pid(pid)
true
iex> id
:job_pool
Specs
Returns the ID of the job pool
.
Examples
iex> OddJob.pool_id(:job)
:job_pool
Specs
Returns the pid of the job pool
's supervisor.
There is no guarantee that the process will still be alive after the results are returned,
as it could exit or be killed or restarted at any time. Use supervisor_id/1
to obtain
the persistent ID of the supervisor.
Examples
OddJob.supervisor(:job)
#=> #PID<0.239.0>
Specs
Returns the ID of the job pool
's supervisor.
Examples
iex> OddJob.supervisor_id(:job)
:job_pool_sup
Specs
Returns a list of pid
s for the specified worker pool.
There is no guarantee that the processes will still be alive after the results are returned, as they could exit or be killed at any time.
Examples
OddJob.workers(:job)
#=> [#PID<0.105.0>, #PID<0.106.0>, #PID<0.107.0>, #PID<0.108.0>, #PID<0.109.0>]