defmodule OddJob do @moduledoc """ Job pools for Elixir OTP applications, written in Elixir. Use OddJob when you want to limit concurrency of background processing in your Elixir app. One possible use case is forcing backpressure on calls to external APIs. ## Installation The package can be installed by adding `odd_job` to your list of dependencies in `mix.exs`: ```elixir def deps do [ {:odd_job, "~> 0.3.0"} ] end ``` ## Getting started OddJob automatically starts up a supervised job pool of 5 workers out of the box with no extra configuration. The default name of this job pool is `:job`, and it can be sent work in the following way: ```elixir OddJob.perform(:job, fn -> do_some_work() end) ``` The default pool can be customized in your config if you want to change the name or pool size: ```elixir config :odd_job, default_pool: :work, pool_size: 10 # this changes the size of all pools in your application, defaults to 5 ``` You can also add extra pools to be supervised by the OddJob application supervision tree: ```elixir config :odd_job, extra_pools: [:email, :external_app] ``` If you don't want OddJob to supervise any pools for you (including the default `:job` pool) then pass `false` to the `:default_pool` config key: ```elixir config :odd_job, default_pool: false ``` ## Supervising job pools To supervise your own job pools you can add a tuple in the form of `{OddJob, name}` (where `name` is an atom) directly to the top level of your application's supervision tree or any other list of child specs for a supervisor: ```elixir defmodule MyApp.Application do use Application def start(_type, _args) do children = [ {OddJob, :email}, {OddJob, :external_app} ] opts = [strategy: :one_for_one, name: MyApp.Supervisor] Supervisor.start_link(children, opts) end end ``` The tuple `{OddJob, :email}` will return a child spec for a supervisor that will start and supervise the `:email` pool. The second element of the tuple can be any atom that you want to use as a unique name for the pool. You can supervise as many pools as you want, as long as they have unique `name`s. All of the aforementioned config options can be combined. You can have a default pool (with an optional custom name), extra pools in the OddJob supervision tree, and pools to be supervised by your own application. ## Basic usage and async/await A job pool can be sent jobs by passing its unique name and an anonymous function to one of the `OddJob` module's `perform` functions: ```elixir job = OddJob.async_perform(:external_app, fn -> get_data(user) end) # do something else data = OddJob.await(job) OddJob.perform(:email, fn -> send_email(user, data) end) ``` If a worker in the pool is available then the job will be performed right away. If all of the workers are already assigned to other jobs then the new job will be added to a FIFO queue. Jobs in the queue are performed as workers become available. Use `perform/2` for immediate fire and forget jobs where you don't care about the results or if it succeeds. `async_perform/2` and `await/1` follow the async/await pattern in the `Task` module, and are useful when you need to retrieve the results and you care about success or failure. Similarly to `Task.async/1`, async jobs will be linked and monitored by the caller (in this case, through a proxy). If either the caller or the job crash or exit, the other will crash or exit with the same reason. ## Scheduled jobs Jobs can be scheduled for later execution with `perform_after/3` and `perform_at/3`: ```elixir OddJob.perform_after(1_000_000, :job, fn -> clean_database() end) # accepts a timer in milliseconds time = ~T[03:00:00.000000] OddJob.perform_at(time, :job, fn -> verify_work_is_done() end) # accepts a valid Time or DateTime struct ``` The scheduling functions return a unique timer reference which can be read with `Process.read_timer/1` and cancelled with `OddJob.cancel_timer/1`, which will cancel execution of the job itself *and* cause the scheduler process to exit. When the timer is up the job will be sent to the pool and can no longer be aborted. ```elixir ref = OddJob.perform_after(5000, :job, fn -> :will_be_canceled end) # somewhere else in your code if some_condition() do OddJob.cancel_timer(ref) end ``` Note that there is no guarantee that a scheduled job will be performed right away when the timer runs out. Like all jobs it is sent to the pool and if all workers are busy at that time then the job enters the queue to be performed when a worker is available. For more usage, explore the [documentation](https://hexdocs.pm/odd_job). ## License [MIT - Copyright (c) 2022 M. Simon Borg](https://github.com/msimonborg/odd_job/blob/main/LICENSE.txt) """ alias OddJob.{Async, Job, Pool, Scheduler} @type job :: Job.t() @type pool :: Pool.t() @type child_spec :: %{ id: atom, start: {OddJob.Supervisor, :start_link, [atom]}, type: :supervisor } @doc """ A macro for creating jobs with an expressive DSL. `perform_this` accepts a `do` block instead of an anonymous function. You can pass a few different arguments and options to receive different results: ## Arguments * `:async` - Passing the atom `:async` as the second argument before the `do` block is equivalent to calling `async_perform/2` * `options` - Pass keyword options as the second argument to access scheduling features. `at: time` is like `perform_at/3`, while `after: timer` is like `perform_after/3`. ## Examples You must `import` or `require` `OddJob` to use macros: import OddJob perform_this :work do some_work() some_other_work() end time = ~T[03:00:00.000000] perform_this :work, at: time do scheduled_work() end perform_this :work, after: 5000, do: something_important() perform_this :work, :async do get_data() end |> await() """ defmacro perform_this(pool, do: block) do quote do OddJob.perform(unquote(pool), fn -> unquote(block) end) end end defmacro perform_this(pool, [{key, val}, {:do, block}]) do quote do perform_this(unquote(pool), [{unquote(key), unquote(val)}], do: unquote(block)) end end defmacro perform_this(pool, [{key, val}], do: block) do quote do OddJob.unquote(:"perform_#{key}")(unquote(val), unquote(pool), fn -> unquote(block) end) end end defmacro perform_this(pool, :async, do: block) do quote do OddJob.async_perform(unquote(pool), fn -> unquote(block) end) end end @doc false @spec child_spec(atom) :: child_spec defdelegate child_spec(name), to: OddJob.Supervisor @doc """ Performs a fire and forget job. ## Examples iex> parent = self() iex> :ok = OddJob.perform(:job, fn -> send(parent, :hello) end) iex> receive do ...> msg -> msg ...> end :hello """ @spec perform(atom, function) :: :ok def perform(pool, fun) when is_atom(pool) and is_function(fun) do job = %Job{function: fun, owner: self()} GenServer.call(pool_id(pool), {:perform, job}) end @doc """ Performs an async job that can be awaited on for the result. Functions like `Task.async/1` and `Task.await/2`. ## Examples iex> job = OddJob.async_perform(:job, fn -> :math.exp(100) end) iex> OddJob.await(job) 2.6881171418161356e43 """ @spec async_perform(atom, function) :: job def async_perform(pool, fun) when is_atom(pool) and is_function(fun) do Async.perform(pool, fun) end @doc """ Awaits on an async job and returns the results. ## Examples iex> OddJob.async_perform(:job, fn -> :math.log(2.6881171418161356e43) end) ...> |> OddJob.await() 100.0 """ @spec await(job, timeout) :: any def await(job, timeout \\ 5000) when is_struct(job, Job) do Async.await(job, timeout) end @doc """ Awaits replies form multiple async jobs and returns them in a list. This function receives a list of jobs and waits for their replies in the given time interval. It returns a list of the results, in the same order as the jobs supplied in the `jobs` input argument. If any of the job worker processes dies, the caller process will exit with the same reason as that worker. A timeout, in milliseconds or `:infinity`, can be given with a default value of `5000`. If the timeout is exceeded, then the caller process will exit. Any worker processes that are linked to the caller process (which is the case when a job is started with `async_perform/2`) will also exit. This function assumes the jobs' monitors are still active or the monitor's :DOWN message is in the message queue. If any jobs have been demonitored, or the message already received, this function will wait for the duration of the timeout. ## Examples iex> job1 = OddJob.async_perform(:job, fn -> 2 ** 2 end) iex> job2 = OddJob.async_perform(:job, fn -> 3 ** 2 end) iex> [job1, job2] |> OddJob.await_many() [4, 9] """ @spec await_many([job], timeout) :: [any] def await_many(jobs, timeout \\ 5000) when is_list(jobs) do Async.await_many(jobs, timeout) end @doc """ Sends a job to the `pool` after the given `timer` has elapsed. `timer` is an integer that indicates the number of milliseconds that should elapse before the job is sent to the pool. The timed message is executed under a separate supervised process, so if the caller crashes the job will still be performed. A timer reference is returned, which can be read with `Process.read_timer/1` or canceled with `Process.cancel_timer/1`. ## Examples timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end) Process.read_timer(timer_ref) #=> 2836 # time remaining before job is sent to the pool Process.cancel_timer(timer_ref) #=> 1175 # job has been cancelled timer_ref = OddJob.perform_after(5000, :job, fn -> deferred_job() end) Process.sleep(6000) Process.cancel_timer(timer_ref) #=> false # too much time has passed to cancel the job """ @spec perform_after(integer, atom, function) :: reference def perform_after(timer, pool, fun) when is_integer(timer) and is_atom(pool) and is_function(fun) do Scheduler.perform_after(timer, pool, fun) end @doc """ Sends a job to the `pool` at the given `time`. `time` can be a `Time` or a `DateTime` struct. If a `Time` struct is received, then the job will be done the next time the clock strikes the given time. The timer is executed under a separate supervised process, so if the caller crashes the job will still be performed. A timer reference is returned, which can be read with `Process.read_timer/1` or canceled with `OddJob.cancel_timer/1`. ## Examples time = Time.utc_now() |> Time.add(600, :second) OddJob.perform_at(time, :job, fn -> scheduled_job() end) """ @spec perform_at(Time.t() | DateTime.t(), atom, function) :: reference def perform_at(time, pool, fun) when (is_struct(time, Time) or is_struct(time, DateTime)) and is_atom(pool) and is_function(fun) do Scheduler.perform_at(time, pool, fun) end @doc """ Cancels a scheduled job. `timer_ref` is the unique reference returned by `perform_at/3` or `perform_after/3`. This function returns the number of milliseconds left in the timer when cancelled, or `false` if the timer already expired. If the return is `false` you can assume that the job has already been sent to the pool for execution. NOTE: Cancelling the timer with this function ensures that the job is never executed *and* that the scheduler process is exited and not left "hanging". Using `Process.cancel_timer/1` will also cancel execution, but may leave hanging processes. A hanging scheduler process will eventually timeout, but not until one second after the expiration of the original timer. ## Examples iex> ref = OddJob.perform_after(500, :work, fn -> :never end) iex> time = OddJob.cancel_timer(ref) iex> is_integer(time) true iex> ref = OddJob.perform_after(10, :work, fn -> :never end) iex> Process.sleep(11) iex> OddJob.cancel_timer(ref) false """ @spec cancel_timer(reference) :: non_neg_integer | false def cancel_timer(timer_ref) when is_reference(timer_ref) do Scheduler.cancel_timer(timer_ref) end @doc """ Returns the pid and state of the job `pool`. ## Examples iex> {pid, %OddJob.Pool{id: id}} = OddJob.pool(:job) iex> is_pid(pid) true iex> id :job_pool """ @spec pool(atom) :: {pid, pool} def pool(pool) when is_atom(pool) do pool_id = pool_id(pool) state = pool_id |> Pool.state() pid = pool_id |> GenServer.whereis() {pid, state} end @doc """ Returns the ID of the job `pool`. ## Examples iex> OddJob.pool_id(:job) :job_pool """ @spec pool_id(atom) :: atom defdelegate pool_id(pool), to: OddJob.Supervisor @doc """ Returns the pid of the job `pool`'s supervisor. There is no guarantee that the process will still be alive after the results are returned, as it could exit or be killed or restarted at any time. Use `supervisor_id/1` to obtain the persistent ID of the supervisor. ## Examples OddJob.supervisor(:job) #=> #PID<0.239.0> """ @spec supervisor(atom) :: pid def supervisor(pool) when is_atom(pool) do pool |> supervisor_id() |> GenServer.whereis() end @doc """ Returns the ID of the job `pool`'s supervisor. ## Examples iex> OddJob.supervisor_id(:job) :job_pool_sup """ @spec supervisor_id(atom) :: atom defdelegate supervisor_id(pool), to: OddJob.Pool.Supervisor, as: :id @doc """ Returns a list of `pid`s for the specified worker pool. There is no guarantee that the processes will still be alive after the results are returned, as they could exit or be killed at any time. ## Examples OddJob.workers(:job) #=> [#PID<0.105.0>, #PID<0.106.0>, #PID<0.107.0>, #PID<0.108.0>, #PID<0.109.0>] """ @spec workers(atom) :: [pid] def workers(pool) when is_atom(pool) do {_, %{workers: workers}} = pool(pool) workers end end