gen_stage v0.8.0 Experimental.DynamicSupervisor behaviour

A supervisor that dynamically supervises and manages children.

A supervisor is a process which supervises other processes, called child processes. Different from the regular Supervisor, DynamicSupervisor was designed to start, manage and supervise these children dynamically.

Note: if you want to perform hot code upgrades, the DynamicSupervisor can only be used as the root supervisor in your supervision tree from Erlang 19 onwards.

Note: this module is currently namespaced under Experimental.DynamicSupervisor. You will need to alias Experimental.DynamicSupervisor before writing the examples below.

Example

Before we start our dynamic supervisor, let’s first build an agent that represents a stack. That’s the process we will start dynamically:

defmodule Stack do
  def start_link(state) do
    Agent.start_link(fn -> state end, opts)
  end

  def pop(pid) do
    Agent.get_and_update(pid, fn [h|t] -> {h, t} end)
  end

  def push(pid, h) do
    Agent.cast(pid, fn t -> [h|t] end)
  end
end

Now let’s start our dynamic supervisor. Similar to a regular supervisor, the dynamic supervisor expects a list of child specifications on start. Different from regular supervisors, this list must contain only one item. The child specified in the list won’t be started alongside the supervisor. Instead, the child specification will be used as a template for all future supervised children.

Let’s give it a try:

# Import helpers for defining supervisors
import Supervisor.Spec

# We are going to supervise the Stack server which
# will be started with a single argument [:hello]
# and the default name of :sup_stack.
children = [
  worker(Stack, [])
]

# Start the supervisor with our template
{:ok, sup} = DynamicSupervisor.start_link(children, strategy: :one_for_one)

With the supervisor up and running, let’s start our first child with DynamicSupervisor.start_child/2. start_child/2 expects the supervisor PID and a list of arguments. Let’s start our child with a default stack of [:hello]:

{:ok, stack} = DynamicSupervisor.start_child(sup, [[:hello]])

Now let’s use the stack:

Stack.pop(stack)
#=> :hello

Stack.push(stack, :world)
#=> :ok

Stack.pop(stack)
#=> :world

However, there is a bug in our stack agent. If we call :pop and the stack is empty, it is going to crash because no clause matches. Let’s try it:

Stack.pop(stack)
** (exit) exited in: GenServer.call(#PID<...>, ..., 5000)

Since the stack is being supervised, the supervisor will automatically start a new agent, with the same default stack of [:hello] we have specified before. However, if we try to access it with the same PID, we will get an error:

Stack.pop(stack)
** (exit) exited in: GenServer.call(#PID<...>, ..., 5000)
   ** (EXIT) no process

Remember, the agent process for the previous stack is gone. The supervisor started a new stack but it has a new PID. For now, let’s use DynamicSupervisor.which_children/1 to fetch the new PID:

[stack] = DynamicSupervisor.which_children(sup)
Stack.pop(stack) #=> :hello

In practice though, it is unlikely we would use children/1. When we are managing thousands to millions of processes, we must find more efficient ways to retrieve processes. We have a couple of options.

The first option is to ask if we really want the stack to be automatically restarted. If not, we can choose another restart mode for the worker. For example:

worker(Stack, [], restart: :temporary)

The :temporary option will tell the supervisor to not restart the worker when it exits. Read the “Exit reasons” section later on for more information.

The second option is to give a name when starting the Stack agent:

DynamicSupervisor.start_child(sup, [[:hello], [name: MyStack]])

Now whenever that particular agent is started or restarted, it will be registered with a MyStack name which we can use when accessing it:

Stack.pop(MyStack)
#=> [:hello]

And that’s it. If the stack crashes, another stack will be up and have registered itself with the name MyStack.

Module-based supervisors

In the example above, a supervisor was started by passing the supervision structure to start_link/2. However, supervisors can also be created by explicitly defining a supervision module:

defmodule MyApp.Supervisor do
  use DynamicSupervisor

  def start_link do
    DynamicSupervisor.start_link(__MODULE__, [])
  end

  def init([]) do
    children = [
      worker(Stack, [[:hello]])
    ]

    {:ok, children, strategy: :one_for_one}
  end
end

Note: differently from Supervisor, the DynamicSupervisor expects a 3-item tuple from init/1 and it does not use the supervise/2 function. The goal is to standardize both implementations in the long term.

You may want to use a module-based supervisor if you need to perform some particular action on supervisor initialization, like setting up an ETS table.

Strategies

Currently dynamic supervisors support a single strategy:

  • :one_for_one - if a child process terminates, only that process is restarted.

GenStage consumer

A DynamicSupervisor can be used as the consumer in a GenStage pipeline. A new child process will be started per event, where the event is appended to the arguments in the child specification.

A DynamicSupervisor can be attached to a producer by returning :subscribe_to from init/1 or explicitly with GenStage.sync_subscribe/3 and GenStage.async_subscribe/2.

Once subscribed, the supervisor will ask the producer for max_demand events and start child processes as events arrive. As child processes terminate, the supervisor will accumulate demand and request more events once min_demand is reached. This allows the DynamicSupervisor to work similar to a pool, except a child process is started per event. The minimum amount of concurrent children per producer is specified by min_demand and the maximum is given by max_demand.

Exit reasons

From the example above, you may have noticed that the transient restart strategy for the worker does not restart the child if it crashes with reason :normal, :shutdown or {:shutdown, term}.

So one may ask: which exit reason should I choose when exiting my worker? There are three options:

  • :normal - in such cases, the exit won’t be logged, there is no restart in transient mode and linked processes do not exit

  • :shutdown or {:shutdown, term} - in such cases, the exit won’t be logged, there is no restart in transient mode and linked processes exit with the same reason unless trapping exits

  • any other term - in such cases, the exit will be logged, there are restarts in transient mode and linked processes exit with the same reason unless trapping exits

Name Registration

A supervisor is bound to the same name registration rules as a GenServer. Read more about it in the GenServer docs.

Summary

Types

Options used by the start* functions

Functions

Returns a map containing count values for the supervisor

Callback implementation for Experimental.GenStage.init/1

Starts a child in the dynamic supervisor

Starts a supervisor with the given children

Starts a dynamic supervisor module with the given arg

Terminates the given child pid

Returns a list with information about all children

Callbacks

Callback invoked to start the supervisor and during hot code upgrades

Types

options()
options :: [registry: atom, name: Supervisor.name, strategy: Supervisor.Spec.strategy, max_restarts: non_neg_integer, max_seconds: non_neg_integer, max_dynamic: non_neg_integer | :infinity]

Options used by the start* functions

Functions

count_children(supervisor)
count_children(Supervisor.supervisor) :: %{specs: non_neg_integer, active: non_neg_integer, supervisors: non_neg_integer, workers: non_neg_integer}

Returns a map containing count values for the supervisor.

The map contains the following keys:

  • :specs - always 1 as dynamic supervisors have a single specification

  • :active - the count of all actively running child processes managed by this supervisor

  • :supervisors - the count of all supervisors whether or not the child process is still alive

  • :workers - the count of all workers, whether or not the child process is still alive

format_status(arg1, list)

Callback implementation for Experimental.GenStage.format_status/2.

init(arg)

Callback implementation for Experimental.GenStage.init/1.

start_child(supervisor, args)

Starts a child in the dynamic supervisor.

The child process will be started by appending the given list of args to the existing function arguments in the child specification.

If the child process starts, function returns {:ok, child} or {:ok, child, info}, the pid is added to the supervisor and the function returns the same value.

If the child process starts, function returns ignore, an error tuple or an erroneous value, or if it fails, the child is discarded and :ignore or {:error, error} where error is a term containing information about the error is returned.

start_link(children, options)

Starts a supervisor with the given children.

A strategy is required to be given as an option. Furthermore, the :max_restarts, :max_seconds, :max_dynamic and :subscribe_to values can be configured as described in the documentation for the init/1 callback.

The options can also be used to register a supervisor name. The supported values are described under the Name Registration section in the GenServer module docs.

Note that the dynamic supervisor is linked to the parent process and will exit not only on crashes but also if the parent process exits with :normal reason.

start_link(mod, args, opts \\ [])
start_link(module, any, [options]) :: Supervisor.on_start

Starts a dynamic supervisor module with the given arg.

To start the supervisor, the init/1 callback will be invoked in the given module, with arg passed to it. The init/1 callback must return a supervision specification which can be created with the help of the Supervisor.Spec module.

If the init/1 callback returns :ignore, this function returns :ignore as well and the supervisor terminates with reason :normal. If it fails or returns an incorrect value, this function returns {:error, term} where term is a term with information about the error, and the supervisor terminates with reason term.

The :name option can also be given in order to register a supervisor name, the supported values are described under the Name Registration section in the GenServer module docs.

terminate_child(supervisor, pid)
terminate_child(Supervisor.supervisor, pid) ::
  :ok |
  {:error, :not_found}

Terminates the given child pid.

If successful, the function returns :ok. If there is no such pid, the function returns {:error, :not_found}.

which_children(supervisor)
which_children(Supervisor.supervisor) :: [{:undefined, pid | :restarting, Supervisor.Spec.worker, Supervisor.Spec.modules}]

Returns a list with information about all children.

Note that calling this function when supervising a large number of children under low memory conditions can cause an out of memory exception.

This function returns a list of tuples containing:

  • id - as defined in the child specification but is always set to :undefined for dynamic supervisors

  • child - the pid of the corresponding child process or the atom :restarting if the process is about to be restarted

  • type - :worker or :supervisor as defined in the child specification

  • modules - as defined in the child specification

Callbacks

init(args)
init(args :: term) ::
  {:ok, [Supervisor.Spec.spec], options :: keyword} |
  :ignore

Callback invoked to start the supervisor and during hot code upgrades.

Options

  • :strategy - the restart strategy option. Only :one_for_one is supported by dynamic supervisors.

  • :max_restarts - the maximum amount of restarts allowed in a time frame. Defaults to 3 times.

  • :max_seconds - the time frame in which :max_restarts applies in seconds. Defaults to 5 seconds.

  • :max_dynamic - the maximum number of children started under the supervisor via start_child/2. Defaults to infinity children.

  • :subscribe_to - a list of producers to subscribe to. Each element represents the producer or a tuple with the producer and the subscription options. e.g. [Producer] or [{Producer, max_demand: 10, min_demand: 20}]