Loadex.Scenario (Loadex v0.2.0) View Source

A set of macros used to create Loadex scenarios.

What is a scenario?

TL;DR: Scenario is basically a load test case.

The common problem with load testing is that a synthtetic load doesn't really produce great results. Performance is measured, the application is deployed to production and then it crashes under much smaller load, than we generated using our favourite load testing tool.

The reason is that synthetic load is, well, synthetic. While testing a single REST endpoint probably may not be a problem, more complex workflows can run into such issues quite easily. It makes a lot of sense, then, for our load tests to reflect the real world use cases of the application we're testing. This may include aquiring a token for authentication, creating a persistent connection using a required protocol, executing some specific handshake or initialization etc.

It requires expressiveness usually associated with programming languages. Scenarios provide a way of describing complex workflows with Elixir code and libraries. They're then executed concurrently to generate substantial loads.

Creating a scenario

Setup

Each Loadex scenario starts with a setup/1 macro. It defines how many workers need to be started and what data should they receive as an optional seed. This can be done in one of two ways: either returning a Range:

setup do
  1..10
end

...or a Stream of Loadex.Scenario.Spec structs with an unique id and a seed for each scenario:

setup do
  load_users_from_csv()
  |> Stream.map(fn %User{id: id} = user ->
    Loadex.Scenario.Spec.new(id, user)
  end)
end

Scenario

This is where magic happens. Scenario's code is executed in a separate process for every element returned by the setup. This element, a seed, is given as a prameter to the scenario/2 macro:

defmodule ExampleScenario do
  use Loadex.Scenario

  setup do
    load_users_from_csv()
    |> Stream.map(fn %User{id: id} = user ->
      Loadex.Scenario.Spec.new(id, user)
    end)
  end

  scenario %User{login: login, password: password} do
    token = AuthClient.get_token(login, password)

    loop_after 2000, 10, _repetition do
      ExternalServiceClient.generate_some_load(token)
    end
  end
end

This simple scenario above loads a bunch of users from a CSV during the setup stage. Then each user concurrently aquires a token and finally starts making calls, every two seconds and ten in total, to the external service we want to test.

Note that we're using the loop_after/4 macro instead of :timer.sleep/1 and Enum.each/2 or a list comprehension. The reason for this is that our scenario is run in a process and iterating on a list (or Range) and :timer.sleep/1 calls are blocking it. Meanwhile, loop_after/4 is asynchronous to ensure the worker can receive and process messages.

While it may not be an issue in your case, it is strongly advised to use built-in helpers to ensure all the performance benefits, that using Elixir and OTP gives us. Please refer to their documentation for more details.

Teardown

If there's any setup you'd like to undo after your scenario finishes, teardown/2 is a place to do it:

defmodule ExampleScenario do
  use Loadex.Scenario

  setup do
    load_users_from_csv()
    |> Stream.map(fn %User{id: id} = user ->
      ExternalServiceClient.create_account(user)

      Loadex.Scenario.Spec.new(id, user)
    end)
  end

  scenario %User{login: login, password: password} do
    # do stuff...
  end

  teardown %User{} = user do
    ExternalServiceClient.delete_account(user)
  end
end

Link to this section Summary

Functions

Terminates the scenario. teardown/2 will be executed after this call.

A helper for creating an asynchronous, non-blocking loops using message-passing.

A helper for creating an asynchronous, non-blocking loops using message-passing.

Scenario's implementation.

Sets up the scenario.

Cleans up after a scenario.

Allows user to act upon receiving a specific message.

Link to this section Functions

Link to this macro

end_scenario()

View Source (macro)

Terminates the scenario. teardown/2 will be executed after this call.

Link to this macro

loop(iterations, hibernate_or_standby \\ :standby, match, list)

View Source (macro)

Specs

loop(
  iterations :: non_neg_integer(),
  hibernate_or_standby :: execution_mode(),
  match :: match_pattern(),
  do_block()
) :: Macro.t()

A helper for creating an asynchronous, non-blocking loops using message-passing.

loop 10, iteration do
  IO.puts("#{iteration}")
end

Params:

  • iterations - how many times should the code in the do block be executed
  • hibernate_or_standby - (optional) allows you to hibernate the underlying GenServer between each pass. Defaults to :standby
  • match - a match pattern. Currently only an iteration number is passed here
Link to this macro

loop_after(time, how_many_times, hibernate_or_standby \\ :standby, match, list)

View Source (macro)

Specs

loop_after(
  time :: non_neg_integer(),
  iterations :: non_neg_integer(),
  hibernate_or_standby :: execution_mode(),
  match :: match_pattern(),
  do_block()
) :: Macro.t()

A helper for creating an asynchronous, non-blocking loops using message-passing.

loop_after 100, 10, iteration do
  IO.puts("#{iteration}")
end

Params:

  • time - the delay beteewn each pass
  • iterations - how many times should the code in the do block be executed
  • hibernate_or_standby - (optional) allows you to hibernate the underlying GenServer between each pass. Defaults to :standby
  • match - a match pattern. Currently only an iteration number is passed here
Link to this macro

scenario(seed, list)

View Source (macro)

Specs

scenario(seed :: any(), block :: do_block()) :: Macro.t()

Scenario's implementation.

A single seed element returned from setup/2 is passed as an argument.

As code inside this macro will be executed inside a concurrent process, using helpers provided by this module is strongly advised for operations such as loops, to prevent the process from blocking.

Specs

setup(do_block()) :: Macro.t()

Sets up the scenario.

Must return a Range or a list of Loadex.Scenario.Spec structs. Each value will be passed as a seed to a separate process running the scenario.

This callback is executed in by the runner, before any scenario starts.

Link to this macro

teardown(seed, list)

View Source (macro)

Specs

teardown(seed :: any(), block :: do_block()) :: Macro.t()

Cleans up after a scenario.

Is given a seed from setup/2 as a parameter.

This callback is executed by each individual scenario worker.

Link to this macro

wait_for(match, list)

View Source (macro)

Specs

wait_for(match :: match_pattern(), do_block()) :: Macro.t()

Allows user to act upon receiving a specific message.

wait_for {:msg, message} do
  IO.puts("{message}")
end

While this macro has blocking semantics, in a sense it will execute blocks for consecutive calls one after another, it doesn't actually block the process. This means any code placed after wait_for/2 will be executed immediately. For blocking behaviour, use receive/1.

Params:

  • match - a match pattern for a specific message