Loadex.Scenario (Loadex v0.1.0) View Source
A set of macros used to create Loadex scenarios.
What is a scenario?
TL;DR: Scenario is basically a load test case.
The common problem with load testing is that a synthtetic load doesn't really produce great results. Performance is measured, the application is deployed to production and then it crashes under much smaller load, than we generated using our favourite load testing tool.
The reason is that synthetic load is, well, synthetic. While testing a single REST endpoint probably may not be a problem, more complex workflows can run into such issues quite easily. It makes a lot of sense, then, for our load tests to reflect the real world use cases of the application we're testing. This may include aquiring a token for authentication, creating a persistent connection using a required protocol, executing some specific handshake or initialization etc.
It requires expressiveness usually associated with programming languages. Scenarios provide a way of describing complex workflows with Elixir code and libraries. They're then executed concurrently to generate substantial loads.
Creating a scenario
Setup
Each Loadex
scenario starts with a setup/1
macro. It defines how many workers need to be started and what data should they receive as an optional seed.
This can be done in one of two ways: either returning a Range
:
setup do
1..10
end
...or a Stream
of Loadex.Scenario.Spec
structs with an unique id
and a seed
for each scenario:
setup do
load_users_from_csv()
|> Stream.map(fn %User{id: id} = user ->
Loadex.Scenario.Spec.new(id, user)
end)
end
Scenario
This is where magic happens. Scenario's code is executed in a separate process for every element returned by the setup
.
This element, a seed, is given as a prameter to the scenario/2
macro:
defmodule ExampleScenario do
use Loadex.Scenario
setup do
load_users_from_csv()
|> Stream.map(fn %User{id: id} = user ->
Loadex.Scenario.Spec.new(id, user)
end)
end
scenario %User{login: login, password: password} do
token = AuthClient.get_token(login, password)
loop_after 2000, 10, _repetition do
ExternalServiceClient.generate_some_load(token)
end
end
end
This simple scenario above loads a bunch of users from a CSV during the setup
stage.
Then each user concurrently aquires a token and finally starts making calls, every two seconds and ten in total, to the external service we want to test.
Note that we're using the loop_after/4
macro instead of :timer.sleep/1
and Enum.each/2
or a list comprehension.
The reason for this is that our scenario is run in a process and iterating on a list (or Range
) and :timer.sleep/1
calls are blocking it.
Meanwhile, loop_after/4
is asynchronous to ensure the worker can receive and process messages.
While it may not be an issue in your case, it is strongly advised to use built-in helpers to ensure all the performance benefits, that using Elixir and OTP gives us. Please refer to their documentation for more details.
Teardown
If there's any setup you'd like to undo after your scenario finishes, teardown/2
is a place to do it:
defmodule ExampleScenario do
use Loadex.Scenario
setup do
load_users_from_csv()
|> Stream.map(fn %User{id: id} = user ->
ExternalServiceClient.create_account(user)
Loadex.Scenario.Spec.new(id, user)
end)
end
scenario %User{login: login, password: password} do
# do stuff...
end
teardown %User{} = user do
ExternalServiceClient.delete_account(user)
end
end
Link to this section Summary
Functions
Terminates the scenario. teardown/2
will be executed after this call.
A helper for creating an asynchronous, non-blocking loops using message-passing.
A helper for creating an asynchronous, non-blocking loops using message-passing.
Scenario's implementation.
Sets up the scenario.
Cleans up after a scenario.
Link to this section Functions
Terminates the scenario. teardown/2
will be executed after this call.
loop(iterations, hibernate_or_standby \\ :standby, match, list)
View Source (macro)Specs
loop( iterations :: non_neg_integer(), hibernate_or_standby :: execution_mode(), match :: match_pattern(), do_block() ) :: Macro.t()
A helper for creating an asynchronous, non-blocking loops using message-passing.
loop 10, iteration do
IO.puts("#{iteration}")
end
Params:
iterations
- how many times should the code in thedo
block be executedhibernate_or_standby
- (optional) allows you to hibernate the underlyingGenServer
between each pass. Defaults to:standby
match
- a match pattern. Currently only an iteration number is passed here
loop_after(time, how_many_times, hibernate_or_standby \\ :standby, match, list)
View Source (macro)Specs
loop_after( time :: non_neg_integer(), iterations :: non_neg_integer(), hibernate_or_standby :: execution_mode(), match :: match_pattern(), do_block() ) :: Macro.t()
A helper for creating an asynchronous, non-blocking loops using message-passing.
loop_after 100, 10, iteration do
IO.puts("#{iteration}")
end
Params:
time
- the delay beteewn each passiterations
- how many times should the code in thedo
block be executedhibernate_or_standby
- (optional) allows you to hibernate the underlyingGenServer
between each pass. Defaults to:standby
match
- a match pattern. Currently only an iteration number is passed here
Specs
Scenario's implementation.
A single seed element returned from setup/2
is passed as an argument.
As code inside this macro will be executed inside a concurrent process, using helpers provided by this module is strongly advised for operations such as loops, to prevent the process from blocking.
Specs
setup(do_block()) :: Macro.t()
Sets up the scenario.
Must return a Range
or a list of Loadex.Scenario.Spec
structs.
Each value will be passed as a seed to a separate process running the scenario.
This callback is executed in by the runner, before any scenario starts.
Specs
Cleans up after a scenario.
Is given a seed from setup/2
as a parameter.
This callback is executed by each individual scenario worker.