PropCheck - Property Testing v1.1.2 PropCheck.StateM.DSL behaviour View Source
This module provides a shallow DSL (domain specific langauge) in Elixir for property based testing of stateful systems.
The basic approach
Property based testing of stateful systems is different from ordinary property based testing. Instead of testing operations and their effects on the datastructure directly, we construct a model of system and generate a sequence of commands operating on both, the model and the system. Then we check, that after each command step, the system has evolved accordingly to the model. This is same idea which is used in model checking and is sometimes called a bismulation.
After defining a model, we two phases during executing the property. In phase 1, the generators create a list of (symbolic) commands including their parameters to be run against the system under test (SUT). A state machine guides the generation of commands.
In phase 2, the commands are executed and the state machine checks that the SUT is in the same state as the state machine. If an invalid state is detected, then the command sequence is shrunk towards a shorter sequence serving then as counter examples.
This approach works exactly the same as with PropCheck.StateM
and
PropCheck.FSM
. The main difference is the API, grouping pre- and postconditions,
state transitions and argument generators around the commands of the SUT. This
leads towards more logical locality compared to the former implementations.
Quickcheck EQC has a similar appraoch for structuring their modern state machines.
The DSL
A state machine acting as a model of the SUT can be defined by focussing on
states or on transitions. We focus here on the transitions. A transition is a
command calling the SUT. Thefore the main phrase of the DSL is the defcommand
macro.
defcommand :find do
# define the rules for executing the find command here
end
Inside the command
macro, we define all the rules which the command must
obey. As an example, we discuss here as an example the slightly simplified
command :find
from test/cache_dsl_test.exs
. The SUT is a cache
implementation based on an ETS and the model is is based on a list of
(key/value)-pairs. This example is derived from Fred Hebert’s PropEr Testing,
Chapter 9
The find
-command is a call the find/1
API function. Its arguments are
generated by key()
, which boils down to numeric values. The arguments for
the command are defined by the function args(state)
returning a list
of generators. In our example, the arguments do not depend on the model state.
Next, we need to define the execution of the command by defining function
impl/n
. This function takes as many arguments as args/1
has elements in
the argument list. The impl
-function allows to apply conversion of
parameters and return values to ease the testing. A typical example is the
conversion of an {:ok, value}
tuple to only value
which can simplify
working with value
.
defcommand :find do
def impl(key), do: Cache.find(key)
def args(_state), do: [key()]
end
After defining how a command is executed, we need to define in which state
this is allowed. For this, we define function pre/2
, taking the model state
and the generated list of arguments to check whether this call is
allowed in the current model state. In this particular example, find
is always
allowed, hence we return true without any further checking. This also the
default implementation and the reason why the precondition is missing
in the test file.
defcommand :find do
def impl(key), do: Cache.find(key)
def args(_state), do: [key()]
def pre(_state, [_key]}), do: true
end
If the precondition is satisfied, the call can happen. After the call, the SUT
can be in a different state and the model state must be updated according to
the mapping of the SUT to the model. Function next/3
takes the state before
the call, the list of rguments and the symbolic or dynamic result (depending
on phase 1 or 2, respectively). next/3
returns the new model state. Since
searching for a key in the cache does not modify the system nor the model
state, nothing is to do. This is again the default implementation and thus
dismissed in the test file.
defcommand :find do
def impl(key), do: Cache.find(key)
def args(_state), do: [key()]
def pre(_state, [_key]}), do: true
def next(old_state, _args, call_result), do: state
end
The missing part of the command definition is the post condition, checking
that after calling the system in phase 2 the system is in the expected state
compared the model. This check is implemented in function post/3
, which
again has a trivial default implementation for post conditions that are always
true. In this example, we check if the call_result
is {:error, :not_found}
,
then we also do not find the key in our model list entries
. The other case is
that if we a return value of {:ok, val}
, then we also find the value via
the key
in our list of entries
.
defcommand :find do
def impl(key), do: Cache.find(key)
def args(_state), do: [key()]
def pre(_state, [_key]}), do: true
def next(old_state, _args, _call_result), do: state
def post(entries, [key], call_result) do
case List.keyfind(entries, key, 0, false) do
false -> call_result == {:error, :not_found}
{^key, val} -> call_result == {:ok, val}
end
end
end
This completes the DSL for command definitions.
Additional model elements
In addition to commands, we need to define the model itself. This is the
ingenious part of stateful property based testing! The initial state
of the model must be implemented as function initial_state/0
. From this
function, all model evolutions start. In our simplified cache example the
initial model is an empty list:
def initial_state(), do: []
The commands are generated with the same frequency by default. Often, this
is not appropriate, e.g. in the cache example we expect many more find
then
cache
commands. Therefore, commands can have a weight, which is technically used
inside a PropCheck.BasicTypes.frequency/1
generator. The weights are defined
in callback function weight/1
, taking the current model state and returning
a map of command and frequency pairs to be generated. In our cache example
we want the find
command to appear three times more often than other commands:
def weight(_state), do: %{find: 1, cache: 3, flush: 1}
The property to test
The property to test the stateful system is for all systems more or less
equal. We generate all commands via generator commands/1
, which takes
a module with callbacks as parameter. Inside the test, we first start
the SUT, execute the commands with run_commands/1
, stopping the SUT
and evaluating the result of the executions as a boolean expression.
This boolean expression can be adorned with further functions and macros
to analyze the generated commands (via PropCheck.aggregate/2
) or to
inspect the history if a failure occurs (via PropCheck.when_fail/2
).
In the test cases, you find more examples of such adornments.
property "run the sequential cache", [:verbose] do
forall cmds <- commands(__MODULE__) do
Cache.start_link(@cache_size)
execution = run_commands(cmds)
Cache.stop()
(execution.result == :ok)
end
end
Link to this section Summary
Types
A value of type command
denotes the execution of a symblic command and
storing its result in a symbolic variable
The name of a command must be an atom
A dynamic state can be anything and appears only during phase 2
The history of command execution in phase 2 is stored in a history element. It contains the current dynamic state and the call to be made
The result of the command execution. It contains either the state of the failing
precondition, the command’s return value of the failing postcondition,
the exception values or :ok
if everything is fine
The sequence of calls consists of state and symbolic calls
The combination of symbolic and dynamic states are required for functions which are used in both phases 1 and 2
A symbolic call is the typical mfa-tuple plus the indicator :call
A symbolic state can be anything and appears only during phase 1
Each result of a symbolic call is stored in a symbolic variable. Their values are opaque and can only used as whole
The combined result of the test. It contains the history of all executed commands,
the final state, the final result and the environment, mapping symbolic
vars to their actual values. Everything is fine, if result
is :ok
Functions
Takes a list of generated commands and returns a list of mfa-tuples. This can be used for aggregation of commands
Generates the command list for the given module
Defines a new command of the model
Runs the list of generated commands according to the model
Callbacks
The initial state of the state machine is computed by this callback
The optional weights for the command generation. It takes the current
model state and returns a map of command/weight pairs. Commands,
which are not allowed in a specific state, should be ommitted, since
a frequency of 0
is not allowed
Link to this section Types
A value of type command
denotes the execution of a symblic command and
storing its result in a symbolic variable.
The name of a command must be an atom.
A dynamic state can be anything and appears only during phase 2.
history_event() :: {state_t, symbolic_call, {any, result_t}}
The history of command execution in phase 2 is stored in a history element. It contains the current dynamic state and the call to be made.
The result of the command execution. It contains either the state of the failing
precondition, the command’s return value of the failing postcondition,
the exception values or :ok
if everything is fine.
The sequence of calls consists of state and symbolic calls.
The combination of symbolic and dynamic states are required for functions which are used in both phases 1 and 2.
A symbolic call is the typical mfa-tuple plus the indicator :call
.
A symbolic state can be anything and appears only during phase 1.
Each result of a symbolic call is stored in a symbolic variable. Their values are opaque and can only used as whole.
t() :: %PropCheck.StateM.DSL{env: environment, history: [history_event], result: result_t, state: state_t}
The combined result of the test. It contains the history of all executed commands,
the final state, the final result and the environment, mapping symbolic
vars to their actual values. Everything is fine, if result
is :ok
.
Link to this section Functions
Takes a list of generated commands and returns a list of mfa-tuples. This can be used for aggregation of commands.
Generates the command list for the given module
Defines a new command of the model.
Inside the command, local functions define
- how the command is executed (
impl(...)
). This is required. - how the arguments in the current model state are generated (
args(state)
. The default is the empty list of arguments. - if the command is allowed in the current model state (
pre(state, arg_list) :: bolean
) This istrue
per default. - what the next state of the model is after the call (
next(old_state, arg_list, result) :: new_state
). The default implementaiton does not change the model state, sufficient for queries. - if the system under test is in the correct state after the call
(
post(old_state, arg_list, result) :: boolean
). This istrue
in the default implementation.
These local functions inside the macro are effectively callbacks to guide and evolve the model state.
Runs the list of generated commands according to the model.
Returns the result, the history and the final state of the model.
Link to this section Callbacks
The initial state of the state machine is computed by this callback.
weight(symbolic_state) :: %{required(command_name) => pos_integer}
The optional weights for the command generation. It takes the current
model state and returns a map of command/weight pairs. Commands,
which are not allowed in a specific state, should be ommitted, since
a frequency of 0
is not allowed.
def weight(state), do: %{x: 1, y: 1, a: 2, b: 2}