PropCheck - Property Testing v1.1.0 PropCheck.StateM.DSL behaviour View Source

This module provides a shallow DSL (domain specific langauge) in Elixir for property based testing of stateful systems.

The basic approach

Property based testing of stateful systems is different from ordinary property based testing. Instead of testing operations and their effects on the datastructure directly, we construct a model of system and generate a sequecence of commands operating on both, the model and the system. Then we check, that after each command step, the system has evolved accordingly to the model. This is same idea which is used in model checking and is sometimes called a bismulation.

After defining a model, we two phases during executing the property. In phase 1, the generators create a list of (symbolic) commands including their parameters to be run against the system under test (SUT). A state machine guides the generation of commands.

In phase 2, the commands are executed and the state machine checks that the SUT is in the same state as the state machine. If an invalid state is detected, then the command sequence is shrunk towards a shorter sequence serving then as counter examples.

This appraoch works exactly the same as with PropCheck.StateM and PropCheck.FSM. The main difference is the API, grouping pre- and postconditions, state transitions and argument generators around the commands of the SUT. This leads towards more logical locality compared to the former implementations. Quickcheck EQC has a similar appraoch for structuring their modern state machines.

The DSL

A state machine acting as a model of the SUT can be defined by focussing on states or on transitions. We focus here on the transitions. A transition is a command calling the SUT. Thefore the main phrase of the DSL is the defcommand macro.

defcommand :find do
  # define the rules for executing the find command here
end

Inside the command macro, we define all the rules which the command must obey. As an example, we discuss here as an example the slightly simplified command :find from test/cache_dsl_test.exs. The SUT is a cache implementation based on an ETS and the model is is based on a list of (key/value)-pairs. This example is derived from Fred Hebert’s PropEr Testing, Chapter 9

The find-command is a call the find/1 API function. Its arguments are generated by key(), which boils down to numeric values. The arguments for the command are defined by the function args(state) returning a list of generators. In our example, the arguments do not depend on the model state. Next, we need to define the execution of the command by defining function impl/n. This function takes as many arguments as args/1 has elements in the argument list. The impl-function allows to apply conversion of parameters and return values to ease the testing. A typical example is the conversion of an {:ok, value} tuple to only value which can simplify working with value.

defcommand :find do
  def impl(key), do: Cache.find(key)
  def args(_state), do: [key()]
end

After defining how a command is executed, we need to define in which state this is allowed. For this, we define function pre/2, taking the model state and the generated list of arguments to check whether this call is allowed in the current model state. In this particular example, find is always allowed, hence we return true without any further checking. This also the default implementation and the reason why the precondition is missing in the test file.

defcommand :find do
  def impl(key), do: Cache.find(key)
  def args(_state), do: [key()]
  def pre(_state, [_key]}), do: true
end

If the precondition is satisfied, the call can happen. After the call, the SUT can be in a different state and the model state must be updated according to the mapping of the SUT to the model. Function next/3 takes the state before the call, the list of rguments and the symbolic or dynamic result (depending on phase 1 or 2, respectively). next/3 returns the new model state. Since searching for a key in the cache does not modify the system nor the model state, nothing is to do. This is again the default implementation and thus dismissed in the test file.

defcommand :find do
  def impl(key), do: Cache.find(key)
  def args(_state), do: [key()]
  def pre(_state, [_key]}), do: true
  def next(old_state, _args, call_result), do: state
end

The missing part of the command definition is the post condition, checking that after calling the system in phase 2 the system is in the expected state compared the model. This check is implemented in function post/3, which again has a trivial default implementation for post conditions that are always true. In this example, we check if the call_result is {:error, :not_found}, then we also do not find the key in our model list entries. The other case is that if we a return value of {:ok, val}, then we also find the value via the key in our list of entries.

defcommand :find do
  def impl(key), do: Cache.find(key)
  def args(_state), do: [key()]
  def pre(_state, [_key]}), do: true
  def next(old_state, _args, _call_result), do: state
  def post(entries, [key], call_result) do
    case List.keyfind(entries, key, 0, false) do
        false       -> call_result == {:error, :not_found}
        {^key, val} -> call_result == {:ok, val}
    end
  end
end

This completes the DSL for command definitions.

Additional model elements

In addition to commands, we need to define the model itself. This is the ingenious part of stateful property based testing! The initial state of the model must be implemented as function initial_state/0. From this function, all model evolutions start. In our simplified cache example the initial model is an empty list:

def initial_state(), do: []

The commands are generated with the same frequency by default. Often, this is not appropriate, e.g. in the cache example we expect many more find then cache commands. Therefore, commands can have a weight, which is technically used inside a PropCheck.BasicTypes.frequency/1 generator. The weights are defined in function weight/2, taking the current model state and the command to be generated. The return value is an integer defining the frequency. In our cache example we want three times more find than other commands:

def weight(_state, :find),  do: 1
def weight(_state, :cache), do: 3
def weight(_state, :flush), do: 1

The property to test

The property to test the stateful system is for all systems more or less equal. We generate all commands via generator commands/1, which takes a module with callbacks as parameter. Inside the test, we first start the SUT, execute the commands with run_commands/1, stopping the SUT and evaluating the result of the executions as a boolean expression. This boolean expression can be adorned with further functions and macros to analyze the generated commands (via PropCheck.aggregate/2) or to inspect the history if a failure occurs (via PropCheck.when_fail/2). In the test cases, you find more examples of such adornments.

property "run the sequential cache", [:verbose] do
  forall cmds <- commands(__MODULE__) do
    Cache.start_link(@cache_size)
    execution = run_commands(cmds)
    Cache.stop()
    (execution.result == :ok)
  end
end

Link to this section Summary

Types

A value of type command denotes the execution of a symblic command and storing its result in a symbolic variable

The name of a command must be an atom

A dynamic state can be anything and appears only during phase 2

The history of command execution in phase 2 is stored in a history element. It contains the current dynamic state and the call to be made

The result of the command execution. It contains either the state of the failing precondition, the command’s return value of the failing postcondition, the exception values or :ok if everything is fine

The sequence of calls consists of state and symbolic calls

The combination of symbolic and dynamic states are required for functions which are used in both phases 1 and 2

A symbolic call is the typical mfa-tuple plus the indicator :call

A symbolic state can be anything and appears only during phase 1

Each result of a symbolic call is stored in a symbolic variable. Their values are opaque and can only used as whole

t()

The combined result of the test. It contains the history of all executed commands, the final state, the final result and the environment, mapping symbolic vars to their actual values. Everything is fine, if result is :ok

Functions

Takes a list of generated commands and returns a list of mfa-tuples. This can be used for aggregation of commands

Generates the command list for the given module

Defines a new command of the model

Runs the list of generated commands according to the model

Callbacks

The initial state of the state machine is computed by this callback

The optional weights for the command generation. It takes the current model state and returns a list of command/weight pairs. Commands, which are not allowed in a specific state, should be ommitted, since a frequency of 0 is not allowed

Link to this section Types

A value of type command denotes the execution of a symblic command and storing its result in a symbolic variable.

Link to this type command_name() View Source
command_name() :: atom

The name of a command must be an atom.

Link to this type dynamic_state() View Source
dynamic_state() :: any

A dynamic state can be anything and appears only during phase 2.

Link to this type history_event() View Source
history_event() :: {state_t, symbolic_call, {any, result_t}}

The history of command execution in phase 2 is stored in a history element. It contains the current dynamic state and the call to be made.

Link to this type result_t() View Source
result_t ::
  :ok |
  {:pre_condition, state_t} |
  {:post_condition, any} |
  {:exception, any} |
  {:ok, any}

The result of the command execution. It contains either the state of the failing precondition, the command’s return value of the failing postcondition, the exception values or :ok if everything is fine.

The sequence of calls consists of state and symbolic calls.

The combination of symbolic and dynamic states are required for functions which are used in both phases 1 and 2.

Link to this type symbolic_call() View Source
symbolic_call() :: {:call, module, atom, [any]}

A symbolic call is the typical mfa-tuple plus the indicator :call.

Link to this type symbolic_state() View Source
symbolic_state() :: any

A symbolic state can be anything and appears only during phase 1.

Link to this type symbolic_var() View Source
symbolic_var() :: {:var, pos_integer}

Each result of a symbolic call is stored in a symbolic variable. Their values are opaque and can only used as whole.

Link to this type t() View Source
t() :: %PropCheck.StateM.DSL{env: environment, history: [history_event], result: result_t, state: state_t}

The combined result of the test. It contains the history of all executed commands, the final state, the final result and the environment, mapping symbolic vars to their actual values. Everything is fine, if result is :ok.

Link to this section Functions

Link to this function command_names(cmds) View Source
command_names(cmds :: [command]) :: [mfa]

Takes a list of generated commands and returns a list of mfa-tuples. This can be used for aggregation of commands.

Link to this function commands(mod) View Source
commands(module) :: BasicTypes.type

Generates the command list for the given module

Link to this macro defcommand(name, list) View Source (macro)

Defines a new command of the model.

Inside the command, local functions define

  • how the command is executed (impl(...)). This is required.
  • how the arguments in the current model state are generated (args(state). The default is the empty list of arguments.
  • if the command is allowed in the current model state (pre(state, arg_list) :: bolean) This is true per default.
  • what the next state of the model is after the call (next(old_state, arg_list, result) :: new_state). The default implementaiton does not change the model state, sufficient for queries.
  • if the system under test is in the correct state after the call (post(old_state, arg_list, result) :: boolean). This is true in the default implementation.

These local functions inside the macro are effectively callbacks to guide and evolve the model state.

Link to this function run_commands(commands) View Source
run_commands([command]) :: t

Runs the list of generated commands according to the model.

Returns the result, the history and the final state of the model.

Link to this section Callbacks

Link to this callback initial_state() View Source
initial_state() :: symbolic_state

The initial state of the state machine is computed by this callback.

Link to this callback weight(symbolic_state, command_name) View Source (optional)
weight(symbolic_state, command_name) :: pos_integer

The optional weights for the command generation. It takes the current model state and returns a list of command/weight pairs. Commands, which are not allowed in a specific state, should be ommitted, since a frequency of 0 is not allowed.

def weight(state), do: [x: 1, y: 1, a: 2, b: 2]