Scientist v0.2.0 Scientist.Experiment behaviour

A behaviour module for creating and running experiments.

An experiment contains all information about how your control and candidate functions operate, and how their observations are reported. Experiments include functionality for determining when they should run and how they behave when exceptions are thrown.

The macros exposed by Scientist are a thin wrapper around the functions in this module. If you would like to, you can use the corresponding functions to create and run your experiment.

In addition to the required callbacks, you can also define custom defaults and exception handling behaviour.

Custom Defaults

default_name/0 and default_context/0 determine the default name and context, respectively, of unconfigured experiments in your module.

Custom Exception Handling

raised/3 and thrown/3 determine how your experiment will handle exceptions during an operation specified by the user. They receive the experiment as well as the operation name and exception. When left unspecified, exceptions thrown during an operation will be unhandled by Scientist.

The following operations report exceptions:

  • :enabled
  • :compare
  • :clean
  • :ignore
  • :run_if

Summary

Functions

Adds fun to the experiment as a candidate

Adds fun to the experiment as the control

Adds a function to the experiment that is used to clean observed values

Adds a function to the experiment that is used to compare observations

Adds an ignore function to the experiment

Returns true if the given observations match

Runs the experiment

Returns the value of the experiment’s run_if function

Adds a function to the experiment that should only execute when the experiment is run

Adds a function to the experiment that is used to determine if it should run

Returns true if a mismatch should be ignored

Returns true if the experiment should run

Macros

Executes the given block, reporting exceptions

Callbacks

Returns true if the experiment should be run

Publish the result of an experiment

Functions

add_candidate(exp, name \\ "candidate", fun)

Adds fun to the experiment as a candidate.

Raises Scientist.DuplicateError if the experiment already has a candidate with name.

add_control(exp, fun)

Adds fun to the experiment as the control.

Raises Scientist.DuplicateError if the experiment already has a control.

clean_with(exp, cleaner)

Adds a function to the experiment that is used to clean observed values.

When handling observations, the result of cleaner will be available under cleaned_value.

If an exception is thrown in cleaner, it will be reported through the thrown and raised callbacks as operation :clean.

compare_with(exp, compare_fn)

Adds a function to the experiment that is used to compare observations.

If an exception is thrown in compare_fn, it will be reported through the thrown and raised callbacks as operation :compare.

ignore(exp, ignore_fn)

Adds an ignore function to the experiment.

The experiment will ignore a mismatch whenever this function returns true. There is no limit on the number of ignore functions that can be configured.

If an exception is thrown in ignore_fn, it will be reported through the thrown and raised callbacks as operation :ignore.

new(name \\ "#{__MODULE__}", opts \\ [])

Creates an experiment.

Creates an experiment with the name and opts.

The following options are available:

  • :module - The callback module to use, defaults to Scientist.Default.
  • :context - A map of values to be stored in an observation, defaults to %{}.
  • :raise_on_mismatches - If true, any mismatches in this experiment’s observations will raise a Scientist.MismatchError, defaults to false.
observations_match?(experiment, control, candidate)

Returns true if the given observations match.

This uses the experiment’s compare function, if any. If none is configured, ==/2 will be used.

Reports a :compare error to the callback module if an exception is caught.

run(exp, opts \\ [])

Runs the experiment.

If enabled?/0 or a configured run_if function return a falsey value, the experiment will not be run and only the control will be executed.

Raises Scientist.MissingControlError if the experiment has no control.

Raises Scientist.MismatchError if the experiment has mismatched observations and is configured with raise_on_mismatched: true.

run_if_allows?(experiment)

Returns the value of the experiment’s run_if function.

If the experiment has no run_if function configured, true is returned.

Reports a :run_if error to the callback module if an exception is caught.

set_before_run(exp, before_run)

Adds a function to the experiment that should only execute when the experiment is run.

set_run_if(exp, run_if_fn)

Adds a function to the experiment that is used to determine if it should run.

If this function returns false, the experiment will not be run.

If an exception is thrown in run_if_fn, it will be reported through the thrown and raised callbacks as operation :run_if.

should_ignore_mismatch?(exp, control, candidate)

Returns true if a mismatch should be ignored.

Runs each of the configured ignore functions in turn, ignoring a mismatch when any of them return true.

Reports an :ignore error to the callback module if an exception is caught.

should_run?(experiment)

Returns true if the experiment should run.

Reports an :enabled error to the callback module if an exception is caught.

Macros

guarded(exp, operation, list)

Executes the given block, reporting exceptions.

Executes block and calls thrown/3 or raised/3 with the given reason if the block throws or raises an exception.

Callbacks

enabled?()

Specs

enabled? :: Boolean | nil

Returns true if the experiment should be run.

If a falsey value is returned, the candidate blocks of the experiment will be ignored, only running the control.

publish(result)

Specs

publish(result :: %Scientist.Result{candidates: term, control: term, experiment: term, ignored: term, mismatched: term}) :: any

Publish the result of an experiment.