README

Copy Markdown

eta

Deterministic simulation testing for the BEAM.

(Said like the Greek letter η, AY-tuh or EE-tuh.)

When your project's tests run, eta transforms your code via an Erlang parse_transform so that it can be driven by a special deterministic scheduler and a virtual clock. You define a meaningful workload to exercise your system, and eta will help you inject faults to reach interesting behavior. When pathology is found, eta gives you a perfectly reproducible trace to debug. This all happens at test time; the code that gets deployed to prod is untouched.

What it does

  • Serializes scheduling. eta_sched suspends every process it owns and resumes exactly one at a time, letting it run until it blocks. The next process to run is decided by the seed.
  • Virtualizes time. eta_time is a clock that only moves when the driver moves it, which it does when nothing is runnable, jumping straight to the next deadline. A 30 second timeout costs nothing.
  • Records and replays. eta_run records every decision it made, and replay/3 follows that trace back exactly.
  • Shrinks failures. eta_shrink delta-debugs a failing trace down to the decisions that matter, then verifies the result still reproduces.
  • Explains failures. A trace says which process ran, not what it did. eta_log records both on one timeline, with your processes named, so a counterexample reads as a story instead of a column of integers.
  • Reads frozen state. eta_observe gets a suspended process's state out without asking it anything, which is the only way an invariant can inspect a system the scheduler has stopped.
  • Injects network faults. eta_net can be configured to automatically inject faults in message delivery on simulated Erlang nodes, and to deliver the events a real link or node failure produces: nodedown/nodeup signals, noconnection DOWNs for every monitor held across the break, and node death as a single operation.

Getting Started

A system under test provides some strictly deterministic callbacks (see eta_harness), and a run is one call, with a seed. In this example the seed is 7:

#{outcome := Outcome, trace := Trace} =
    eta_run:run(my_harness, #{seed => 7, max_ops => 25, max_steps => 20000, preload => [my_app]}).

When a run fails, shrink it:

#{trace := Minimal, verified := true} =
    eta_shrink:shrink(my_harness, Trace, #{seed => 7, max_ops => 25, preload => [my_app]}).

The seed produces the same trace each time, and the shrink reduces the size of that trace.

eta includes some simple distributed systems in order to illustrate DST:

  1. A two-phase commit implementation, which is used in eta's own tests.
  2. An ABD Qurom Register, that is written as an end-to-end walkthrough.
  3. A leader election by heartbeat lease, for which correctness relies heavily on the clock, unlike the first two examples.

Documentation

  1. What DST is. Discusses the kinds of bugs we're looking for, how the technique works, and how your project implementation needs to change to use it.
  2. Setting up a project. Includes build configuration, some first steps to remove nondeterminism, and getting the system running for the first time.
  3. A worked example. A simple two-phase commit implementation, with an included bug for demonstration.
  4. Writing a system under test. The behaviour callbacks and the practices around them.
  5. Gotchas and footguns. Nondeterminism can leak in from many sources. This page details some common pain points.
  6. A journey through DST. The whole thing end to end: an empty directory becomes a quorum register with a planted bug, and the bug becomes a minimized, replayable failure that explains itself.

docs/design.md is still a work in progress. Eventually, it will be the internal specification for the eta project itself.

Installation

We suggest setting up a separate build profile called dst.

def deps do
  [{:eta, "~> 0.1", only: :dst}]
end

and in rebar3, a dst profile can run as rebar3 as dst eunit.

Your modules don't include eta.hrl directly. They include a wrapper of your own, include/myapp_eta.hrl, which is what keeps eta out of every other build:

-ifdef(DST).
-include_lib("eta/include/eta.hrl").
-else.
-define(ETA_LABEL(Label), ok).
-define(ETA_LOG(Event), 0).
-endif.

Setting up a project has the whole configuration for both build tools, and the reasons for the design choices.

Status

Early and under development.

Known gaps, roughly by size

  • Limited Elixir support. The Erlang parse_transform is the mechanism for keeping your code deployable to prod without disruptive changes. Elixir support may come later.
  • One simulation per VM. eta_time and eta_log keep state in named ETS tables, so runs must be serial. Keep the tests async: false. This is one of the reasons we suggest running simulations from their own build profile, so the constraint stays off your ordinary suite.

AI full disclosure

  • This software is developed with strong assistance from LLMs and with humans leading the ideas, testing, and debugging. We say this openly because it shaped how the project was built. If you are not happy with AI-developed code, this software is not for you. This disclosure was adopted from antirez/ds4.
  • We strive to write and edit the documentation for human consumption. LLM-speak will eventually be rooted out in favor of imperfect human writing. Documentation generated wholly by LLMs must be annotated as such.

Etymology

The name is short for étalon. You may also think of it as the Erlang Trace Augur.

The eta in the logo is outlined from STIX General Italic, used under the SIL Open Font License.