View Source Mneme (Mneme v0.3.3)

/ni:mi:/ - Snapshot testing for Elixir ExUnit

Early days

Mneme is fairly new. Please feel free to submit any feedback, bugs, or suggestions as issues on Github. Thanks!

Hex.pm Docs CI

Snapshot tests assert that some expression matches a reference value. It's like a regular assert, except that the reference value is generated for you by Mneme.

Mneme follows in the footsteps of existing snapshot testing libraries like Insta (Rust), expect-test (OCaml), and assert_value (Elixir). Instead of simple value or string comparison, however, Mneme focuses on pattern matching.

a-brief-example

A brief example

Let's say you're working on a function that removes even numbers from a list:

test "drop_evens/1 should remove all even numbers from an enum" do
  auto_assert drop_evens(1..10)

  auto_assert drop_evens([])

  auto_assert drop_evens([:a, :b, 2, :c])
end

Notice that these assertions don't really assert anything yet. That's okay, because the first time you run mix test, Mneme will generate the patterns and prompt you with diffs. When you accept them, your test is updated for you:

test "drop_evens/1 should remove all even numbers from an enum" do
  auto_assert [1, 3, 5, 7, 9] <- drop_evens(1..10)

  auto_assert [] <- drop_evens([])

  auto_assert [:a, :b, :c] <- drop_evens([:a, :b, 2, :c])
end

The next time you run your tests, you won't receive prompts (unless something changes!), and these auto-assertions will act like a normal assert. If things do change, you're prompted again and can choose to accept and update the test or reject the change and let it fail.

a-brief-tour

A brief tour

To see Mneme in action without adding it to a project, you can download and run the standalone tour:

curl -o tour_mneme.exs https://raw.githubusercontent.com/zachallaun/mneme/main/examples/tour_mneme.exs
elixir tour_mneme.exs

quick-start

Quick start

  1. Add :mneme do your deps in mix.exs:

    defp deps do
      [
        {:mneme, ">= 0.0.0", only: :test}
      ]
    end
  2. Add :mneme to your :import_deps in .formatter.exs:

    [
      import_deps: [:mneme],
      inputs: ["{mix,.formatter}.exs", "{config,lib,test}/**/*.{ex,exs}"]
    ]
  3. Start Mneme right after you start ExUnit in test/test_helper.exs:

    ExUnit.start()
    Mneme.start()
  4. Add use Mneme wherever you use ExUnit.Case:

    defmodule MyTest do
      use ExUnit.Case, async: true
      use Mneme
    
      test "arithmetic" do
        # use auto_assert instead of ExUnit's assert - run this test
        # and delight in all the typing you don't have to do
        auto_assert 2 + 2
      end
    end

pattern-matching

Pattern matching

Snapshot testing is a powerful tool, allowing you to ensure that your code behaves as expected, both now and in the future. However, traditional snapshot testing can be brittle, breaking whenever there is any change, even ones that are inconsequential to what is being tested.

Mneme addresses this by introducing Elixir's pattern matching to snapshot testing. With pattern matching, tests become more flexible, only failing when a change affects the expected structure. This allows you to focus on changes that matter, saving time and reducing noise in tests.

To facilitate this, Mneme generates the patterns that you were likely to have written yourself. If a value contains some input variable in its structure, Mneme will try to use a pinned variable (e.g. ^date). If the value is an Ecto struct, Mneme will omit autogenerated fields like timestamps that are likely to change with every run. And if Mneme doesn't get it quite right, you can update the pattern yourself -- you won't be prompted unless the pattern no longer matches.

generated-patterns

Generated patterns

Mneme tries to generate match patterns that are equivalent to what a human (or at least a nice LLM) would write. Basic data types like strings, numbers, lists, tuples, etc. will be as you would expect.

Some values, however, do not have a literal representation that can be used in a pattern match. Pids are such an example. For those, guards are used:

auto_assert self()

# generates:

auto_assert pid when is_pid(pid) <- self()

Additionally, local variables can be found and pinned as a part of the pattern. This keeps the number of hard-coded values down, reducing the likelihood that tests have to be updated in the future.

test "create_post/1 creates a new post with valid attrs", %{user: user} do
  valid_attrs = %{title: "my_post", author: user}

  auto_assert create_post(valid_attrs)
end

# generates:

test "create_post/1 creates a new post with valid attrs", %{user: user} do
  valid_attrs = %{title: "my_post", author: user}

  auto_assert {:ok, %Post{title: "my_post", author: ^user}} <- create_post(valid_attrs)
end

In many cases, multiple valid patterns will be possible. Usually, the "simplest" pattern will be selected by default when you are prompted, but you can cycle through the options as well.

non-exhaustive-list-of-special-cases

Non-exhaustive list of special cases

  • Pinned variables are generated by default if a value is equal to a variable in scope.

  • Date and time values are written using their sigil representation.

  • Struct patterns only include fields that are different from the struct defaults.

  • Structs defined by Ecto schemas exclude primary keys, association foreign keys, and auto generated fields like :inserted_at and :updated_at. This is because these fields are often randomly generated and would fail on subsequent tests.

formatting

Formatting

Mneme uses Rewrite to update source code, formatting that code before saving the file. Currently, the Elixir formatter and FreedomFormatter are supported. If you do not use a formatter, the first auto-assertion will reformat the entire file.

continuous-integration

Continuous Integration

In a CI environment, Mneme will not attempt to prompt and update any assertions, but will instead fail any tests that would update. This behavior is enabled by the CI environment variable, which is set by convention by many continuous integration providers.

export CI=true

editor-support

Editor support

Guides for optional editor integration can be found here:

acknowledgements

Acknowledgements

Special thanks to:

  • What if writing tests was a joyful experience?, from the Jane Street Tech Blog, for inspiring this library.

  • Sourceror, a library that makes complex code modifications simple.

  • Rewrite, which provides the diff functionality present in Mneme.

  • Owl, which makes it much easier to build a pretty CLI.

  • Insta, a snapshot testing tool for Rust, whose great documentation provided an excellent reference for snapshot testing.

  • assert_value, an existing Elixir project that provides similar functionality. Thank you for paving the way!

configuration

Configuration

Mneme can be configured globally in your application config, usually config/test.exs.

config :mneme,
  defaults: [
    default_pattern: :last
  ]

Configuration can also be set at the module level with use Mneme, in a describe block with @mneme_describe or for an individual test using @mneme.

defmodule MyTest do
  use ExUnit.Case

  # reject all changes to auto-assertions by default
  use Mneme, action: :reject

  test "this test will fail" do
    auto_assert 1 + 1
  end

  describe "some describe block" do
    # accept all changes to auto-assertions in this describe block
    @mneme_describe action: :accept

    test "this will update without prompting" do
      auto_assert 2 + 2
    end

    # prompt for any changes in this test
    @mneme action: :prompt
    test "this will prompt before updating" do
      auto_assert 3 + 3
    end
  end
end

Configuration that is "closer to the test" will override more general configuration:

@mneme > @mneme_describe > use Mneme > config :mneme

The exception to this is the CI environment variable, which causes all updates to be rejected. See the "Continuous Integration" section for more info.

options

Options

  • :action - The action to be taken when an auto-assertion updates. Actions are one of :prompt, :accept, or :reject. If CI=true is set in environment variables, the action will always be :reject. The default value is :prompt.

  • :default_pattern - The default pattern to be selected if prompted to update an assertion. Can be one of :infer, :first, or :last. The default value is :infer.

  • :diff - Controls the diff engine used to display changes when an auto- assertion updates. If :semantic, uses a custom diff engine to highlight only meaningful changes in the value. If :text, uses the Myers Difference algorithm to highlight all changes in text. The default value is :semantic.

  • :diff_style - Controls how diffs are rendered when the :diff option is set to :semantic. If :side_by_side, old and new code will be rendered side-by-side if the terminal has sufficient space. If :stacked, old and new code will be rendered one on top of the other. The default value is :side_by_side.

  • :force_update (boolean/0) - Setting to true will force auto-assertions to update even when they would otherwise succeed. This can be especially helpful when adding new keys to maps or structs since a pattern like %{} would not normally prompt as the match still succeeds. The default value is false.

  • :target - The target output for auto-assertions. If :mneme, the expression will remain an auto-assertion. If :ex_unit, the expression will be rewritten as an ExUnit assertion. The default value is :mneme.

Link to this section Summary

Setup

Sets up Mneme configuration for the calling module and imports Mneme's assertion macros.

Starts Mneme to run auto-assertions as they appear in your tests.

Link to this section Setup

Link to this macro

__using__(opts)

View Source (macro)

Sets up Mneme configuration for the calling module and imports Mneme's assertion macros.

This call accepts all options described in the "Configuration" section above.

example

Example

defmodule MyTest do
  use ExUnit.Case
  use Mneme # <- add this

  test "..." do
    auto_assert ...
  end
end

Starts Mneme to run auto-assertions as they appear in your tests.

This will almost always be added to your test/test_helper.exs, just below the call to ExUnit.start():

# test/test_helper.exs
ExUnit.start()
Mneme.start()

options

Options

  • :restart (boolean) - Restarts Mneme if it has previously been started. This option enables certain IEx-based testing workflows that allow tests to be run without a startup penalty. Defaults to false.

Link to this section Assertions

Link to this macro

auto_assert(expression)

View Source (macro)

Pattern-generating variant of ExUnit.Assertions.assert/1.

examples

Examples

auto_assert generates assertions when tests run, issuing a terminal prompt before making any changes (unless configured otherwise).

auto_assert [1, 2] ++ [3, 4]

# after running the test and accepting the change
auto_assert [1, 2, 3, 4] <- [1, 2] ++ [3, 4]

If the match no longer succeeds, a warning and new prompt will be issued to update it to the new value.

auto_assert [1, 2, 3, 4] <- [1, 2] ++ [:a, :b]

# after running the test and accepting the change
auto_assert [1, 2, :a, :b] <- [1, 2] ++ [:a, :b]

Prompts are only issued if the pattern doesn't match the value, so that pattern can also be changed manually.

# this assertion succeeds, so no prompt is issued
auto_assert [1, 2, | _] <- [1, 2] ++ [:a, :b]

differences-from-exunit-assert

Differences from ExUnit assert

The auto_assert macro is meant to match assert very closely, but there are a few differences to note:

  • Pattern-matching assertions use the <- operator instead of the = match operator.

  • Unlike ExUnit's assert, auto_assert can match falsy values. The following are equivalent:

    falsy = nil
    auto_assert nil <- falsy
    assert falsy == nil
  • Guards can be added with a when clause, while assert would require a second assertion. For example:

    auto_assert pid when is_pid(pid) <- self()
    
    assert pid = self()
    assert is_pid(pid)
  • Bindings in an auto_assert are not available outside of that assertion. For example:

    auto_assert pid when is_pid(pid) <- self()
    pid # ERROR: pid is not bound

    If you need to use the result of the assertion, it will evaluate to the expression's value.

    pid = auto_assert pid when is_pid(pid) <- self()
    pid # pid is the result of self()
Link to this macro

auto_assert_raise(function)

View Source (since 0.3.0) (macro)

See auto_assert_raise/3.

Link to this macro

auto_assert_raise(exception, function)

View Source (since 0.3.0) (macro)

See auto_assert_raise/3.

Link to this macro

auto_assert_raise(exception, message, function)

View Source (since 0.3.0) (macro)

Pattern-generating variant of ExUnit.Assertions.assert_raise/3.

If the given function does not raise, the assertion will fail.

Like auto_assert/1, you will be prompted to automatically update the assertion if the raised raised exception changes.

examples

Examples

You can pass an anonymous function that takes no arguments and is expected to raise an exception.

auto_assert_raise fn ->
  some_call_expected_to_raise()
end

# after running the test and accepting changes
auto_assert_raise Some.Exception, fn ->
  some_call_expected_to_raise()
end

# optionally include the message
auto_assert_raise Some.Exception, "perhaps with a message", fn ->
  some_call_expected_to_raise()
end

A captured function of arity zero can also be used.

auto_assert_raise &some_call_expected_to_raise/0

# after running the test and accepting changes
auto_assert_raise Some.Exception, &some_call_expected_to_raise/0
Link to this macro

auto_assert_receive()

View Source (since 0.3.0) (macro)

See auto_assert_receive/2.

Link to this macro

auto_assert_receive(pattern)

View Source (since 0.3.0) (macro)

See auto_assert_receive/2.

Link to this macro

auto_assert_receive(pattern, timeout)

View Source (since 0.3.0) (macro)

Pattern-generating variant of ExUnit.Assertions.assert_receive/3.

timeout is in milliseconds and defaults to 100.

examples

Examples

Process.send_after(self(), {:some, :message}, 50)

auto_assert_receive()

# after running the test, messages appearing within 100ms
# will be available as options
auto_assert_receive {:some, :message}

A custom timeout can be specified as a second argument.

Process.send_after(self(), {:some, :message}, 150)

auto_assert_receive nil, 300

# messages appearing within 300ms will now appear as options
auto_assert_receive {:some, :message}, 300
Link to this macro

auto_assert_received()

View Source (since 0.3.0) (macro)

See auto_assert_received/1.

Link to this macro

auto_assert_received(pattern)

View Source (since 0.3.0) (macro)

Pattern-generating variant of ExUnit.Assertions.assert_received/2.

Similar to auto_assert_receive/2, except that the timeout is set to 0, so the expected message must already be in the current process' mailbox.

examples

Examples

send(self(), {:some, :message})

auto_assert_received()

# after running the test, messages in the current process
# inbox will be available as options
auto_assert_receive {:some, :message}