View Source Approval (Approval v0.2.2)

Lightweight approval testing for Elixir

This package provides some utilities to help automatically generate reference images for test cases, review those reference images and compare snapshots to the reference images to look for regressions.

Summary

Functions

Sets up an approval test.

Functions

Sets up an approval test.

Takes the following arguments:

  • :snapshot - the data generated by your code. If the data represents an image, it should be given as File.read!(some_path).

  • :reference - the reference data with which the snapshots will be compared. Again, if it represents an image, it should be given as File.read!(some_path). This file doesn't need to exist. In fact, the whole point is that the approval macro will generate it the first time the test is run and you'll only have to review it later. If you already have a reference file, it won't be overwritten.

  • :reviewed (optional, default: false) - whether the reference data has been reviewed by the user or not

Support for approval testing, especially of code that generated image.

First, the test wouldn't have been manually reviewed:

import Approve

test "example test" do
  snapshot = File.write!("snapshot.png", create_new_image_data(...))

  approve snapshot: File.read!("snapshot.png"),
          reference: File.read!("reference.png"), # this file doesn't exist, it will be created
          reviewed: false # The user hasn't manually reviewed the reference yet.
                          # After it has been reviewed, this value should be
                          # updated to `true`
end

When you run the test above, the code will copy the snapshot into the reference. This creates the new "reference.png" file, which you must review manually. Further snapshots will be compared to this reference. Until you review the reference file, the test will fail. After reviewing the reference file, you should update the code above by replacing reviewed: false by reviewed: true, so that it reads:

import Approve

test "example test" do
  snapshot = File.write!("snapshot.png", create_new_image_data(...))

  approve snapshot: File.read!("snapshot.png"),
          reference: File.read!("reference.png"), # this file exists and has been reviewed
          reviewed: true # the user has reviewed the reference
end

If at any point the snapshot file becomes different from the reference, the test will fail and the approve macro will generate an HTML file next to the snapshot named "snapshot.png.diff.html" (in general it will be snapshot_path <> "diff.html"). That HTML file contains the old and new versions of the images side by side. In the middle it shows the difference between the two images.

If at any point you want to change the reference file (let's say you have found a bug in the code that generated the previous reference), you can simply add a new reference file manually.

This is an example diff.html file made from two images:

diff.html

Implementation notes

The most portable way of comparing two images is by running javascript inside a web browser, which is what we do here. One possible criticism is that the user needs to manually open the HTML files in a web browser to view the results, instead of having the test suite spin up a web app which would centralize all images and even allow one to approve changes from the web browser itself.

However, that implementation is not very portable and is quite complex. This implementation is optimized for simplicity, and it has settled on using the filesystem as the only source of persistence. The downside is the whole manual fiddling with files, but that is not too bad in practice

In the future, I plan to support other kinds of data besides images, and that is the goal of the somewhat verbose snapshot: File.read!(path) syntax, instead of the shorter snapshot: path alternative.