How Eyes compares checkpoints and baseline images

As the application being tested evolves, so do its tests. New checkpoints are added to test new functionality and checkpoints are removed because the application functionality or flow has changed.

Instead of having a static association between every checkpoint and a corresponding baseline image, Eyes compares a sequence of images captured by checkpoints to a sequence of baseline images, allowing for the fact that checkpoints may have been added or removed. Then, very much like a textual "diff" program, Eyes finds images that have been added or removed as well as images that have a corresponding baseline image and either match or mismatch. Eyes then enables you to both verify existing checkpoints and ensure that the set of baseline images is up to date.

Tests results and steps

When a visual UI test executes, Eyes verifies the entire sequence of checkpoints against the sequence of baseline images. The result of this verification - the test result - consists of a sequence of steps, each denoting a missing checkpoint, a new checkpoint, a checkpoint that matches its baseline counterpart, or one that differs from it.

For example, the baseline of a test may consist of the following five images:

Now, imagine that the application and test have been changed, and the test generates a different sequence of images. For example:

Comparing the two sequences, we can see that:

• Image B has been removed
• Image C has changed to C1
• Images F and G have been added
• Images A, D, and E have not changed and are in the same order

Eyes will detect these differences and generate a sequence of steps as follows:

Each step represents one of the four possible outcomes:

• Match: Eyes has found a baseline and checkpoint image that match each other.
• Diff: Eyes has found a baseline and checkpoint image that correspond to each other, but there are mismatches.
• Missing: Eyes has detected that an existing baseline image has no corresponding checkpoint image.
• New: Eyes has detected a new checkpoint image (i.e. an image in the captured sequence which has no corresponding image in the baseline).

Notes

While you can freely add and remove checkpoints, Eyes assumes that the order of captured images that are not new does not change. If you change the order of two checkpoints, then depending on the content of these checkpoints, Eyes could report two missing steps and two new steps, or it could report two steps with mismatches.

When you define a checkpoint in the code, you can optionally provide a name for that checkpoint. The name is intended to be descriptive so as to allow you to easily and conveniently navigate and search within the test results. Eyes does not take the name into account when matching checkpoint and baseline images. The name is optional, does not need to be unique, and you are free to change it without impacting the testability of the code.