Reviewing test results and updating the baseline
After a test run completes, you will typically want to review the results to see which checkpoints matched baseline images and which did not, and take any necessary corrective action. The Eyes Test Manager provides tools to do the following tasks:
- View the match results for every step.
- Decide if a check with have failed or passed.
- If a mismatch was caused by an out-of-date baseline image, update the baseline image with the screenshot from the test.
- Remove baseline images that no longer correspond to an existing checkpoint and add baseline images for new checkpoints.
- Report and assign bugs to team members.
- Deal with special cases such as dynamic text (e.g. time of day), which is different on every run, and can cause mismatch detection.
Accessing the Eyes Test Manager
To see the results, log in to the Applitools cloud by entering the following link into your browser:
https://applitools.com/users/login
You will be prompted to enter your username and password. After you enter these, the browser will switch to the Eyes Test Manager.
Test Manager screen overview
The Eyes Test Manager screen is divided into two main areas. In the left panel, Test Manager displays information about the batches that were run by your team and allows you to select the “current” batch to review. On the right, Test Manager displays the tests and steps of the batch you selected. More specifically:
- The batch list panel on the left displays the most recently run batches. Clicking on any entry in the list will make that batch the "current batch".
- The batch management panel above the batch list provides tools to update, filter, and delete entries in the batch list.
- The main panel displays the tests in the current batch and the steps in each test.
- In the toolbar above the main panel, Test Manager provides buttons that allow you to select from a number of different views of this information. Each view displays a different subset of the available information. This allows you to focus on the information you need for your current task. Other tools are provided to delete tests, assign tests to team members, filter tests and steps, group tests by differences, and more.
- Above the main panel, Test Manager provides a summary of the results of the current batch and the progress of your review.
Test Manager screen details
The screenshot below is annotated with the main features available in the Test Manager
| 1 | Displays how many batches are currently available in the batch list. Provides controls to filter the set of batches you see in the batch list, delete batches, and update the batch list with the latest results. |
| 2 | The batch list, with the newest batches at the top of the list. Clicking any entry in the list will make it the current batch and load its information in the main panel. |
| 3 | Information about the currently loaded batch and the status of the review process (how many tests are passed, failed, remain unresolved and so on). |
| The toolbar has four parts: | |
| 4 | Operations on selected tests and steps. |
| 5 | Select which view (of the five available) is used to display the details of the currently selected batch in the main panel (8). |
| 6 | Buttons to reload the current batch, filter the current batch, and group steps by similar differences. |
| 7 | The Auto Maintenance menu used to select the Automated test maintenance scope and a toggle for enabling/disabling Group steps by similar differences. |
| 8 | The main panel displays the detailed batch, test, and step information. What is displayed in this panel depends on the view selected (5). |
| 9 | Provides access to the Notifications list, help, and a menu with administration options. |
10 |
Obtain a hyperlink to the current view that can be pasted and shared with other users. |
11 |
Save all changes you have made to the baseline and make them permanent (i.e. used in subsequent testing). |
12 |
Start a chat session with the Applitools support team. |
Reviewing and resolving a test run
The Test Manager provides multiple ways of viewing the test and step results. Each view focuses on different aspects of the results.
See
Test Manager Views.
In the description that follows, we will use the “Test details” view. You choose this view by clicking the 
button. Initially, you will see a screen similar to the one below with a list of the tests in the batch.
If you click on any of the tests, then it will expand, and you will see a series of thumbnail images that represent the “steps” of the test.
What is a “step"?
A step corresponds to a checkpoint in the test code. Eyes creates a step in the results for every new checkpoint that it discovers. If you remove a checkpoint in the code, the baseline image that corresponds to the removed checkpoint still exists in the baseline checkpoint. So, when you view the results, you will see steps that correspond to a baseline image and either match or don’t match, and you will also see steps that represent new checkpoints without baseline images or baseline images of deleted checkpoints. The small icons at the top left of each step indicate if there was a mismatch and what type (from left to right in the image above: missing checkpoint, match, mismatch, new checkpoint).
When you review the results, you must resolve two types of step mismatches:
- Steps whose baseline image was paired with a checkpoint but did not match visually (i.e. there is a visual mismatch).
- Steps for checkpoints without a corresponding baseline image or baseline images without a corresponding checkpoint image.
Accepting and rejecting steps
Eyes provides tools to either “Accept” or “Reject” a step. These icons appear when you hover the cursor over a step thumbnail. We will explain below exactly what accept and reject mean in the context of a particular step result. In general, they impact two things: the first is if the step is set to "passed" or "failed", and the second is whether the baseline image should be replaced by the new checkpoint image.
Note that in addition to the accept and reject commands, some other operations are also available.
Before going into more detail about what accept and reject do, let's first understand the difference in workflow between the very first run of a test and subsequent runs of that test.
Example: Reviewing a test run
The workflow for handling test results the first time you run a test is different from the workflow on subsequent runs. The first time a test is run, there are no baseline images, all the checkpoints are new, and they are automatically set to be the sequence of baseline images. Apart from reviewing the checkpoint images, you are not required to resolve anything. In subsequent runs, you need to analyze and resolve all the mismatches.
Reviewing the first run
After the test is run for the first time, it’s status is "new". The screenshot below shows what this looks like in the Eyes Test Manager. The 
label to the left of the test name indicates that the test is new.
The 
icon in the top left corner of the step thumbnail indicates that the step is a new checkpoint. In a new test, all the checkpoints are assumed to be correct, and each step has a "pass" status, as indicated by the green bar.
The
icon underneath the indicates that the checkpoint image captured during the test was accepted as a new baseline image.
On the first run of a test, when there is no existing baseline, by default, Eyes automatically adds the checkpoint image to the baseline. This means that in this case, you do not need to take any further action. However, it is good practice for you to review the step images and ensure they look as expected. If required by your workflow, you can change the default via the SDK save new tests method. With this option, all tests, including new test, will start with a status of “unresolved” for the test and all the steps until you have explicitly accepted or rejected them.For more information see Changing the Default Pass/Fail results fail
Reviewing subsequent runs
After the first test run, Eyes will have automatically created a baseline with all the checkpoint images captured during that first run. In subsequent test runs, Eyes compares the sequence of captured images to the sequence of images in the baseline, and there are four possible outcomes, as you can see in the following screenshot of a test result:
Note the four different symbols at the top left of each step thumbnail, and the green or orange bars to the left of the thumbnail. The green bar indicates that the test has passed, and the orange bar indicates that there are differences you need to resolve (i.e. investigate the cause of the mismatch and decide what to do about it).
The four possible match results when there is an existing baseline are as follows:
|
The checkpoint image matches the baseline image. In most cases, this is the expected result, hence the default green “pass” status. However, in some rare cases, a match can indicate a bug. An example of this is if a new feature was supposed to be displayed but is missing in the UI. In this case, you should reject the step and report the issue, then the step will be marked as failed (a red bar), and the baseline image will not be changed. On the next run, after the UI is fixed, the difference will be found and you can accept the image so that it is updated in the baseline. |
|
Eyes did not find a checkpoint image that corresponds to an existing baseline image. If the checkpoint was intentionally removed from the test, then you should accept the step and Test Manager will remove the image from the baseline and pass the step. If this is not intentional, then you can report a bug (in this case, presumably, the bug is in the test and not the application). |
|
The checkpoint image does not match the corresponding baseline image. The pink areas in the thumbnail indicate regions where differences were detected. Differences may arise for different reasons, and it is up to you to analyze the differences using the tools that Test Manager provides and take appropriate actions. Typical scenarios are:
|
|
There is a new checkpoint image with no corresponding baseline image. Typically, this is because a new checkpoint has been added, in which case you will accept the Step. Otherwise, you can reject it and report a bug (probably in the test code and not in the application). Note that we previously saw this icon in the case of a new test with a green bar. In the case of the first run of a test, all the steps are automatically accepted. In the case of a subsequent test with an existing baseline, new steps need to be resolved. |
Accept and reject status
After you click the accept or reject buttons, icons appear on the step thumbnail that indicate that the action has been completed. The status of the test is indicated by the color of the bar on the left – green if the step passed and red if the test failed. Eyes logs which steps have been reviewed (accepted or rejected) and which haven't, and this determines whether or not the test is considered resolved and fully reviewed.
Remember that it is only in the case of a new test that Eyes displays the accept icon automatically, indicating that Eyes has automatically accepted the step.
Test status and batch status
In addition to the step status that changes when the accept and reject buttons are clicked, Eyes also displays the status of each test and the overall status of the batch. These are updated automatically to reflect the overall state of the test and the batch. For details, see Test and Batch Status.
Updating the baseline
Any changes you make to test steps by accepting or rejecting them or by adding or removing annotations are automatically persisted. However, your changes will only apply to subsequent test runs after you explicitly save them to the baseline using the button.
Eyes allows you to save changes to the baseline even if some steps are unresolved. This is useful if you have resolved those steps and want to rerun the test even though other steps are still unresolved. Take care that such incomplete test resolutions do not introduce inconsistencies between your test and the updated baseline, especially where steps have been added or removed.
The Test Manager indicates that there are changes that have not been saved to the baseline in several ways:
- For the step, an asterisk (*) is added to the right of the step name
- For the test, an asterisk (*) is added to the right of the test name
- For the current Batch, an indication of the number of unsaved steps in all the tests of the batch
- For a batch in the batch list, the "Unsaved" label is added to the right of the batch status
- The
icon "pulses"
If you have selected one or more tests, then when you hover the cursor over the save button 
you are offered the following choices (see 1 in the image below):
- Save all. Choosing this option will save all the unsaved changes in the batch.
- Save selected tests. Choosing this option only saves tests whose checkboxes have been checked, even if they were not changed (see 2 in the image below).
After you save to the baseline you may see a message regarding expired variations. For more information regarding variation and these messages see Working with baseline variations.
For further information, see our video, A Quick Look at the Applitools Test Results Dashboard: