Automated test maintenance of baseline variations

This article explains how the Eyes Automated test maintenance feature works together with the Baseline variations feature.

Baseline variations is an Eyes feature that allows you to define multiple reference images for a baseline step. This is useful when running Eyes tests on applications that are undergoing A/B testing where the checkpoint image may be intentionally different in different test runs.

Automated test maintenance is an Eyes features that automatically detects when actions taken by a Test Manager user and duplicates any action taken on similar steps. In the case of Baseline variations, steps are considered similar when they are assigned to the same baseline group. Checkpoints A point in the test code at which the screen image is grabbed and compared to a baseline image that corresponds to the check point. can be grouped by the user explicitly by assigning them a common variation group ID using the SDK, or Eyes can automatically detect checkpoints that are likely to be in the same variation group.

Automated test maintenance is triggered on Baseline variations when a user defines that a step is a new variation, or deletes an existing variation. Eyes identifies steps that are in the same variation group and duplicates the creation/deletion action for those steps as well.

Here are some examples where Automated test maintenance of the Baseline variations feature can simplify your workflow and make it more efficient:

You can read about Baseline variations in the article Working with baseline variations and about Automated test maintenance in the article Automated test maintenance.

The remainder of this article consists of the following sections:

  • Auto maintenance of variation groups. This section describes which Test Manager Baseline variations user actions trigger Automated test maintenance and provides links to the relevant Test Manager documentation.

How to define variation groups using the SDK

You can associate a checkpoint with a variation group by assigning it a unique string ID that represents that group. This is done by appending the checksettings$variationgroupid method to the class$eyes$check fluent API chain, as shown in the following snippet:


The "variationGroupID1" string parameter is the variation group ID and will be associated with all of the test result steps that result from executing this checkpoint. This may be a result of running the test multiple times or the result of running the test once using the Ultrafast Grid with multiple execution environments.

Typically a unique variation group Id is used for every checkpoint, and the multiple instances of the variation IDs come about from running the test under different execution environments orscenarios.

Cases where Eyes automatically defines variation groups

The recommended way to achieve a smooth and accurate variation group workflow is to define variation groups explicitly, as described in Automated test maintenance of baseline variations. Eyes also automatically groups together steps into a variation group when they meet all the following conditions:

  • The steps were created on a single test execution using the Ultrafast Grid.

  • The steps have not been explicitly assigned to a variation group.

  • The steps are part of a test with the same test name.

  • The steps have the same step name and:

    • The step name is unique within its test, or

    • The common step name is not unique within the test, but all of the steps share the same step index (the position of the step within the series of steps in the test).

Auto maintenance of variation groups

Automated test maintenance is a Test Manager feature that intelligently identifies steps in a batch with similar differences and duplicates actions that you take on a step on the steps found to be similar. You can then review these changes and undo or change the automatically taken actions.

When steps in multiple tests are assigned the same version group ID, they are considered similar and trigger Automated test maintenance for the following user actions:

  • Add a new variation

  • Remove variations

Automated test maintenance is introduced in the article Automated test maintenance. The following sub-topics provide more specific details:

Setting the automated test maintenance scope: Automated test maintenance always applies to steps in the current batch. Within the current batch, you can limit the scope further to given selected steps or tests, or to the "current step," which effectively disables Automated test maintenance.

Automated test maintenance notifications and messages: How the Test Manager indicates that it is looking for, and has found Automated test maintenance tasks.

Reviewing groups and their steps: How to review and, if necessary, change the actions taken by Automated test maintenance.