How To Handle Visual Testing False Positives – Part 1

A Solution

With some simple adjustments to our test code we can account for these issues head on.

Let’s dig in with some examples.

An Example

In Getting Started with Visual Testing write-up we stepped through an example where an element moves to the left or right by a large margin (e.g., 20+ pixels). This is an obvious visual defect.

Let’s use the same test code and point it at an example where the element only moves a single pixel and see what happens.

var assert = require('assert');
var driver = require('webdriverio').remote({
  desiredCapabilities: {
    browserName: 'firefox'
  }
});
require('webdrivercss').init(driver, {
  updateBaseline: true
  }
);
driver
  .init()
  .url('http://the-internet.herokuapp.com/shifting_content
/menu?mode=random&pixel_shift=1')
  .webdrivercss('body', {
    name: 'body',
    elem: 'body'
  }, function(err,res) {
      assert.ifError(err);
    })
  .end();

In case you missed the last write-up, this test creates an instance of Selenium, visits the page specified, grabs a screenshot of the page, and compares it to an initial baseline image. If there is a discrepancy between them, the test will fail. For a more detailed explanation of what’s happening, read this.

If we run this test (e.g., node false_positives.js from the command-line), it will see the single pixel offset and fail. That’s because the mismatch tolerance (the thing that WebdriverCSS uses to determine if there is a visual anomaly) is too sensitive by default for such subtle shifts.

To address this, we can increase the mismatch tolerance from it’s sensible default (e.g., from 0.05 to 0.20).

# filename: false_positives.js
...
require('webdrivercss').init(driver, {
  updateBaseline: true,
  misMatchTolerance: 0.20
  }
...

Keep in mind that changing the mismatch tolerance introduces risk into our test. Raising the mismatch tolerance will solve our immediate issue, but it can introduce other issues by creating a possible gap in our coverage. For example, if a page we’re testing is corrupt within this new tolerance (e.g., typo, missing icon, etc.), then our test could miss it. So use this sparingly, and with caution.

Alternatively, we could leave the mismatch tolerance alone and narrow our test’s focus to just look at the element that’s moving (instead of the page body). This would enable us to test the element regardless of it’s location, but it would strip us of benefit of overall page coverage — so let’s not do that. The best approach would be to use a more sophisticated image comparison engine (which I’ll cover in the next write-up).

Since we’re dealing with a reasonably small shift in the mismatch tolerance (e.g., raising it 0.05% to 0.20%) let’s stick with our tweak, save our file, and run it (e.g., node false_positives.js from the command-line). Now our test will run without getting tripped up by single pixel offsets. And if we point the test back at the original example (with the element that moves by 20+ pixels) the test will still catch the visual bug.

# filename: false_positives.js
...
driver
  .init()
  .url('http://the-internet.herokuapp.com/shifting_content
/menu?mode=random')

Another Example

Now if we take the same test code and point at something else (e.g., an example that loads content dynamically), it will fail.

# filename: false_positives.js
...
driver
  .init()
  .url('http://the-internet.herokuapp.com/dynamic_content')
...

We accounted for pixel offsets by adjusting the mismatch tolerance. But the elements in this new example don’t shift. It fails because the contents of the elements are changing on each page load. To account for this, we will need to selectively ignore pieces of the page. Specifically, the pieces of the page with dynamic content.

This can be done one of two ways. We can either specify an element (e.g., with a CSS selector), or use X and Y coordinates of the page. The latter approach works if you have a page with limited locators to choose from. Whereas using locators is the simplest and most scalable way. Fortunately, our example has good semantic markup, so let’s use locators.

# filename: false_positives.js
...
driver
  .init()
  .url('http://the-internet.herokuapp.com/dynamic_content')
  .webdrivercss('body', {
    name: 'body',
    elem: 'body',
    exclude: ['#content > .row']
...

With the exclude: parameter we can specify more than one element. But we just need to exclude each of the content rows, which we can easily do with the #content > .row CSS selector.

Now when we run our test (e.g., node false_positives.js from the command-line), it will pass. You can see a screenshot of what it looks like with the exclusion here:

 

One More Example

If we want to run this test against different page sizes to see if it holds up, it’s a simple matter of adding an additional initialization parameter for the different screen widths (e.g., screenWidth).

# filename: false_positives.js
...
    exclude: ['#content > .row'],
    screenWidth: [320,640,960]
...
“`

If we save this and run it (e.g., node false_positives.js from the command-line), the test will resize the browser for each of the screen widths specified, exclude the elements with dynamic content, and then perform visual checks for each of the sizes using the mismatch tolerance we specified — and pass.

In Conclusion…

Now that we’ve stepped through the basics of automated visual testing and some common issues that you will run into, we’re ready to dig into some of the more interesting bits of automated visual testing.

Stay tuned!

Read Dave’s next post in this series: How to Handle False Positives in Visual Testing – Part 2

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

 

 

Dave Haeffner is the writer of Elemental Selenium – a free, once weekly Selenium tip newsletter read by thousands of testing professionals. He’s also the creator and maintainer of ChemistryKit (an open-source Selenium framework), and author of The Selenium Guidebook. He’s helped numerous companies successfully implement automated acceptance testing; including The Motley Fool, ManTech International, Sittercity, Animoto, and Aquent. He’s also a founder/co-organizer of the Selenium Hangout, and has spoken at numerous conferences and meet-ups about automated acceptance testing.