How Do I Validate Visually?

How does anyone maintain their functional test automation infrastructure without missing visual errors?

When have you had all your functional tests pass and have something like this end up in production?

Yes, Instagram tests their code. Yet, this outcome showed up on a user’s device.

Why?

Because you build tests to check assertions of element locators in HTML, and you don’t check the rendered page. If the assertions continue to pass, these errors will sneak up on you. Do you know how much you spend maintaining your assertion infrastructure – and do you still miss these kinds of functional errors?

What if you could get comprehensive functional testing without all that coding and maintenance effort?

Isn’t there a better way?

Introduction to Angie Jones’s Automated Visual Testing

I took Angie Jones’s course, “Automated Visual Testing: A Fast Path To Test Automation Success” to understand how she sees visual as a component of test automation. Angie makes it clear how straightforward it is to write visual assertions with Applitools. And, she made it clear how much time you save when you use visual assertions for your functional tests.

Angie’s course is great. She includes code examples on Github, and she links to relevant web applications you can use in your own testing to execute the same tests yourself. Also, she includes links to Applitools tutorials and other pages that give you a sense of how Applitools works, and how easily you can add Applitools to your environment.

Let me give you a chapter-by-chapter summary so you can get the gist.

Chapter 1 – Introduction to Visual Testing

In this chapter, Angie makes the point that automated testing doesn’t catch visual errors. You have to admit that’s true.

How often do you write assertions to ensure that not only does the output page contain the proper elements, but placed properly relative to other elements on the page? Not often. Why?

1. It’s a lot of code to write
2. It’s brittle, because all that can change release to release

So, you write code to drive a page behavior that sets data, initiates an action, and validate elements in the resulting output that generates the resulting page – without validating the page itself.

Angie fills this chapter with real-world examples of visual bugs that escape functional tests.

She shows this real-world Twitter display. Twitter has great engineers. Why does this bug escape? Because this page has all the proper assertions but the order and spacing have failed. The functional test passed, and they didn’t have automated visual testing.

Chapter 2 – Installing Applitools

Applitools can be added to a project pretty easily. In this chapter, broken into four pieces, Angie shows you how to:

1. Download the Applitools SDK code from the Maven repository
2. Obtain your Applitools API Key
3. Add your API key to your project
4. Link your project with Applitools Eyes

By following these steps, you are ready to begin using Applitools in your existing functional tests.

Angie codes her tests using Java (you can use a range of test control languages – she shows you this later). She creates a new method called initiateEyes() to create a new Applitools Eyes session. This code informs Applitools that a new test session is beginning. Angie adds a call to initiateEyes() in her testSetup() method.

She also modifies the test tear-down class to include a call to eyes.abortIfNotClosed(), which ensures that an eyes session is closed if it has not encountered a previous test close. This call ensures that Applitools records a session correctly.

With the above four steps in place, it’s on to testing.

Chapter 3 – Running Your First Test

In this chapter, Angie shows you how to set up a test.

This is the chapter that can make you think hard about the way you are testing today.

Angie takes a sample app and shows how you might think of testing it functionally. For demonstration purposes, she chooses a bookstore app that displays all the titles in the bookstore.

Here is the app image. You can see the search filter at the top.

She tests the search function, where the function takes “text” as an input and displays only the titles that match “text”. The test automation code uses a specific text value and checks to see that a book with the matching text is displayed. That was the code I showed at the beginning of the article.

Angie starts by pointing out that the test as described has its own flaws. In one failure mode, the filtering has no effect, but the test still passes because a matching title is still displayed. So, this test code will miss a set of failure modes.

Next, Angie shows how to add visual testing to this page – which would spot a visual difference, even when the functional test passed.

When Applitools first tests a page, it captures the page and lets you know that you have a new image. If you accept that image, it becomes the baseline. When you run subsequent tests, Applitools compares the new capture, called a Checkpoint, against, the baseline image. If the two are visually matched by the match levels (described in Chapter 4), the test passes. Otherwise, Applitools identifies an unresolved difference that you must inspect.

Angie then walks you through various scenarios and shows how Applitools Eyes will let you create a baseline for your test, and then see how Applitools spots visual differences as you make modifications to the app you are testing. She also shows you how easy it is to either accept differences (to create a new baseline) or reject differences (to flag a failure).

Chapter 4 – Match Levels

In Chapter 4, Angie walks through the concept of match levels. These vary by level of restriction.

• Exact – requires a pixel-by-pixel match (this is what pixel comparators do)
• Strict – Flags human-identifiable visual differences
• Content – Looks at text content and ignores color changes
• Layout – Looks only at page structure

These are easy to code into your tests:

For automation purposes, the last three are the most useful.

“Strict” looks for human-noticeable visual differences. This is the default setting for Applitools Eyes.

“Content” looks for content differences and ignores color changes. You might use this setting when you modify your CSS for a new look-and-feel without changing your content.

The “Layout” level looks at the structure of the page itself but ignores its content. You might use this setting for pages with content that can change dynamically.  Newspapers, magazines, and retailers find this setting useful as their content changes regularly.

“Exact” exists only for demonstration purposes. Pixel-level tools used to be the only visual test tools available, and they run exact pixel comparisons. You can find plenty of commercial and open-source tools compare pixels.  These tests flag false positives for many reasons. Three examples include font anti-aliasing, color gradations, and video card differences. Never use Exact in production.

Chapters 5-9 Using Applitools

The subsequent chapters in her course go through key concepts, including:

• Checkpoints
• Organizing Tests
• Visual Validation of PDF Files
• Analyzing Test Results
• Integrations

Some big ideas include:

1. Applitools can scroll horizontally and vertically through the page for a given viewport size and stitch together a snapshot of the complete rendered page
2. You are not limited to using a single match level for a given page. Once you have a baseline, you can identify regions with different match levels from the standard. You might use your default “Strict” match level on the bulk of a page, while having regions that retain the same layout but with changing content on which you use “Layout” match levels.
3. Applitools records the browser, OS, and viewport size for each test. You can organize your results so that you compare tests of the same page on the same viewport size with different browsers.
4. Applitools integrates with your workflow. You can integrate with code repositories, like GitHub. A pull request from your code-building tools, like Jenkins, can launch a batch visual comparison of your build with the existing baseline automatically. You can also integrate with Slack, Jira, and other communication tools to notify team members when you encounter errors in their code.
5. Applitools has an optional root-cause analysis tool that can display underlying DOM changes responsible for visual differences. You can use this capability to share the code changes you uncover with your development team.
6. Applitools can validate PDF files as easily as it validates web pages. You can test the output of a PDF generator – with the same kind of match level control that you have in the rest of Applitools.

There is a lot more to this course worth investigating.

Conclusions

As a test engineer, you know the effort you must expend to write and maintain functional tests – and how much of that effort you expend on maintaining output HTML assertions. And, due to the repetitive nature of visual tests, manual testing can miss visual errors. Visual testing addresses these challenges by reducing the number of assertions needed – and it catches more of the failures without requiring manual testing. Finally, Applitools addresses a host of issues that people raised previously about running visual tests, including the problems of false positives, image maintenance, workflow integration.

You’re probably thinking that visual testing won’t catch every issue, and you’re right. It won’t catch bad audio. It won’t expose broken, missing, or wrong links that remain untested. But, visual testing helps you maintain your web application by reducing the effort needed to monitor the behavior of your application. And that can speed up your application development process.

As always, when I finish a course on Test Automation University, I like to display my completion certificate.

For More Information

• Sign up for Test Automation University
• Request a demo.
• Sign up for a free Applitools account.
• Check out the Applitools tutorials.