You’re running functional tests with visual validation, and you have dynamic data. Dynamic data looks different every time you inspect it. How do you do functional testing with visual validation, when your data changes all the time?
I arrived at Chapter 7 of Raja Rao DV’s course on Test Automation University, Modern Functional Test Automation Through Visual AI. Chapter 7 discusses dynamic data – content that changes every time you run a test.
Dynamic data pervades client screens. Just look at the screenshot above. Digital clocks. Wireless signal strength. Location services. Alarm settings. Bluetooth connectivity. All these elements change the on-screen pixels, but don’t reflect a change in functional behavior.
This bank website updates the time until the local branch closes, in case you want to visit the branch. That time information will change on the screen. And Visual AI captures visual differences. So, how do you automate tests that will contain an obvious visual difference?
Dynamic regions can be the rule, rather than the exception, in web apps. But, for the purposes of visual validation, dynamic elements on-screen comprise the group of test exceptions. You need a way to handle these exceptions.
Match Levels
Visual validation depends on having a good range of comparison methods because dynamic data can impact visual validation. Visual AI groups pixels into visual elements and compares elements to each other. The strictness you use in your comparison is called the match level.
Applitools Visual AI determines the element relative location, boundary, and properties (colors, contents, etc.) of each visual element. If there is no prior baseline, these elements are saved as the baseline. Once a baseline exists, Applitools will check the checkpoint image against the baseline.
Raja introduces three kinds of match levels for Applitools to compare your checkpoint against your baseline. You can use these match levels to inspect a subset of a page, a screenful, or an entire web page. Here are the three main match levels:
- “Strict” – Visual AI distinguishes location, dimension, color, and content differences as a human viewer would.
- “Content” – Visual AI distinguishes location, dimension, and content differences. Color differences are ignored as long as the content can be distinguished. Imagine wanting to see the impact of a global CSS color change.
- “Layout” – Visual AI distinguishes location and dimension and ignores content like text and pictures. This match level makes it easy to validate the layout of shopping and publication sites with a consistent layout and dynamic content.
You choose the match level appropriate for your page. If you have a page with lots of changing content, you choose “Layout” – which checks the existence and of regions but ignores their content. If you just made a global color change, you use “Content.” In most cases, you use “Strict.”
You set the match level in your call to open Applitools Eyes. If you don’t specify a match level, it defaults to “Strict.”
Handling Exceptions – Regions
We like to think of our applications on a page-by-page basis. Each locator points to a unique page that behaves a certain way. In some cases, the relevant application content resides on a single screen. Often, though, applications and pages can extend beyond the bottom of the visible screen. Occasionally, content extends across wider than the visible screen as well.
By default, Applitools captures a screenful – the current viewport. Raja covered this code specifically in the prior chapters when he showed us how to use:
eyes.checkWindow();
Using the same command, with the “fully” option, you can capture the full page, not just the current viewport. Assuming the page scrolls beyond the visible screen, you can have Applitools scroll down and across all the screens and stitch together a full page. So you can compare full pages of your application, even if it takes several screens to capture the full page.
Be aware that the default comparison uses strict mode. You can choose a different mode for your comparison. And, you can handle exceptions with regions.
So, now that you know that you can instruct Applitools to capture a full page, or a viewport, what happens when you have dynamic data, or other parts of a page that could change? You need to identify a region that behaves differently.
Applitools adds the concept of “regions.” As Raja describes, “region” describes a rectangular subset of the screen capture – identified by a starting point X pixels across and Y pixels down relative to the top of the page, plus W pixels wide and H pixels high.
Control Region Comparisons
Once you have a region, you can use one of the following selections to control the inspection of that region:
- Ignore – ignore that region completely. Its contents do not matter to the test identifying differences. Useful for counters
- Floating – the content within the region can shift around. Text that can shift around.
- Strict – content that should stay in the same place and color from screen to screen
- Content – Content that should stay in the same place with varying color from screen to screen
- Layout – Content that can change but has a common layout structure from screen to screen
Regions let you be permissive when you use a restrictive match level. “Ignore” literally means that – ignore the content of the region. There may be times you want to ignore a region. More often, though, you might want to ensure that the region boundary and content exists – for this you use “Layout.”
Regions let you handle exceptions on a more restrictive basis as well. For example, on a page using layout mode, you can create a region and use “strict” to compare content and color that should be identical – such as header or menu bar.
Testers Choice
One big point Raja makes is that you get to choose how to deploy Visual AI. Select the mode that matches the page behavior you expect, and then set the appropriate mode for handling exceptions.
Raja demonstrates how you can choose to define exceptions in the UI or in your test code. You can choose to set the exceptions in the Applitools UI. Once you set a region with a specific match level, that region with that match level persists through future comparisons. Alternatively, you can add regions to handle exceptions directly in your test code. Those region definitions persist as long as they persist in your code.
You don’t need to capture an entire window or page. You can run your eyes.open() and use:
eyes.checkRegion()
choose just to capture individual regions at an appropriate comparison level. This kind of test can be useful during app development when you want to distinguish the behavior of specific elements you are building.
If you’re really focused on using element-based checks, you can even run:
eyes.checkElement()
The checkElement instruction uses web element locators to find specific on-page elements. checkElement lets you use a legacy identifier in a modern functional test approach. In general, though, checkElement adds more complexity compared with visual validation.
The key understanding is that, for a given capture page, you can define your mode for the full capture and exceptions for specific regions, so that you cover the entire page.
Handling Expected Changes
When you make changes, all your captures must be updated. CSS, icons, menus, and other changes can affect multiple pages – or even your entire site. Imagine having to maintain all those changes – page by page. Yikes.
Fortunately, Applitools makes it easy to accept common changes across multiple pages.
Whenever you encounter a difference on a single page, you are instructed to accept the change or reject it. If you reject the change – it’s an error and you can flag development. But, if you accept the change, you can also use a feature called automated maintenance to accept the change on all other pages where the change has been discovered.
Update your corporate logo. Done.
Install a new menu system. Easy.
You can use Automated Maintenance to accept changes. You can also use Automated Maintenance to deploy regions across all the pages – such as ignore regions.
Of course, the more comprehensive your changes, the more challenging it is to use automated maintenance. If you make some significant changes in your layout, expect to create new baselines as well as use automated maintenance.
Conclusions about Visual Validation and Dynamic Data
We all want to build applications that achieve business outcomes. We often build visually interesting pages with changing content designed to keep buyers engaged. But, we also know that testing requires repeatability – meaning that dynamic may be great for business, but testing requires predictable results.
Dynamic data can limit the benefits of visual validation. You need a way to handle dynamic data in your visual validation solution. Applitools gives you tools to handle dynamic parts of your application. You can handle truly dynamic sections by ignoring regions, treating those regions as layout regions, or even handling a whole page as a layout and let sections and content change.
And, when you make global changes, automated maintenance eases the pain of updating all your baseline images.
As Raja makes it clear, Applitools has thought not just about discovering visual changes, but handling unexpected changes that are defects, dynamic data that will produce false-positive defects, and expected global changes affecting multiple pages. All of these features make up key parts of a modern functional testing system.
And now, on to Chapter 8.
For More Information
Blogs about chapters in Raja’s series on Modern Functional Testing:
- Modernize Your Functional Testing – Chapter 1 of Raja’s course.
- Advanced Tools for Testing Tables – Chapter 2 of Raja’s course.
- Data-Driven Testing with Visual AI – Chapter 3 of Raja’s course.
- How Do you Test Dynamic Content? – Chapter 4 of Raja’s course.
- Testing iFrames with Visual AI – Chapter 5 of Raja’s course.
- Complex Functional Testing Simplified – Chapter 6 of Raja’s course.
Actions you can take today.
- Sign up for Test Automation University
- Read all about Functional Test Myopia
- Visit the Automated Visual Testing Course on Test Automation University
- Blog: How Do I Validate Visually?
- Sign up for a free Applitools account
- Request an Applitools Demo
- Visit the Applitools Tutorials
