There is a lot of buzz around Visual Testing these days. You might have read or heard stories about the benefits of visual testing. You might have heard claims like, “more stable code,” “greater coverage,” “faster to code,” and “easier to maintain.” And, you might be wondering, is this a hype of a reality?

So I conducted an experiment to see how true this really is.

I used the instructions from this recently concluded hackathon to conduct my experiment.

I was blown away by the results of this experiment. Feel free to try out my code, which I published on Github, for yourself.

Visual Assertions – my learnings

Before I share the details of this experiment, here are the key takeaways I had from this exercise:

1. Functional Automation is limiting. You only simulate known user behaviors, in predetermined conditions and in the process only validate & verify conditions what you know about. There has to be a more optimized and value generating approach!
2. The Automation framework should have advanced capabilities like soft assertions and good reporting to allow quick RCA.
3. Save time by integrating Applitools’ Visual AI Testing with the Ultrafast Grid to increase your test coverage and scale your test executions.

Let us now look at details of the experiment.

Context: What are we trying to automate?

We need implement the following tests to check the functionality of https://demo.applitools.com/tlcHackathonMasterV1.html

1. Validate details on landing / home page
   This should include checking headers / footers, filters, displayed items
2. Check if Filters are working correctly
3. Check product details for a specific product

For this automation, I chose to use Selenium-Java for automation with Gradle as a build tool.

The code used for this exercise is available here: https://github.com/anandbagmar/visualAssertions

Step 1 – Pure Functional Automation using Selenium-Java

Once I spent time in understanding the functionality of the application, I was quickly able to automate the above mentioned tests.

Here is some data from the same.

Refer to HolidayShoppingWithSeTest.java

Activity	Data (Time / LOC / etc.)
Time taken to understand the application and expected tests	30 min
Time taken to implement the tests	90 min
Number of tests automated	3
Lines of code (actual Test method code)	65 lines
Number of locators used	23
Test execution time: Part 1: Chrome browser	32 sec
Test execution time: Part 2: Chrome browser	57 sec
Test execution time: Part 3: Chrome: 29 sec	29 sec
Test execution time: Part 3: Firefox: 65 sec	65 sec
Test execution time: Part 3: Safari: 35 sec	35 sec

Observations

A few interesting observations from this test execution:

1. I added only superficial validations for each test.
   • I only added validations for the number of filters and items in each filter. But I have not added the validations for actual content of the filters.
   • To add actual validations for each aspect of the page will take 8-10x the time taken for my current implementation, and the number of locators and assertions would also probably increase by 4-6x of the current numbers.
   • Definitely does not seem worth the time and effort.

2. The tests would not capture all errors based on the assertions added, as the first assertion failure would cause the test to stop.
3. In order to check everything, instead of hard assertions, the framework would need to implement and support soft assertions
4. The test implementation is heavily dependent on the locators in the page. Any small change in the locators will cause the test to fail. Ex: In the Product Details page, the locator of the Footer is different from that of the home page
5. Scaling: I was limited by how many browsers / devices I could run my tests on. I needed to write additional code to manage browser drivers, and that too only for browsers that I had available on my laptop

Step 2 – Add Applitools Visual Assertions to Functional Automation

When I added Applitools Visual AI to the already created Functional Automation (in Step 1), the data was very interesting.

Refer to HolidayShoppingWithEyesTest.java

Activity	Data (Time / LOC / etc.)
Time taken to add Visual Assertions to existing Selenium tests	10 min
Number of tests automated	3
Lines of code (actual Test method code)	7 lines
Number of locators used	3
Test execution time: Part 1: Chrome browser	81 sec (test execution time) 38 sec (Applitools processing time)
Test execution time: Part 2: Chrome browser	92 sec (test execution time) 42 sec (Applitools processing time)
Test execution time: (using Applitools Ultrafast Test Cloud) Part 3: Chrome + Firefox + Safari + Edge + iPhone X	125 sec (test execution time) 65 sec (Applitools processing time)

Observations

Here are the observations from this test execution:

1. My test implementation got simplified
   • Less lines of code
   • Fewer locators and assertions
   • Test became easier to read and extend

2. Test became more stable
   • Fewer locators and assertions
   • It does not matter if the locators change for elements in the page as long as the user experience / look and feel remains as expected. (Of course, locators on which I need to do actions using Selenium need to be the same)

3. Comprehensive coverage of functionality and user experience
   • My test focuses on specific functional aspects – but with Visual Assertions, I was able to get validation of the functional change from the whole page, automatically

See these below examples of the nature of validations that were reported by Applitools:

Version Check – Test 1:

Filter Check – Test 2:

Product Details – Test 3:

4. Scaling test execution is seamless
   • I needed to run the tests only on any 1 browser which is available on my machine. I chose Chrome
   • With the Applitools Ultrafast Test Cloud, I was able to get results of the functional and visual validations across all supported platforms without any code change, and almost in the same time as a single browser execution.

Lastly, an activity I thoroughly enjoyed in Step 2 was the aspect of deleting code that now became irrelevant because of using Visual Assertions.

Conclusion

To conclude, the experiment made it clear – Visual Assertions are not a hype. The below table shows in summary the differences in the 2 approaches discussed earlier in the post.

Activity	Pure Functional Testing	Using Applitools Visual Assertions
Number of Tests automated	3	3
Time taken to implement tests	90 min (implement + add relevant assertions)	–
Time taken to add Visual Assertions to existing Selenium tests	–	10 min Includes time taken to delete the assertions and locators that now became irrelevant
Lines of code (actual Test method code)	65 lines	7 lines
Number of locators used	23	3
Number of assertions in Test implementation	16 This approach validates only specific behavior based on the assertions. The first failing assertion stops the test. Remaining assertions do not even get executed	3 (1 in for each test) Validates the full screen, captures all regressions and new changes as well in 1 validation
Test execution time: Chrome + Firefox + Safari browser	129 sec (for 3 browsers)	–
Test execution time: (using Applitools Ultrafast Test Cloud) Part 3: Chrome + Firefox + Safari + Edge + iPhone X	–	125 sec (test execution time) 65 sec (Applitools processing time) (for 4 browsers + 1 device)

Visual Assertions help in the following ways:

• Make your tests more stable
• Lower maintenance as there are less locators to work with
• Increase test coverage – you do not need to add assertions for each and every piece of functionality as part of your automation. With Visual Assertions, you will get the full – functional & user experience validation by 1 call
• Scaling is seamless – with the Ultrafast Test Cloud, you run your test just once, and get validation results across all supported browsers and devices

You can get started with Visual Testing by registering for a free account here. Also, you can take this course from the Test Automation University on “Automated Visual Testing: A Fast Path To Test Automation Success”