Fast, Efficient and Effective Cross Browser Testing

What do you think about cross browser testing?

Developers likely develop on only one browser – and maybe only one operating system. How does an app maker ensure that defects on other browsers will not escape to their user base? In theory, cross browser testing can help companies catch product defects before products get released to customers. But the legacy approach – setting up a bunch of parallel devices and running tests across each – incurs significant engineering skill and resource cost.

Choose Your Cross Browser Testing Camp

With this legacy infrastructure requirement, most engineering managers find themselves in one of two camps:

Managers in Camp 1 run cross browser tests. These managers have been held accountable for missed bugs due to an untested platform in the past. They know that cross browser testing provides coverage, even though it requires resources for infrastructure deployment and test maintenance.

Managers in Camp 2 have not experienced significant field bugs that were caused by untested behavior in different browsers. Camp 2 managers have evaluated cross browser testing. They have concluded that cross browser testing has a low likelihood of exposing unique bugs.

So, yes, Camp 1 and Camp 2 share a common view of cross browser testing – slow, cumbersome to manage with limited effectiveness in catching defects. Camp 1 people have been burned by a bug they could have caught with cross browser testing. Members of Camp 2 have not.

Camp 3 – Use Modern Cross Browser Testing

The first two camps get stifled by the legacy approach to cross browser testing. In the legacy approach, you take your existing test infrastructure and port it from your browser of choice (Chrome, Firefox, Edge, Safari) on your test operating system and check out combinations of browsers and operating systems. You might even check out different viewport sizes to validate responsive app behavior.

Camp 3 uses Modern Cross Browser Testing, made possible by Applitools Visual AI and Applitools Ultrafast Grid. Applitools Visual AI provides highly accurate visual comparisons between a captured screen and a previous baseline. With Applitools Ultrafast Grid, the Visual AI screen capture also returns the DOM of the captured screen.  Then, Applitools Ultrafast Grid reruns the DOM against each target browser/operating system/viewport combination specified by the tester. For each target, Applitools captures the resulting screen and compares it against the relevant baseline.

Modern cross browser testing leverages key insights:

• The bulk of apps generate a single DOM response regardless of the target browser;
• Cross browser tests expose visual defects, as functional defects get caught elsewhere;
• Legacy test code, which inspects elements of the DOM, requires lots of code to capture and compare all the visual elements for an app response;
• Visual AI makes cross browser test automation feasible and effective.

Proving the Value – Applitools Cross Browser Testing Hackathon

At Applitools, we know that our Ultrafast Grid, combined with Visual AI, create a simple approach to cross browser testing. We overcome all the objections and complaints from both Camp 1 and Camp 2. And our approach contradicts the conventional wisdom of engineers who sit in Camp 1 and in Camp 2.

We found ourselves stuck with a question:

How do you get a group of really smart people who think one way to change their minds?

Evidence? Maybe. Let them try it for themselves? Possibly – but why would someone want to try something out?

What if you give them an incentive? Like a contest?

Applitools had a lot of experience from the 2019 Visual AI Rockstar Hackathon. We knew we could get engineers to try their hand at using Visual AI. Perhaps, we could get a similar response with another hackathon pitting legacy cross browser testing against Visual AI plus Ultrafast Grid.

So, we ran the Applitools Cross Browser Testing Hackathon in June 2020. My next few blog posts will go through the hackathon in detail and present you with some of the results and conclusions. They include:

• Ultrafast Grid provides faster test execution
• You need fewer test runs with Ultrafast Grid
• Your tests require less coding (and less time debugging test code)
• Ultrafast grid needs no additional hardware (whether onsite or provided by a service)
• Cross browser tests with Ultrafast Grid are easier to maintain
• A large number of the participants who were not previously inclined towards cross browser testing would recommend Visual AI and Ultrafast Grid.

You can skip the blog posts and read the full report instead.

You can read Marie Drake’s summary of her experience as a Hackathon participant and winner.

Or, you can wait for my next blog post.