Who recommends cross browser testing to their organizations?
In this series, we discuss the results of the Applitools Ultrafast Cross Browser Hackathon. Today, we will explain how those who were skeptics about cross browser testing would recommend that their organizations run Applitools Ultrafast Grid for cross browser testing.
Reviewing Past Posts In This Series
In this series we have covered the results of the Applitools Ultrafast Cross Browser Hackathon.
- My first post covered the overall results. I wrote specifically about the ease of creating cross browser tests with Applitools and introduced the topics to follow.
- My second post covered how the speed of Applitools Ultrafast Grid makes cross browser testing a reality within the application build process.
- In the third, I explained how Applitools Visual AI tests provide much greater code stability compared with legacy cross browser tests, making the code easier to develop and maintain over time.
With this post, I will cover the survey topic in the hackathon: would Hackathon participants recommend that their organizations adopt legacy cross browser testing approach to their peers, and would they recommend Applitools Ultrafast Grid for that purpose?
Methodology
For this survey, Applitools used an approach called Net Promoter Score (NPS). Net Promoter Score uses a survey question with the highest correlation to satisfaction:
“On a scale of 0 to 10, with 0 being not likely and 10 being highly likely, how likely are you to recommend [the survey object] to others?”
Researchers have shown that this question correlates most highly with satisfaction. Respondents who give a 9 or 10 (highly likely to recommend) get classified as “promoters.” Promoters have high satisfaction. Those who give a 7 or 8 get classified as neutral – they are neither satisfied or dissatisfied. Others with a 6 or below get classified as detractors. Detractors have been dissatisfied with something and have no willingness to recommend the survey object.
Break the survey responses into respondents by value and count. Add 1 for each 10 and 9. Give 0 for each 7 or 8. Subtract 1 for each 6 or below. Then, normalize the results to 100 by taking your count and dividing it by the number of respondents, and multiplying by 100.
Results can range from -100 to 100. According to Bain & Co, the developers of NPS:
- 80 rates as ‘world class’
- 50 rates as ‘excellent’
- 20 rates as ‘favorable’
- Around 0 rates as ‘neutral’
- -10 and lower rates as ‘negative’
Asking Hackathon Participants
Applitools surveyed the the 2,224 Hackathon participants about their willingness to recommend the use of Applitools Ultrafast Grid to their peers. The survey also asked their willingness to recommend the use of legacy cross browser testing. Of the participants, 203 engineers were able to run both the legacy and the Ultrafast cross browser tests.
Here were the survey responses:
Few Fans Of Traditional Cross Browser Testing
For legacy cross browser testing, using a traditional test application and validation process, 68% of participants got classified as detractors. 17% got classified as passives. Only 15% promoted the legacy approach. This gave a NPS of -54 (rounded down). Participants were, in general, not fans of legacy cross browser testing.
This result mirrors how much people use the legacy approach to cross browser testing. Companies listed in the graphic above provide the infrastructure to run tests. They don’t reduce the test load, or the code load. Their pricing reflects the cheaper cost for them to set up and maintain that infrastructure of devices, browsers, operating systems, and viewport sizes. Given the cost of setting up and maintaining legacy cross browser tests, it makes sense that not a lot of companies use cross browser testing actively.
Willing to Recommend Applitools Ultrafast Grid for Cross Browser Testing
Also, the survey asked participants their willingness to recommend Applitools Ultrafast Grid for Cross Browser testing. Here, 75% gave a 10 or 9 and got classified as promoters. Another 20% responded 7 or 8 and got classified as neutral.
The promoters valued:
- Speed of tests – noting they could run their tests in the build process
- Simplicity of management – no tests needed to be run and tuned across multiple plaforms
- Simplified code management – fewer locators meant easier to set up and manage
- More accurate – the underlying Visual AI wasn’t plagued by false positives and caught all the code errors
The promoters discovered the ease of creating and maintaining cross browser tests using Applitools Ultrafast Grid. The promoters also realized that, with tests completed and analzyed accurately in well under 10 minutes, Applitools Ultrafast Grid made it possible to run cross browser tests in the scope of a build or during unit tests. Legacy tests, even when run in parallel, took tens of minutes to complete and analyze.
Implications of Recommending Cross Browser Testing
If you read my earlier posts, you know that two camps existed related to cross browser testing. One camp ran cross browser tests because they had encountered issues in the past and saw cross browser tests as a safe approach. The other camp avoided it altogether and thought cross browser testing unnecessary.
There is a third camp using Applitools Ultrafast Grid. This camp recognizes that the combination of:
- test speed appropriate for software build processes
- ease of deployment
- lack of infrastructure to manage, and
- simplified test code management,
made cross browser testing feasible. This third camp can deploy Applitools Ultrafast Grid to run and validate rendering behavior for any combination of browser, operating system, and viewport size and run this test set quickly at the unit, build, merge, and final test timeframes.
What then are the implications for the Applitools Ultrafast Cross Browser Hackathon? 75% of the 208 engineers who completed both sets of tests could see the value of Applitools Ultrafast Grid just by using it. And, they would be willing to recommend it. They realized that, whether they had previously released a bug to the field that they could have caught with cross browser testing, that Ultrafast Grid changed their understanding about this kind of testing completely.
Applitools users know that Applitools Visual AI makes it possible to run visual tests as part of their unit tests. These users can incorporate visual tests at every build and merge. And with Applitools Ultrafast Grid, they can incorporate cross browser tests as well.
Importantly, Hackathon participants learned these lessons just through their time on the Hackathon.
What These Recommendations Mean For You
When you have 75% of engineers recommending something, it might be worth trying. Whether you classify yourself as a cross browser skeptic or a grudging participant, it is clear that Applitools Ultrafast Grid offers something that differs from your current conception of cross browser testing.
At the very least, read the Applitools results in detail. You will learn why the engineers gave these recommendations.
More importantly, why not give Applitools a try? Sign up for a free account? Or, if you prefer, request a demo from an Applitools representative.
Next Week
Next week, in my last blog post in this series, I will help you draw your own conclusions about Modern Cross Browser Testing.
