Ask 288 Of Your Peers About Visual AI

It’s a difficult time. We all know why, so I won’t dwell on it other than to wish you and yours health and safety above all else. What I will dwell on is the human need to retreat and replenish. Trapped at home, I’ve found myself learning to cook with Thomas Keller, exploring the universe with Neil deGrasse Tyson, or entertaining like Usher through Masterclass.com. My kids are coding their own games and learning about the history of art at Khan Academy. These entertaining explorations not only give us a much-needed break, but they also give us an opportunity to learn and grow even as we struggle with the realities around us. It’s a welcome and much-needed distraction.

With that sentiment in mind – here’s an idea for you. Why not learn about Visual AI (Artificial Intelligence) from 288 of your fellow quality engineers?

Each one of them spent 11 hours on average comparing their current test framework of either Cypress, Selenium, or WebdriverIO to that same framework modernized through Visual AI. You can get a summary of what they learned here. Even better, you can take the same free Test Automation University course on Modern Test Automation Through Visual AI and do it all yourself through video tutorials and hands-on learning. Either way, you will find yourself blissfully distracted while learning a cutting-edge approach to test automation.

288 Testers. 11 Hours Each. That’s 1.5 Years of Quality Engineering Effort!

Yes — we were blown away by the enthusiasm to learn Visual AI among the testing community. It says a lot about this group of individuals who recognize the need to keep pushing themselves. In the end, they ended up creating the industry’s largest, highest quality, and freely available data set for understanding the impact of Visual AI on test automation, and ultimately on the impact on quality management and release velocity for modern applications. It’s an amazing amount of learning highly representative of the world of test automation.

We had representation from major test frameworks:

Representation from major languages:

Representation from 101 countries around the world

Why Should You Learn Visual AI? Ask your peers.

I get it. Quality engineers always seem to be on a treadmill to learn everything. You have new application development frameworks, new coding structures, new test frameworks, and new tools rumbling your way daily. If you plan to learn one more thing, you need a return on your time.

But, let’s face it – testing needs to keep up with the pace of the business. Survey data tells us that the majority of software teams are struggling with their quality engineering efforts. In a recent survey, 68% of teams cited quality management as a key blocker to more agile releases and ultimately CI/CD.

Why? For every test with a handful of conditions and an action, test writers need to write dozens to hundreds of code-based assertions to validate a single response. Traditional frameworks simply don’t have the technical ability to provide front-end functional and visual test automation coverage with the speed and efficiency you need. You end up writing and maintaining too much test code, only to see bugs still escape. It’s maddening and, even worse, it prevents us from doing our core job of managing app quality.

Isn’t AI Just Smoke and Mirrors?

The answer depends on your application. AI promises to solve many modern technical problems, including testing and quality management problems, but it’s hard to separate the truth from the reality in what really works. Many experiments using AI have failed in testing, or these AI approaches require you to “rip and replace” your existing tech stack – a dreaded approach that is unrealistic for most teams.

Rather than asking you to simply trust that Visual AI is different, we decided to prove it, objectively, using real-world examples, in partnership with real quality engineers at real companies dealing with test automation every day.

Gathering Learning – The Visual AI Rockstar Hackathon

To generate all this learning, we built an application involving five common but complex use cases. In November 2019, we issued a challenge to testers all over the world to compete, and learn, by comparing test approaches side-by-side. The competitors created test suites for each of the five use cases using their preferred code-based approach, including Selenium, Cypress, and WebdriverIO. These same quality engineers then repeated the process for the exact same five use cases using Visual AI from Applitools.

To make it fun and push people to do their absolute best, testers competed for 100 prizes worth a total of $42,000. We judged their submissions on their ability to:

• Provide test coverage on all use cases
• Successfully run these tests, and
• Most importantly catch all potential bugs

using both testing approaches.

You can learn about 100 winners here.

Your Takeaways

The data we collected from Hackathon participants makes one clear point: using visual AI makes you more efficient. You gain this efficiency no matter what your level of experience.

Each Hackathon use case – from handling dynamic data to testing table sorts – requires you to apply your best coding skills. We found that even the most diligent engineers encountered challenges as they developed test code for the use cases.

For example, many of your peers can easily apply test conditions to filter a search or sort a table. Many can grab data from web element locators. However, many of them struggle to calculate expected responses consistently. And, many have challenges creating accurate reports when tests fail.

We encourage you to review the Hackathon report and results. We also encourage you to sign up for a free Applitools account and try out these tests for yourself.  But, if you do nothing else, just check out the five cases. Ask yourself how frequently you encounter these cases in your own test development. We think you will conclude – just as our test data shows – that Visual AI can help you do your job more easily, more comprehensively, and faster.

To Read, Learn, and Do

• Read Complex Functional Testing Simplified
• Read Angie Jones’s course on Removing Visual Blind Spots
• Sign up for Test Automation University and start taking classes
• Take Raja Rao’s Course on Modern Functional Testing
• Sign up for a free Applitools account
• Request an Applitools demo

James Lamberti is CMO at Applitools.