Enhancing UI/UX Testing with AI

Andrew Knight — Tue, 17 Jan 2023 23:42:06 +0000

This article is based on our recent webinar, How to Enhance UI/UX Testing by Leveraging AI, led by Chris Rolls from TTC and Andrew Knight from Applitools. Editing by Marii Boyken.

Last week, I hosted a webinar with Chris Rolls from TTC. In the webinar, Chris and I talked about the current state of software testing, where it’s going, and how visual AI and automation will impact the future of software testing. In this article, I’ll be recapping the insights shared from the webinar.

The current state of testing

Software testing is often seen in businesses as a necessary evil at the end of the software development lifecycle to find issues before they reach production. Chris and I strongly agree that software quality is crucial to modern businesses to help to achieve modern businesses’ goals and needs to be thought of throughout the software development lifecycle.

Digital transformation and continuous delivery

The largest and most relevant companies today have embraced digital transformation and technology to run their businesses and meet their customers’ needs. To keep up with digital transformation, you need modern software development practices like DevOps and continuous delivery. DevOps requires continuous testing to be successful, but we’re seeing that reliance on manual testing is the biggest challenge that organizations face when adopting DevOps.

Software testing and quality assurance

The software world is changing, so we need to change how we deliver technology. That requires modern software development approaches, which requires modern software testing and software quality approaches. Thankfully, testing and quality are far more top of mind now than in previous years.

From the audience poll results from our webinar, we see that continuous delivery is here to stay. When asked how often do you deploy changes to production, over 40% of people stated that they deploy either daily or multiple times per day. This wasn’t the case 10 to 20 years ago for most organizations unless technology was the product. Now, daily deployments are pretty common for most organizations.

Low rates of test automation

However, we’re seeing that getting high test automation coverage is still a huge challenge. In the survey, 55% of the respondents automate less than half of their testing.

The numbers may have a bit of sample bias because Applitools users actually automate on average 1.7 times higher than other respondents. The responses align with anecdotal experience that a lot of organizations are still in lower test automation coverage around 20% to 50%.

Testing complexity and the amount of testing needed are going up increasingly, and this shows in our survey as well. More than 50% of the respondents test two languages or more, three browsers or more, three devices or more, and two applications or more.

With two thirds of the respondents saying that one of the hardest parts of testing UIs is that they are constantly changing, traditional automation tools can’t handle testing at that speed scale and complexity.

Using AI in testing

We know that there’s a lot of excitement around AI tools, and the survey shows that.

When asked What parts of the testing process are you supporting or planning on supporting with AI, test offering, test prioritization, test execution, test management, and visual regression we all mentioned. The top two answers among respondents were test execution and visual regression.

It’s important to remember that continuous testing is about more than just test automation.

While test automation is key to the process, you still need to incorporate other software testing and software quality practices like load testing, security, user experience, and accessibility.

The future of testing

What we’re trying to achieve in the future of testing is to support modern software development practices. The best way we’re seeing to do this is to have software testing and software quality more tightly integrated into the process. Let’s talk about what a modern software approach looks like.

Testing sooner and more frequently

To get tighter integration of quality into the process, testing can’t just be an activity at the end of the development lifecycle. Testing has to happen continuously for teams to be able to provide fast feedback across disciplines and ensure a quality product. When this is done, we see increased speed not just of testing, but of overall software development and deployment. We also see reduced costs and increased quality. More defects are found before production, and we see quicker responses when finding defects in production.

Increasing how much testing is automated

Traditional testing approaches tend to be done mostly manually. Increasing test coverage doesn’t mean manual testing goes away.

Automating your test cases frees up time to do more exploratory testing. Exploratory testing should be assisted by different tools, so AI has a good role to play here. Tools like ChatGPT are useful to brainstorm things like what to test next. Obviously we want to increase test automation coverage at all levels, including unit, API, and UI. Intelligent automated UI tests provide us more information than functional tests alone.

Knowing when to use AI in testing

What does the future of testing with AI look like? It’s a combination of people, processes, and technology. Software testers need to be thinking about what skills we need to have to support these new ways of testing and delivering quality software.

We need to uncover if a use case is better served by AI and machine learning than an algorithmic solution. To do this, we need to ask the following questions:

What is the pain point that we are trying to solve?
Are there good algorithmic or heuristic solutions available to address the pain point?
Are there enterprise ready AI and machine learning solutions that help?
- If there are solutions, what happens if the AI doesn’t work?
- If there aren’t solutions yet, how do we position ourselves to be ready for them?

“It’s quite trendy to talk about artificial intelligence, but the reason why we’re partnered with Applitools is that they apply real machine learning and artificial intelligence to a problem that is not well solved by other types of solutions on the market.”
Chris Rolls, CEO, Americas, TTC

Integrating visual AI into test automation

Let’s talk about how we can integrate AI into our testing to get some of those advantages of increased speed and coverage discussed earlier.

What is visual AI?

I like to explain visual AI visually. Do you remember those spot-the-difference pictures we had in our activity books from when we were kids?

As humans, we could sit around and play with this to find the differences manually. But what we want is to be able to find these differences immediately. And that is what visual AI has the power to do.

Even when the images are a little bit skewed or off by a couple pixels here or there, visual AI can pinpoint any differences between one view and another.

Now you might be thinking, Andy, that’s cute, but how’s this technology gonna help me in the real world? Is it just gonna solve little activity book problems? Well, think about all the apps that you would develop – whether web, mobile, desktop, whatever you have – and all the possible ways that you could have visual bugs in your apps.

Here we’ve got three different views from mobile apps. One for Chipotle, one for a bank, and another one for a healthcare provider.

Visual AI test automation

You can see that visual bugs are pervasive and they come in all different shapes and sizes. Sometimes the formatting is off, sometimes a particular word or phrase or title is just nulled out. What’s really pesky is that sometimes you might have overlapping text.

Traditional automation struggles to find these issues because traditional automation usually hinges on text content purely or on particular attributes of elements on a page. So as long as something appears and it’s enactable, most traditional scripts will pass. Even though we as humans visually can inspect and see when something is completely broken and unuseable.

This is where visual AI can help us, because what we can do is we can take snapshots of our app over time and use visual AI to detect when we have visual regressions. Because if, let’s say, one day the title went from being your bank’s name to null, it’ll pick it up right away in your continuous testing.

Visual AI in action

In the webinar, I gave a live demo of automated visual testing using Applitools Eyes. In case you missed the demo, you can check it out here:

The future will be autonomous

So all this really cool stuff is powered by visual AI, a real world application of AI looking at images and being able to find things in them like they were humanized. Now you may think this is really cool, but what’s even cooler is this is just the beginning of what we can do with the power of AI and machine learning in the testing and automation space.

What we’re going to be seeing in the next couple years is a new thing called autonomous testing where not only are we automating our tests, but we’re automating the process of developing and maintaining our tests. The tests are kind of almost writing themselves in a sense. And visual AI is going to be a key part of that, because if testing is interaction and verification, what we want to make autonomous is both interaction and verification. And visual AI has already made verification autonomous. We’re halfway there, folks.

Be sure to check out our upcoming events page for new webinars coming soon!Learn how Applitools Eyes uses AI to catch visual differences between releases while reducing false positives. Happy testing!

TTC Archives - Automated Visual Testing | Applitools