As a quality engineer, what do you think about when you consider the return on investing an hour of your time? Or two hours? What if it could save you up to half the work hours you spent over the past year?
If you spend a lot of time writing or maintaining your end-to-end tests, our data shows that you’ll find lots of savings. What would you do with that time?
I have written previously about our Impact of Visual AI on Test Automation report. And, I have written about the benefits that many of your peers discovered by using Applitools Visual AI. They reduced the number of lines of code they had to write. Their code became more stable and easier to maintain. They caught more bugs with less code. And they learned how to do all this in about one to two hours.
So, what can an hour or two of your time give you?
ROI For A Quality Engineer Hour
Business people like me often talk about return on investment (ROI). If we invest this money today, what does the future payoff look like, and when does it come?
ROI offers a good model for thinking for everyone. Including a decision about how to spend your time to learn a new skill. If you spend an hour learning something now, what is the future benefit to you, and when do you get it?
The ROI of a quality engineer hour might be measured by:
- Increasing the number of test cases you can run
- Increasing the number of bugs you can uncover
- Decreasing the amount of low-value code you write.
- Decreasing your code maintenance effort.
So if you’re going to invest one or two hours into learning a new technology, like, say, Visual AI, you would like to see many hours worth of return on your time.
Visual AI frees you from writing inspection code in your end-to-end tests. Inspection code has the highest chance to contain errors, gets written selectively to provide limited coverage, and still can require a high degree of coding skill.
You might think that your value as a software engineer comes from your coding skills, and that often devolves into the inane measure of the value of a software engineer by counting lines of code written. Truthfully, not all code has the same value. In fact, code in of itself has zero value. It’s what that code does to help you test more conditions that matters.
High-Value Code And Low-Value Code
Your end-to-end tests contain both high-value code and low-value code.
The high-value code exercises your application. It sets the test cases. It runs execution variations. The high-value code correlates to source code and UI test coverage.
The low-value code inspects your results. And, if you’re using state-of-the-art coding, you’re inspecting the DOM for output web elements. Some you find with an ID. Others you find with a CSS selector. Some you find with a relative Xpath expression. Sometimes this code can involve incredible complexity. For example, writing all the assertions for reordering a table can involve elegant coding skills. And, yet…
If you review your end-to-end tests, much of your code determines whether the DOM contains the proper elements to let you conclude that the app passed. And, you have become good at triage. You cannot cover the state of every element in the DOM, so you become great at selecting code to inspect that tells you if your test passed or failed. Still, you write a lot of inspection code compared to execution code – and you only inspect a selected subset of the DOM.
But, the time you take writing this low-value code detracts from some of the high-value activity you can add to your end-to-end tests. For instance – you can write the preconditions for each test to ensure that they can be run independently – and, thus, in parallel. You can review the test conditions evaluated by each test to eliminate redundancy and improve automation accuracy.
What about maintenance? We didn’t even include code maintenance efforts you undertake between releases. That is yet more end-to-end coding effort you need – to validate and update existing tests, add new tests, and resolve failures – every time you modify the application. And, yes, some of that code provides new or modified test conditions. As well, some of that code needs to modify your inspection code.
Visual AI Replaces Low-Value Code
When we developed Visual AI, we recognized that a quality engineer makes trade offs. One engineer can only write a finite number of lines of code. An incomplete test has no value. So, every line of code needed to execute a test and validate the results makes a complete test.
We also recognized the limitations of DOM inspection for test results validation. Inspected thoroughly, the DOM represents the elements of the page to be rendered by the browser. And, one can write a detailed DOM inspection – and even account for variations in the DOM structure between releases. However, that depth of inspection involves complexity that rarely pays off for the coder. So, most coders spot-check their DOM – and can miss unexpected changes between releases unless validating the application through manual testing.
Visual AI uses one line of code to capture every visual element on your web page – and the state of the DOM that created the page. Once captured, Visual AI compares your captured version to your baseline. Then, Visual AI highlights for you the real user-noticeable differences between the baseline and the new capture. From there, you choose to accept the changes as expected new features or reject them as bugs. And, you can link those differences directly to the DOM code used to generate those differences.
Since inspection statements make up the bulk of your end-to-end test code, by adding Visual AI, you can eliminate the bulk of your inspection statements – letting you write code faster, making your code more stable, and allowing you more time to work on high-value test automation tasks.
How Long Does It Take To Learn Visual AI?
When we started the Applitools Visual AI Rockstar Hackathon, we directed participants to two courses on Test Automation University (TAU). TAU, offered exclusively by Applitools, offers classes on a range of technologies, including:
- Programming languages (like C and Python)
- Test frameworks (like Selenium and Cypress)
- Mobile testing
- API automation
- And, of course, Visual AI
We pointed participants to one course written by Raja Rao describing how to modernize your functional tests with Visual AI. Raja walked through the different test cases on the Hackathon in about an hour. We also pointed participants to a course by Angie Jones on how to add Visual AI to your test automation. Each course took approximately an hour.
Hackathon participants got pretty amazing results. After an hour or two of classes, they applied their knowledge of Visual AI and found:
- Average test coverage jumped from 65% for coded inspection to 95% for Visual AI
- Test writing time, on average, dropped from a little over 7 hours to a little over 1 hour.
- The amount of code they wrote dropped significantly
So, for one or two hours of learning, Hackathon participants got faster test writing, more coverage, and less code.
Testing Visual AI Yourself
In the end, testing is a scientific activity. We run a series of tests and make observations – pass or fail.
In this blog, you’re reading a bunch of claims about Visual AI. These are based on data that we shared in our report about the Hackathon results.
What do these claims mean to you?
I recommend that you test these claims yourself. The claim – one or two hours of learning can help you write tests:
- With comprehensive inspection
- Requiring less code
- More easily maintained
- Let you focus your time on high-value test activity.
If true, would that be worth the investment?
It is up to you to find out.
Learn More
- Modern Functional Test Automation Through Visual AI
- Automated Visual Testing: A Fast Path To Test Automation Success
James Lamberti is CMO at Applitools
