Many thanks to Applitools for the exciting opportunity to learn more about Visual AI testing by competing in the hackathon. While trying to solve the five challenges step by step, you truly grasp the fact that proper UI testing is impossible without visual testing. But let’s start from the beginning.
Two versions of the same website were provided. These two versions represented different builds of the same application.
Build 1: https://demo.applitools.com/hackathon.html
Build 2: https://demo.applitools.com/hackathonV2.html
I will evaluate each task for both approaches from 1 to 5, where 1 is “not applicable” and 5 is “the best choice”, and draw some conclusions at the end.
In my tests, I used the JDI Light test automation framework that is based on Selenium, but it is more effective and easier to use.
Here is the link to my final solution: https://github.com/RomanIovlev/applitools-hackathon
Here is the link to Applitools documentation: https://help.applitools.com/hc/en-us
Login Page View Test
In the first challenge, we needed to validate the view of the Login page in general, meaning to verify that all the elements are displayed properly.
It’s obvious that if you need to validate how the page looks, you can’t rely solely on functional validation. Why?
- You will spend a lot of time and lines of code to describe all elements.
- But more importantly, your validations can still miss critical issues:
- Changes in elements’ size or position
- Changes in images or backgrounds
- Different colors or line lengths
- Something unexpected does not appear
These are a few of the tricky, yet important things that prevent you from properly testing using traditional functional tests. Sometimes it is difficult to describe all the possible failure modes, and some validations are just impossible to verify. And of course, with the functional approach, in most cases, you can’t check that something unexpected does not appear. Sometimes you even can’t imagine that.
Here are the differences detected by Applitools between Build 1 and Build 2
How can you check this using the traditional approach to test automation? There are more than a dozen UI elements on the page and you need to create a method that validates them all
That’s 40 lines of code and we’ve only checked what we know is there, and not some unexpected surprises.
How does this compare to using Applitools to verify this with visual testing? Only one line of code is needed!
eyes.checkWindow("Login Page view");Here we use the checkWindow() method which allows us to validate the entire page.
This one line of code will help you to prevent more risks. And at the same time, thanks to the AI used in Applitools these validations will be stable and reduce the number of typical false-negative cases.
And the last point here, in addition to less code and broader validations, visual tests allow you to easily validate multiple issues at once, which is not a simple task in functional testing approaches.
| Testing Type | Applicable (1-5) | Comment |
|---|---|---|
| Functional approach (JDI Light) | 2 | you can try to test most of the valuable elements, but this requires you to write a lot of code and it provides no guarantees |
| Visual approach (Applitools) | 5 | With just one line of code, you can check most of the possible issues |
Login functionality (data-driven)
The second challenge is represented by a typical task for most applications. We needed to validate general login functionality for all valuable cases: Success login, Failed login, Empty and Big values, all Allowed symbols (maybe even some JS or SQL injection cases).
Definitely, this task is good for functional testing using the data-driven approach. Create test data for positive and negative cases and let it flow through your scenarios. Thanks to JDI you can describe simple forms in one row without complex PageObjects and submit it using a Data entity.
public static Form<User> loginForm;
And then you can use this form in your tests.
@Test(dataProvider = "correctUsers"...)public void loginSuccessTest(User user) {loginForm.loginAs(user);...
We call this – Entities Driven Testing approach (or EDT). If you are interested in more details, please follow the link https://jdi-docs.github.io/jdi-light/?java#entity-driven-testing or see how it looks on Github.
So, let’s get back to test cases. When I wrote tests and run them against build V2 they were passed but… they should not! See how the incorrect password error message looks like.
The text is correct – this is what we always validate with the functional approach, but the message’s background is broken. This is a really enlightening moment where you realize that all the tests you’ve written in your career are not that thorough and at risk of missing such issues.
I have no idea how such cases can be tested using the functional approach, and what’s worse is that you don’t even consider all these types of issues when writing the tests.
The conclusion here is that in addition to functional validation, it’s always good to have visual checks as well. And with Applitools you can do it in a simple and stable manner.
eyes.checkRegion("Alert: "+message, By.cssSelector(".alert-warning"));
The check() or checkRegion() method used here allows us to validate the view of the exact element.
| Testing Type | Applicable (1-5) | Comment |
|---|---|---|
| Functional approach (JDI Light) | 5 | this task is mostly for Functional testing |
| Visual approach (Applitools) | 3 | but in some cases, you can’t avoid visual validation |
Table sorting test
The next challenge in the hackathon was to validate the sorting of a data table. The scenario called for clicking on the table header (Amount column) and validating that the data in the sorted column is incorrect order. But this isn’t enough:
- First of all, you should validate that the values in the sorted column are in the correct order.
- Next, you need to check that the data in the table is still consistent. Data in rows should be the same, with only the order of the rows being changed.
- The problem becomes more complex because the cells in a row sometimes are images or colorful states and you can’t just validate values.
This is a really interesting task and I would like you to try it yourself.
Let’s consider how we’d approach this using the functional approach. Programmatically, you’d need to:
- Include a table header (or at least “Amount” element).
- Include a method that reads the “Amount” column values (remember that we need numbers to compare the values).
- Validate all parts of the row, including images and colors for each row.
With standard frameworks like Selenium, WebDriverIO, Cypres, or Selenide you could write hundreds of lines of code to properly interact with the table (especially with the cells containing images and color-related elements).
Thanks to JDI Light you can describe this table in just one row to test the data.
public static DataTable<TransactionRow, Transaction> transactionsTable;
And a few more rows if you would like to describe in detail the complex structure of each row. (See more details on Github.)
Via the functional approach, the validation has four steps:
- Get unsorted table rows with all expected data in each row.
- Click the Amount column header.
- Assert that values in the column amount are sorted in the correct order.
- Validate data consistency: Content of rows is not mixed and has all values the same in each row.
So, using the functional approach, the test script is about 5-7 lines of code (including row comparisons):
List<Transaction> unsortedTransactions = transactionsTable.allData();
transactionsTable.headerUI().select("AMOUNT");
transactionsTable.assertThat()
.rows(hasItems(toArray(unsortedTransactions)))
.sortedBy((prev,next) -> prev.amount.value() < next.amount.value());
And what about Visual validation with Applitools? Frankly speaking, we can do this validation with just two lines of code:
amountHeader.click(); eyes.checkElement(transactions, "Transactions Ascending");
But this approach has its limitations:
- First of all, we will need to make sure that the data is sorted correctly before saving the baseline image.
- Second and more crucial, we should recreate a screenshot each time data changed in the table.
The best solution here is to mix functional and visual validations with Applitools:
- Get unsorted table rows data and images.
- Click the Amount column header.
- Assert that values in the column amount are sorted in the correct order.
- Validate data consistency: Data in rows is not mixed, and rows images are the same as before sort.
See full code below:
List unsortedTransactions = transactionsTable.allData();
List images = transactionsTable.rowsImages();
transactionsTable.headerUI().select("AMOUNT");
transactionsTable.assertThat()
.rows(hasItems(toArray(unsortedTransactions)))
.sortedBy((prev,next) -> prev.amount.value() < next.amount.value())
.rowsVisualValidation("Description", images);
| Testing Type | Applicable (1-5) | Comment |
|---|---|---|
| Functional approach (JDI Light) | 5 | This task is mostly for functional testing. |
| Visual approach (Applitools) | 4 | In the simple case for the table with hardcoded data, you can check the whole table just in one line of code.And in some cases, you can’t verify without visual validation. |
Canvas Chart Test
The next task was to validate chart view – definitely task for visual validation.
At first glance, you have no way to get the data from canvas because the details of the chart do not exist in the DOM. With Applitools, you can validate chart view “before” and “after”.
compareExpenses.click();
eyes.checkElement(chart, "Expenses Chart 2017-2018");
showNextYear.click();
eyes.checkElement(chart, "Expenses Chart 2017-2019");
P.S. I would like to show you just one trick that is useful in this particular case. (Note: In most cases with charts you can’t get data.)
Using the following JavaScript snippet, you can get the data for this chart and use it in functional testing. But in any case, you can’t check colors and chart’s layout in this way.
public ChartData get() {
Object rowChartData = jsExecute("return { " +
"labels: window.barChartData.labels, " +
"dataset: window.barChartData.datasets.map(ds => ({ " +
"bgColor: ds.backgroundColor, " +
"borderColor: ds.borderColor, " +
"label: ds.label, " +
"data: ds.data })) " +
"}");
return gson.fromJson(gson.toJson(rowChartData), ChartData.class);
}
See more details on Github.
| Testing Type | Applicable (1-5) | Comment |
|---|---|---|
| Functional approach (JDI Light) | 1 | Only if you are lucky, you can get some data from the canvas; but in general, this is not a case for functional testing. |
| Visual approach (Applitools) | 5 | Visual validation is the best choice for testing canvas staff. |
Dynamic Content Test
The last task was to validate the advertisement content that comes from an external site.
I think the Applitools team added this task to show Applitools’ MatchLevel: Layout capability, which is great for validating an application’s layout without comparing the exact content.
Here’s the code to do this with visual validation:
eyes.setMatchLevel(MatchLevel.LAYOUT);
eyes.checkElement(advertisements, "Dynamic Advertisement");
But in this case, we can just check isDisplayed() (or validate the size of the advert blocks).
Here, visual validation is a little better, because it also validates the layout, but the difference is not so big for just one ad. However, I can see where the visual validation of an entire page where all of the content is different would be really powerful.
| Testing Type | Applicable (1-5) | Comment |
|---|---|---|
| Functional approach (JDI Light) | 4 | Cannot validate layout. |
| Visual approach (Applitools) | 5 | Validate all possible layout problems. |
Conclusion
The traditional approach that we often use for functional test automation is good for validating scenarios and data on the site; but if you validate only these things, you can easily miss layout or visual issues. Users abandon sites that have significant visual bugs, and in some cases, it makes the app unusable altogether.
Your application is the face of your organization. If you really care about your clients’ comfort and would like to show that clients can trust your company because you have good solutions, you must include visual testing together with functional testing in your overall testing strategy.
Roman Iovlev participated in the Applitools Hackathon in November 2019.
For More Information
- What is Visual Testing? A comprehensive explanation
- Complex Functional Testing – Simplified
- Functional Test Myopia
- Request an Applitools demo
- Sign up for a free Applitools account
Cover Photo by Markus Spiske on Unsplash
