Visual UI Testing as an Aid to Functional Testing

Visual UI Testing vs Functional Testing

Visual tests are usually used to check the visual aspects of your application. Makes sense, no? To automate tests that verify that your application looks OK, you write visual tests, and use visual testing tools such as Applitools Eyes. But visual testing tools can also be used for functional tests.

Really? How can visual testing help us write functional tests, tests that check how the application behaves? First, let’s understand how visual testing and functional testing of applications work.

Functional Testing

A functional test (in this blog post! I am sure that you can find other people that define it differently) is a test that starts an application in a certain state, performs a set of actions, and tests that the application is in the correct state after those actions.

So a functional test looks like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Assert that the actual state of the application is the expected one

Yes, you might argue that it is not the test that starts the application, and that the test should always create the specific set of data using other actions. You could, but that’s bikeshedding. True, no functional tests look exactly like this, but the idea is correct:

• action, action, action
• assertion, assertion, assertion
• action, action, action
• assertion, assertion, assertion

Let’s take an example of a to-do list application:

One functional test would be:

• Action: check a todo
• Assertion: it is checked in the “All” tab
• Assertion: it is gone from the “Active” tab
• Assertion: it is checked in the “Completed” tab

 

Implementing functional assertions

How would one go about implementing the assertions? If it’s a web application, and we’re using a browser automation library like Selenium Webdriver, then the tester would need to check whether the specific todo’s checkbox is checked, and also check that the other checkboxes are not checked. Also, check that the text of the todo is striked out, while the others are not striked out.

Oh, and don’t forget to check the “2 items left” text in the bottom left of the list.

And each such check is a pain in the butt — the tester needs to determine what the id of the control they are checking is. And sometimes there is no specific id (or class), and so the tester needs to go to the developer and ask them to add an id for testing, or if the tester is in an agile team, maybe add it themselves!

Also: this needs to be done across hundreds of tests, and thousands of assertions. This is practically impossible, and definitely impossible to maintain. What usually happens is that the developer chooses a subset of assertions, and tests just those. In our example, it would only assert that the checkbox of the todo item is checked, and forego the other tests.

Visual Testing

We’ve seen what functional tests are, and one of the problems associated with them. Let’s move on to visual tests: a functional test (in this blog post! I am sure that you can find other people that define it differently) is a test that starts an application in a certain state, performs a set of actions, and tests that the application looks good after these actions.

See how similar they are? A visual test looks like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Assert that the application lookscorrect

This is exactly like functional tests, except that the assertions are different. So let’s look at how assertions are performed in visual tests. How does a visual test go about testing that an application looks correct?

Implementing visual assertions

How do visual assertions work? Well, first of all, we take a screenshot of the screen — we are doing a visual assertion after all. But this screenshot, alone, is not enough. What we do is compare this to a baseline screenshot that we took in a previous run of the test. If the algorithm comparing the current screenshot to the baseline one tells us that they’re the same, then the assertion passes, otherwise it fails.

And how do we generate the screenshot? In a two phase process:

1. We define the screenshots that are created by the assertion of the first test the baseline screenshots. The tester manually checks each one of those baseline screenshots.
2. Each time the test failsit could be because there is a bug in the software, but it could also be because the software changed and this is the new The tester in this case manually checks each one of those new screenshots and declares them as the new baseline.

How does visual comparison work?

Unfortunately, visually comparing two screenshots seems easy, but it’s not. Theoretically, it should be as easy as comparing pixels, but this, unfortunately, is not the case. Different operating systems, browsers, even different versions of the same browser generate slightly different images. Moreover what if the screenshot includes the current time? You would need to check everything except that region of the screen.

There are even more concerns, which we should outline in a future blog post. Fortunately, tools like Applitools Eyes hide us from these concerns.

Using Visual Assertions in Functional Tests

Now that we understand how functional testing works, we understand that there’s no difference between functional and visual tests in how they are structured — the only difference is in what they test and in how they implement the assertions:

• Functional assertions check each and every control on the page to ensure that they have the correct value
• Visual assertions compares a screenshot of the page against a manually verified baseline screenshot.

Visual assertions seem pretty easy to use. Just manually verify the first screenshot, and just take screenshots of the pages after the actions in the test. No need to check each control, determine the id of the control, or ask the developers to add ids to controls. Just take a screenshot and let visual testing frameworks like WebDriverCSS or Applitools Eyes compare them.

And we’ve seen how functional assertions are hard. But what if we can use visual assertions in functional tests? Something like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Take a screenshot to verify that the data is correct

Think of the todos page after checking the todo item:

With one screenshot, we can check that the check was checked (ouch, too many “check”-s), verify that the other todo items weren’t, that the “2 items left” is correct, that the todo item is in strike-out, and probably other stuff which we didn’t even notice.

Many front-end developers use this kind of “testing against a baseline”, but what they test is the HTML of the components, and not how they look like. But the idea is similar. The difference is that it is much nicer to check the difference from a baseline between two images, than to compare a textual diff of HTML.

 

Pleasant side benefit of using visual assertions: you get to check the visual look of your application for free!

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.