How Do I Test Visually With Python?

I finished taking Gaurav Singh’s course, Automated Visual Testing with Python, on Test Automation University.  And, I passed. But, before I show you my certificate of completion, I want to give you a quick review of Gaurav’s course.

About The Instructor

Gaurav Singh advocates for test automation as part of his job. Gaurav serves as a test automation lead at Gojek, a logistics company based in Southeast Asia. In addition to Python (the subject of this course), Gaurav programs in Kotlin and Java.

He writes extensively about test automation. Some of his blog posts include:

• Why You Should Automate Tests Early
• Speed Up Your Workflow With IntelliJ
• Advice from a Test Journeyman
• A Guide to Load Testing via Locust

Why Python?

Python provides huge value for testing, according to the Automation Panda blogger, Andrew Knight. He writes:

“Python is object-oriented and functional. It lets programmers decide if functions or classes are better for the needs at hand. This is a major boon for test automation because (a) stateless functions avoid side effects and (b) simple syntax for those functions make them readable. pytest itself uses functions for test cases instead of shoehorning them into classes (à la JUnit).”

Some people think that interpreted code suffers performance issues, but Andrew points out that the performance suits testing – which can wait milliseconds to seconds for tests to complete. And, the flexibility you get from choosing Python provides huge advantages to test development.

Prerequisites

You need to know Python to take Gaurav’s course, because he does not teach the language or teach you how to use it for test purposes. If you know Python but have never used it for testing, Gaurav points you to Jess Ingrasselino Ed.D.’s excellent course on Test Automation University, Python for Test Automation.

You will need Python 3. By default, Macs come with Python 2 installed, so you will need to install Python 3. Gaurav gives you instructions for doing this.

You also need an IDE. Gaurav demonstrates using PyCharm for his IDE – you can download it pretty easily. Again – good instructions. You can also choose another IDE – Virtual Studio Code, for example. But the examples all work with PyCharm.

Course Structure

This course has an introduction and 11 chapters. Each of the 11 chapters includes a quiz. You need to take and succeed with the quizzes to pass the course and get the course certificate.

The 11 chapters include:

• 1 – Introduction to Visual Testing
• 2 – Setting Up Dependencies
• 3 – Getting the API Key
• 4 – Setup PyCharm and Initializing Applitools Eyes
• 5 – Running Your First Test
• 6 – Understanding Match Levels
• 7 – Checkpoints
• 8 – Organizing Tests
• 9 – Visual Validation of PDF Files
• 10 – Analyzing Test Results
• 11 – Integrations

Getting Going

The first three steps are the parts to get organized. In Chapter 1, Gaurav brings up some concepts around visual testing – why it’s important. He shows some examples of pages that cause some percentage of users to abandon an app. For instance, if the button that drives customers to buy your product gets covered with text. The button may still function, but what percentage of your customers pause when they see this kind of error? And, the bummer is that  your existing test automation cannot catch this kind of error. The button may be findable and clickable for your test automation – so your tests will continue to pass.

In Chapter 2, Gaurav gives you instructions for installing all the python test software. If you’re experienced with Python, you will find these instructions straightforward. If not, you might encounter differences between your environment and his. Gaurav points to a couple of sites to help you install Python3. I did, but ran into some issues with my environment on my Mac. I gave up and moved to Linux.

Also in Chapter 2, Gaurav gets you to install the Eyes Selenium software for Python. This lets you call on Applitools Eyes calls in your Python code. Gaurav leaves the explanation for this code to later chapters.

Next, he makes sure you can run Applitools Eyes by getting sure you sign up for a free account in Chapter 3. To run Applitools, you need an API key, which you can get once you have an account.

Finally, he makes sure your PyCharm IDE can work with Applitools Eyes by giving you some initial coding steps to initialize Applitools and ensure the Applitools API Key gets loaded.

Running Visual Tests with Python

Next, Gaurav runs through some tests demonstrating calls to Applitools Eyes once you have navigated to a given page.

In Chapter 5, Gaurav shows how Applitools Eyes can validate the output of a given functional task. He uses a bookstore app that Angie Jones developed to demonstrate the limits of code-based assertions.

Gaurav shows how to build a test for the app in Python. He shows how to exercise the app with Python, and how to call Applitools to check the page. By using a search function that presents a single result, he captures the resulting page image as a baseline.

Gaurav then introduces a visual difference. He changes the color of the pricing box in the CSS for the app. The text is correct, but the visual image differs. He shows that Applitools catches the difference, which a text assertion might catch if a tester coded for the CSS change.

In Chapter 6, Gaurav digs into match levels in Applitools:

• Exact – pixel comparison
• Strict (the default) – using AI to inspect for visually-meaningful differences
• Content – checking for content and ignoring color differences
• Layout – handling dynamic content (like news) and ensuring that the page structure and layout remain consistent

Gaurav explains why each match level matters.  Basically, no one uses Exact in production – it just demonstrates how many more errors your typical pixel checker uncovers that don’t matter to a user. Layout and Content have specific purposes. Most of the time, use Strict.

In Chapter 7, Gaurav goes through some of the special capabilities in Applitools. For instance, if an app page extends beyond the viewport, you can configure Applitools to capture an entire page. Applitools will scroll through and capture the entire page to its horizontal and vertical maximum size for comparison purposes. You can also capture specific regions, or regions-by-element, and run different comparisons on those regions. You can even capture a region inside an iFrame.

Becoming Productive with Visual Validation in Python

The rest of the course focuses on how you can become productive using Applitools for visual testing with Python.

Chapter 8 focuses on how to organize tests into batches for test suites. You can capture multiple tests on a single page and see how the behavior compares for that page. When users have multiple functions to exercise on a single page, they find really powerful to group all the tests for a single page together.

Chapter 9 explains how you can use the Applitools Image Tester application to validate PDF files. How do you test the PDF output of your applications? Applitools gives you a stand-alone app to make those comparisons. The Image Tester application can compare two image files – including PDF files – and give you a comparison. While Image Tester is a .jar and runs in Java, you can direct Selenium to call Image Tester from inside your Selenium Python tests.  You can use the Applitools controls to run strict, content, or layout comparisons of your PDFs – and you can choose to ignore regions you know will differ (like, say, a timestamp).

Chapter 10 dives into the ways you can analyze test results.  Chapter 11 focuses on integrations. Both chapters help you understand how to get the most out of using Applitools with your development and test infrastructure.

As always, I include my certificate of completion:

Conclusion

Gaurav’s course just focuses on using Applitools with Python. While he covers a bit of good test writing style, he wants to show you how to use Applitools. If you already know Applitools, this course provides a great refresher on the basic capabilities, and you will get some idea of the power of adding Applitools to your existing tests written in Python.

Next Steps

I’d recommend taking Raja Rao’s course on Modernizing Functional Test Automation Through Visual AI. While the course focuses on Java, its principles help explain how to cut down on coded assertions in your end-to-end tests and use the Visual AI in Applitools to validate your application output.

Sign up for a free Applitools account.

Request an Applitools demo.