Learn the differences between Selenium, Cypress, and Playwright when it comes to automated screenshot testing and screenshot quality.

The Issue with Testing Screenshots

Capturing screens is a fundamental piece of testing web assets and reporting defects. There are other use cases for these screenshots, too. I work as the Director of Quality Control for my company. We service the pharmaceutical industry. I was tasked with leveraging existing automation processes to deliver high quality images of client sites to submit for legal review more efficiently.

My journey to find the best way to take screenshots to fill this niche was filled with ups and downs. The journey started with Selenium WebDriver, which fell short of meeting the requirement. Next I moved to Cypress because it showed great promise. It also had limitations I could not work around. Finally, I landed on Playwright and was able to meet the desired output.

I will err on the side of caution and use a site that is completely unrelated to any of our client work. Snopes seems like a safe choice. Besides, we will not do any advanced or prolonged calls against their site. Be nice to your internet neighbors!

Screenshot Testing in Selenium WebDriver

Automating full page screenshots with high fidelity has been troublesome in the past. Selenium WebDriver, the de facto champion of browser automation, lacks a method to capture a full page. Having said that, Selenium drivers do have methods for screenshots. The sample code for taking a screenshot with Selenium WebDriver is straightforward:

takeSnopesScreenshot.js

let {Builder} = require('selenium-webdriver');
let fs = require('fs');

(async function checkSnopes() {
    let driver = await new Builder()
    .forBrowser('chrome')
    .build();

    await driver.get('https://www.snopes.com');
    // Returns base64 encoded string
    let encodedString = await driver.takeScreenshot();
    await fs.writeFileSync('./homepage.png', encodedString, 'base64');
    await driver.quit();
}())

Here are the steps we took to capture our image:

1. We start by bringing in our Builder class so we can create new webdriver instances.
2. We also need access to the file system so we can save our image.
3. Next, we instantiate a new webdriver for the Chrome browser.
4. Then we instruct the webdriver to open https://www.snopes.com. This should probably be put into a variable. We’ll leave it here since we are only going to one page on the site.
5. Next, we take a screenshot of the web page.
6. Then we save the image to disk by converting the base64 encoded string into a .png file.
7. Finally, we free the driver from service.

We get a screenshot like this after running the code:

The resulting image looks nice. A couple of things immediately stood out to me as problems here:

1. It is not a full-page image.
2. The scrollbar is displayed.

We attempt to fix these issues by setting the window size after initializing the webdriver:

takeSnopesScreenshot2.js

let {Builder} = require('selenium-webdriver');
let fs = require('fs');

(async function checkSnopes() {
    let driver = await new Builder()
    .forBrowser('chrome')
    .build();

    await driver.manage().window().setRect({ width: 1024, height: 2000 });

    await driver.get('https://www.snopes.com');
    // Returns base64 encoded string
    let encodedString = await driver.takeScreenshot();
    await fs.writeFileSync('./homepage.png', encodedString, 'base64');
    await driver.quit();
}())

Two things come to mind if this works the way I anticipate:

1. I may not know the dimensions of the longest page.
2. I do not want to add more code to maintain.

The resulting screenshot after resizing the window:

The second image is high quality. Obviously, the setRect() method did not force the full page image like I had hoped.

This works great for capturing what is in the current viewport, but not so well for a page where scrolling to see all the content is needed. Instead, we would need to “stitch” the images together to get a full-page screenshot. There is also a third-party plugin named aShot for the Java users of Selenium. Not much help in a .NET and JavaScript shop. No easy, consistent method to do full page captures is a deal breaker for my use case.

Screenshot Testing in Cypress

Speedy delivery, simplicity, and consistency are keys for a successful submission. The Champ (Selenium WebDriver) is too cumbersome. That leads to a scrapy newcomer, Cypress.io. Cypress is an amazing framework. Easy to use – check. Fast as lightning – double check. It even has a built-in Screenshot API that captures full page images – yes, please. Here is a quick example of what that looks like:

snopes.test.js

describe('submission screenshots', () => {
    it('takes screenshot of the home page', () => {     
        cy.visit("https://www.snopes.com");   
        cy.screenshot('/screenshots/cypress-homepage');
    });
});

Here’s the breakdown of what we did:

1. We set up our test suite using the built-in support from Mocha.
2. Next, we set up the test.
3. Then we visit snopes.com.
4. Finally, we take the screenshot and save it as cypress-homepage.png.

A few things to note here:

1. The code feels less intimidating and more accessible.
2. We did get a full-page image!
3. There is a problem, though. Can you see it? (click to view full size image)

Another great-ish image. Cypress, however, appears to suffer from some of the same stitching and consistency issues found in Selenium. Images were not consistent. Some were blurry. Others were missing content – usually near the bottom of the web page. This is not likely the end of the world for most use cases, but those are killer obstacles to overcome to create legal submission documents for a federally-regulated company.

Screenshot Testing in Playwright

I was quickly losing faith in being able to find a good solution to help with our automated screenshot efforts. Then, I found Playwright, Microsoft’s entry for automated UI testing. It is a fantastic framework that checks the boxes for being speedy and easy to use. It also has the capability to capture full page images out of the box. It looks promising. The only question left is the quality of images.

Spoiler alert – It works like a charm . Let’s take a look at some of the code:

snopes.test.js

const {test, expect} = require('@playwright/test');

// The site I will capture screens from.
const url = "https://www.snopes.com/";
const savePath = "./screenshots/";

// I have a wrapper function to beef up Playwright's screenshot method.
async function takeScreenshot(page, label) {   
    await page.screenshot({
      path: `${savePath}${label}.png`,
      fullPage: true,
    });
}

// Start the testing
test.describe('Submission Screenshots:', async () => {

    test('0.0 - Homepage', async({page}) => {
        await page.goto(url);    
        await takeScreenshot(page, "playwright-homepage");
    });

    test('1.0 - Menu', async({page}) => {
        await page.goto(url);
        await page.click("//html/body/header/div[1]/nav/div/div[3]/div/a[2]");
        await takeScreenshot(page, "playwright-menu");
    });

    test('2.0 - Search Results', async({page}) => {
        await page.goto(url);
        await page.fill('input[name="s"]', 'scam');
        await page.press('body', 'Enter');
        await takeScreenshot(page, "playwright-search-results");
    });
});

This snippet is a bit more polished than the others.

1. We moved the URL into a proper variable.
2. Next, we created a variable to hold the saved images path.
3. Next is a wrapper function to take care of the additional information (label) needed for each image.
4. Inside we call Playwright’s screenshot method.

From there the test suite and individual tests take on the Mocha format like Cypress.

These are the screenshots we get after running the code (click for full size):

Those images look amazing. Full page. Crisp. Consistent. The time taken to run through the screens is not bad for what we are doing, but there is considerable overhead when doing file I/O. We added 20 seconds by writing those large files to disk.

The chart below lists the times for capturing just the homepage image versus a visit to the page for each framework. Keep in mind that these times reflect visiting an external site on my personal internet connection. Times on a local build will be significantly faster.

Conclusion

These are simple examples of how to use the screenshot capabilities of three common automated testing frameworks. The differences of the resulting images were quite shocking. I presumed that a screenshot was a screenshot.

It is important to note that different here does not mean bad. You have to take your use case into account. Take the extra 20 seconds needed to write three full page screens to disk for example. That is not a typical use case, but it is perfect for the time savings we set out to recoup by automating submission screenshots. Is it perfect? No, but it gets closer with every site we automate.

There is room for enhancement. We can move from capturing and saving screenshots into automated visual testing with Applitools. Visual testing with Applitools is an enhancement worth making, especially for regulatory or compliance testing, because we will be alerted whenever a visual change is detected. This is done by swapping out the screenshot method calls with calls to Applitools Eyes.

The post Screenshot Testing with Selenium, Cypress and Playwright: 3 Popular Automation Tools Deliver Amazingly Different Screenshots appeared first on Automated Visual Testing | Applitools.

The fourth industrial revolution – the digital revolution – has strong requirements for companies operating under strict business regulations.

Particularly, in the healthcare sector, companies must spend great efforts to survive “digital Darwinism”. The healthcare market is highly competitive and strongly regulated at the same time. Healthcare, pharmaceutical, and medical device companies invent new medicines and other products in a highly volatile business landscape. On one hand, they have to act as agile as possible, considering time-to-market delivery, on the other hand, they face strict compliance regulations, like FDA, HIPAA, etc.

The question is how healthcare companies can deliver new products and services at high speed while meeting their regulatory compliance obligations?

In this article, you will find the answer based on an example of the FDA-Requirement “Back-Box-Warning”.

The FDA (Food and Drug Administration) prescribes warnings and precautions on the package insert for certain prescription drugs. As these warnings are framed in a “Black Box” to catch the eye of the reader, they are also referred to as “Black Box Warnings” and can be found at the beginning of the package insert (see Picture 1) or in the drug description in the online store (see Picture 2).

If the FDA finds serious violations due to missing or unreadable Black Box Warnings, the FDA can take legal actions against a company.

Let me present an example showing how automated visual regression tests for web sites and PDF-documents can be implemented to automatically verifies regulatory requirements.  

The FDA’s General Principles of Software Validations recommends using visual regression testing for images and documents. The FDA makes this recommendation for companies using a software development lifecycle (SDLC) approach that integrates risk management strategies with validation and verification activities, including defect prevention practices.

What is visual regression testing?

Visual regression testing expands regression testing, where a program, or parts of it, repeatedly gets tested after each modification. To additionally avoid unintentional changes in design elements, positioning, and colors, QA-Teams use visual regression testing as part of their testing strategy and general quality assurance.

Visual regression testing can discover visual defects, obvious or not, due to modifications to the software. In practice, a baseline of original or reference images is stored. This “source of truth” can be compared after each program modification against a collection of “new” screenshots of a user interface. Each difference against the baseline will be highlighted and can serve as an alert.

Additionally, visual regression testing doesn’t only look at differences between the source and current status. It has the possibility to compare the source against any historical status on a UI level, independent of HTML, CSS, and JavaScript differences. It can also be used to highlight differences between documents, like PDFs, in the layout, or the content itself. For example, a missing black box warning in a package insert for a drug would be marked as a difference.

How to best use visual regression testing?

Many visual testing tools, such as Selenium, mark differences between screenshots or PDF-documents as passed or failed. With visual regression testing, you can choose which differences across multiple browsers and devices to accept or not. For example, a picture, which is displayed in a different resolution on a web-page after a program change may cause a problem in completing a user-action due to overlapping with an FDA-required text (see Picture 3). This can cause a reporting to the FDA by a competitor and a warning letter would be sent to the legal department of the healthcare company.

Visual regression testing tools and libraries, like Wraith, Gemini, and other Selenium-related frameworks, needs a deep knowledge by testers and high effort in installation and setup. The Applitools AI-Platform, where no installation-, setup- or coding-knowledge is required, could be a great alternative to start automated visual regression testing.

The Applitools Eyes cross-environment testing feature allows you to test your application on multiple platforms using a single, common baseline. The match level (Strict, Layout, Content, Exact) determines the way by which Eyes compares the checkpoint image with the baseline image.

Additionally, the Applitools PDF Tool allows you to easily run visual UI tests on a collection of PDF-files, by placing them inside a directory (see Picture 4). It runs as a standalone jar file and can be invoked as a process by any programming language and in your continuous delivery pipeline.

Summary

If you want to continuously deliver new products and services within a software development lifecycle, at high speed while considering regulatory compliance regulations, you should have an eye on visual regression testing. It can be used for automated testing by comparing hundreds or thousands of artifacts like images and PDF-Documents at very much speed. Therefore, it provides long term cost efficiency by avoiding extensive manual tests, especially when dealing with frequent changes on a UI or document base.

The post Continuous visual regression testing to enable regulatory compliance in the healthcare sector appeared first on Automated Visual Testing | Applitools.