Today’s software applications are getting more complicated, thus every testing team needs to focus on expanding test coverage. To achieve this goal, it is important to use a combination of testing types, such as unit testing, integration testing, system testing, and end-to-end testing, depending on the software application’s complexity and requirements.

End-to-End (E2E) testing is designed to ensure that all components of a software application are working together correctly and that the system as a whole meets the desired functionality, performance, and reliability requirements.

Cypress is a popular open-source end-to-end testing framework for web applications. It is designed to make the testing process easier and more efficient for developers. One of the unique features of Cypress is that it runs the tests within the browser, which means that it can provide better control and visibility over the application under test.
In this blog on End to End testing, we will deep dive into performing Cypress End to End testing on a local Cypress grid and will explain how to start automating visual tests with Applitools Eyes and the Ultrafast Grid using Cypress in JavaScript.

What is End to End Testing?

End-to-end (E2E) testing is a software testing strategy that verifies an application’s complete flow from beginning to end. It is a type of functional testing that tests the application’s behavior as a complete system, rather than testing individual components in isolation.

E2E testing simulates a real user scenario and covers all aspects of the application, including user interfaces, APIs, databases, and other integrations. It typically involves testing multiple components of an application to ensure that they work together as expected and fulfill the requirements of the business or end-users.

E2E testing is typically performed after other types of testing, such as unit testing and integration testing, have been completed. It is used to validate that the entire system works together seamlessly and to identify any issues that may have been missed in earlier stages of testing.

Why is end-to-end testing necessary?

End-to-end testing (E2E testing) is a type of software testing that tests the entire system or application from start to finish, simulating real-world user scenarios.

Unit testing alone is not enough to ensure the quality and reliability of software. While unit testing is an important part of the testing process, it only verifies the behavior of individual components or modules of the software in isolation. It does not guarantee that the software will work correctly when integrated with other components or modules.

This is where integration testing enters into the picture. Integration testing focuses on testing the interaction between two or more components of a system, to ensure that they work together correctly. However, even if all the individual components pass integration testing, there may still be issues with the overall system when all the components are put together. This is where end-to-end testing comes in – it tests the entire system from start to finish.

Cypress is a popular automation testing framework that is designed specifically for end-to-end testing. It runs tests directly in the browser, allowing it to provide an experience that is similar to how users interact with the application. This makes it easier to identify any issues that users might face, as the testing environment is as close to the real-world experience as possible.

To understand End to End testing, Let’s take a closer look at Mike Cohn’s test automation pyramid. We routinely do each level of testing listed in this pyramid while running automated Cypress testing.

Testing Pyramid Layers

The automation pyramid is a popular framework introduced by Mike Cohn that helps teams to plan and prioritize their testing efforts. It includes three levels of testing, which are:

1. Unit Tests: At the base of the pyramid are the unit tests, which test individual code components such as functions, methods, and classes. Unit tests are typically written by developers and are executed frequently during the development cycle. They are essential in ensuring that individual components of the application work as expected and can catch issues early in the development process.

2. Integration Tests: The middle layer of the pyramid consists of integration tests, which test how different components of the system work together. Integration tests ensure that the various parts of the application can communicate and interact with each other seamlessly. These tests are typically automated and are executed after the unit tests have passed.

3. End-to-End Tests: The top layer of the pyramid is end-to-end testing, which tests the entire application workflow from start to finish. These tests simulate real user scenarios and help ensure that the application functions as expected in a production environment. End-to-end tests are typically automated and are executed less frequently than the lower level tests.

Benefits of End-to-End Testing

There are several benefits of End to End testing. Some of the benefits of E2E testing include:

1. Increased Confidence: E2E testing provides a higher level of confidence in the software application by testing all components together. This testing approach ensures that all the components are integrated correctly and are working as expected.
2. Improved Quality: By testing the application from end-to-end, helps to identify and fix bugs earlier in the development process. This enhances the overall quality of the software.
3. Enhanced User Experience: E2E testing ensures that the application is working as expected for the end user. This helps to provide a better user experience and can lead to increased customer satisfaction.
4. Time and Cost Savings: E2E testing helps to identify issues early in the development cycle, which can save time and money by reducing the need for costly rework later in the process.
5. Better Collaboration: E2E testing promotes better collaboration between different teams working on the same application. This testing approach helps to identify issues that may be caused by a lack of communication between teams.
6. Increased Productivity: By automating the testing process, E2E testing can help to increase productivity by reducing the time and effort required to manually test the application.

Faster Time-to-Market: By catching defects earlier in the development process, end-to-end testing can help to reduce delays and accelerate the time-to-market of the application.

Frameworks for End to End testing

There are several popular frameworks for end-to-end testing, including:

Cypress

Cypress is a JavaScript-based end-to-end testing framework that provides a simple and intuitive API for testing web applications. Cypress supports modern web development technologies like React, Angular, Vue.js, and more. It provides a built-in test runner, and it runs tests in the browser, which makes it fast and reliable.

Cypress runs tests inside the browser; it also provides detailed information about what’s happening at every step of the test, including network requests, console output, and DOM changes. This makes it easier to identify and troubleshoot issues and helps ensure that the application is working as intended.

Cypress Trends on GitHub

The following information is taken from the official website of Cypress GitHub repository:

• Stars: 43.3k
• Forks: 2.8k
• Used By: 797k
• Releases: 303
• Contributors: 427

WebdriverIO

WebdriverIO is a popular open-source testing framework for Node.js that allows developers to automate web applications in a simple and efficient way. It uses the WebDriver API to communicate with browsers and supports a variety of testing frameworks, including Mocha, Jasmine, and Cucumber.

.WebdriverIO Trends on GitHub

The following information is taken from the official website of WebdriverIO GitHub repository:

• Stars: 8.1k
• Forks: 2.3k
• Used By: 50.5k
• Releases: 305
• Contributors: 491

Nightwatch.js

Nightwatch.js is an open-source Node.js-based end-to-end testing framework used to automate browser testing. It provides a simple and easy-to-use syntax for writing automated tests in JavaScript and allows you to run tests in real web browsers like Chrome, Firefox, and Safari.

Nightwatch.js uses the WebDriver protocol to communicate with the browser and control its behavior. It also includes a powerful built-in assertion library that makes it easy to write test assertions and helps you quickly identify issues with your web application.

Nightwatch.js Trends on GitHub

The following information is taken from the official website of Nightwatch.js GitHub repository:

• Stars: 11.4k
• Forks: 1.1k
• Used By: 142k
• Releases: 219
• Contributors: 112

Protractor

Protractor is an open-source end-to-end testing framework for Angular and AngularJS applications. It is built on top of WebDriverJS and uses Jasmine syntax for writing test scripts. Protractor is designed to simulate user interactions with the application and to verify that the application behaves as expected.

Protractor Trends on GitHub

The following information is taken from the official website of Protractor GitHub repository:

• Stars: 8.8k
• Forks: 2.4k
• Used By: 1.9m
• Contributors: 250

TestCafe

TestCafe is an open-source end-to-end testing framework that allows you to automate web testing without using browser plugins. TestCafe is built on top of Node.js and provides a simple and powerful API for testing web applications.

TestCafe Trends on GitHub

The following information is taken from the official website of TestCafe GitHub repository:

• Stars: 9.6k
• Forks: 677
• Used By: 12.3k
• Releases: 390
• Contributors: 117

Benefits End to End Testing Using Cypress

Here are some of the features of Cypress End to End testing:

1. Easy Setup: Cypress has a simple setup process that doesn’t require any additional drivers or libraries. You can get started with Cypress by installing a single package.
2. Automatic Waiting: Cypress automatically waits for elements to appear and become intractable before executing commands. This ensures that the tests are not affected by the timing of the application’s response.
3. Real-time Reloads: Cypress provides real-time reloads, which means that as you make changes to your code or tests, the application will automatically reload, and the tests will be re-run.
4. Interactive Debugging: Cypress provides an interactive test runner, which allows you to debug your tests by stepping through them, setting breakpoints, and viewing the application’s state at any point in time.
5. Time Travel: Cypress allows you to go back and forth in time to see what happened during the execution of a test. This feature is useful for debugging and understanding the behavior of your application.
6. Cross-browser Testing: Cypress allows you to run your tests on multiple browsers and viewports simultaneously. This helps you ensure that your application works correctly across different environments.
7. Network Traffic Control: Cypress allows you to control the network traffic of your application. You can stub, spy, and mock network requests to simulate different scenarios.
8. Automatic screenshots and videos: Cypress automatically takes screenshots and records videos of your tests, which makes it easy to see what went wrong when a test fails.

Frameworks for End to End testing

There are several popular frameworks for end-to-end testing, including:

Cypress

Cypress is a JavaScript-based end-to-end testing framework that provides a simple and intuitive API for testing web applications. Cypress supports modern web development technologies like React, Angular, Vue.js, and more. It provides a built-in test runner, and it runs tests in the browser, which makes it fast and reliable.

Cypress runs tests inside the browser; it also provides detailed information about what’s happening at every step of the test, including network requests, console output, and DOM changes. This makes it easier to identify and troubleshoot issues and helps ensure that the application is working as intended.

Cypress Trends on GitHub

The following information is taken from the official website of Cypress GitHub repository:

• Stars: 43.3k
• Forks: 2.8k
• Used By: 797k
• Releases: 303
• Contributors: 427

WebdriverIO

WebdriverIO is a popular open-source testing framework for Node.js that allows developers to automate web applications in a simple and efficient way. It uses the WebDriver API to communicate with browsers and supports a variety of testing frameworks, including Mocha, Jasmine, and Cucumber.

.WebdriverIO Trends on GitHub

The following information is taken from the official website of WebdriverIO GitHub repository:

• Stars: 8.1k
• Forks: 2.3k
• Used By: 50.5k
• Releases: 305
• Contributors: 491

Nightwatch.js

Nightwatch.js is an open-source Node.js-based end-to-end testing framework used to automate browser testing. It provides a simple and easy-to-use syntax for writing automated tests in JavaScript and allows you to run tests in real web browsers like Chrome, Firefox, and Safari.

Nightwatch.js uses the WebDriver protocol to communicate with the browser and control its behavior. It also includes a powerful built-in assertion library that makes it easy to write test assertions and helps you quickly identify issues with your web application.

Nightwatch.js Trends on GitHub

The following information is taken from the official website of Nightwatch.js GitHub repository:

• Stars: 11.4k
• Forks: 1.1k
• Used By: 142k
• Releases: 219
• Contributors: 112

Protractor

Protractor is an open-source end-to-end testing framework for Angular and AngularJS applications. It is built on top of WebDriverJS and uses Jasmine syntax for writing test scripts. Protractor is designed to simulate user interactions with the application and to verify that the application behaves as expected.

Protractor Trends on GitHub

The following information is taken from the official website of Protractor GitHub repository:

• Stars: 8.8k
• Forks: 2.4k
• Used By: 1.9m
• Contributors: 250

TestCafe

TestCafe is an open-source end-to-end testing framework that allows you to automate web testing without using browser plugins. TestCafe is built on top of Node.js and provides a simple and powerful API for testing web applications.

TestCafe Trends on GitHub

The following information is taken from the official website of TestCafe GitHub repository:

• Stars: 9.6k
• Forks: 677
• Used By: 12.3k
• Releases: 390
• Contributors: 117

Benefits End to End Testing Using Cypress

Here are some of the features of Cypress End to End testing:

1. Easy Setup: Cypress has a simple setup process that doesn’t require any additional drivers or libraries. You can get started with Cypress by installing a single package.
2. Automatic Waiting: Cypress automatically waits for elements to appear and become intractable before executing commands. This ensures that the tests are not affected by the timing of the application’s response.
3. Real-time Reloads: Cypress provides real-time reloads, which means that as you make changes to your code or tests, the application will automatically reload, and the tests will be re-run.
4. Interactive Debugging: Cypress provides an interactive test runner, which allows you to debug your tests by stepping through them, setting breakpoints, and viewing the application’s state at any point in time.
5. Time Travel: Cypress allows you to go back and forth in time to see what happened during the execution of a test. This feature is useful for debugging and understanding the behavior of your application.
6. Cross-browser Testing: Cypress allows you to run your tests on multiple browsers and viewports simultaneously. This helps you ensure that your application works correctly across different environments.
7. Network Traffic Control: Cypress allows you to control the network traffic of your application. You can stub, spy, and mock network requests to simulate different scenarios.
8. Automatic screenshots and videos: Cypress automatically takes screenshots and records videos of your tests, which makes it easy to see what went wrong when a test fails.

Set up Cypress For End to End Testing

To create a new project for Cypress automated testing, follow the steps listed below.

Step 1: Generate package.json.

• Create a project, let’s name it as cypress_applitools
• Use the npm init command to create a package.json file

Step 2: Install Cypress.

Install Cypress by running the command in the newly created folder:

npm install cypress –save-dev

OR

yarn add cypress –dev

Above command will install Cypress locally as a dev dependency for your project.

As shown below, Cypress version 12.11.0 is reflected after installation. The newest Cypress version at the time this blog was being written was 12.11.0.

Below is a diagram of Cypress’s default folder layout. The “e2e” folder is where test cases can be created.

About Project structure of Cypress

Cypress has built a default folder hierarchy when it opens for the first time, as can be seen in the screenshots. Each of these files and folders that Cypress created is described in detail below.

• e2e: All test cases are stored under this folder. This folder contains the actual test files, written in JavaScript, that define the tests to be run.
• Fixtures: This folder contains any data files that are needed for the tests, such as JSON or CSV files.
• Support: There are two files inside the support folder: commands.js and e2e.js
  • command.js: Is the file where your frequently used functions and unique commands are added. It has functions like the login function that you may use in various tests. You can alter some of the functions Cypress generated for you right here.
  • e2e.js: This file is executed before each and every spec file. This file is an excellent location for global configuration and behavior that alters Cypress in the same way as before or before. It just imports commands.js by default, but you can import or need more files to keep things organized.
• Node_Modules: The node_modules directory will have all the node packages installed and all test files will have access to them. When you install Node packages using NPM, they are downloaded and installed in the node_modules directory, which is located in the root directory of your project
• cypress.config.json: cypress.config.json is a configuration file used by Cypress to override the default configuration settings for a project. It is similar to cypress.json, but it is intended to be used as a per-environment configuration file.

Some examples of configuration options that can be set in cypress.config.json include:

• baseUrl: The base URL for the application being tested.
• testFiles: A list of test files to include or exclude from the test suite.
• video: Configuration options for Cypress video recording.
• screenshots: Configuration options for Cypress screenshots.

Basic constructs of Cypress

Cypress used Mocha’s syntax for developing test cases. Key constructs that are frequently used in Cypress test development are listed below.

• describe(): This method is used in Cypress (using Mocha’s syntax) to group together related test cases. It takes two arguments. It takes two arguments: A string that describes the group of test cases (e.g. “Login Page Tests”) and another argument, a callback function that contains the individual test cases (using the it() method).
• it(): This method is used to define an individual test case. It requires two arguments: a string that specifies the test scenario and a callback function that has the test code itself.
• before():  This method is used to run the code under before() block before any test case. The before() method takes one argument: a callback function that contains the setup code to be executed before any of the test cases
• after(): This method is used to run a cleanup once all the test cases are executed. The after() method takes one argument: a callback function that contains the cleanup code to be executed after all the test cases
• beforeEach(): This method is used to run the code under beforeEach() block beforeEach.The beforeEach() method takes one argument: a callback function that contains the code to be executed before each test case
• afterEach(): This method is used to run a cleanup function after each test case. The afterEach() function takes one argument: a callback function that contains the cleanup code to be executed after each test case
• .only(): It is used to run a specified suite or test exclusively, ignoring all other tests and suites. This can be useful when you’re debugging a specific test case or working on a specific suite of tests, and you want to focus on that specific test case or suite without running any others.
• .skip(): It is used to skip a specified suite or test, effectively ignoring it during test execution. This can be useful when you’re working on a test suite or test case that isn’t ready to be run yet, or when you want to temporarily disable a test without deleting it.

The post The Ultimate Guide To End-to-End Testing With Cypress appeared first on Automated Visual Testing | Applitools.

The website to test

Last year at Applitools, we replaced our old tutorial website with a new website that uses Docusaurus, a very popular documentation framework based on React. We also rewrote several of the guides for our most popular SDKs. Presently, the site has about 60 pages of varying length. Since many of the pages host similar content, we use components to avoid duplication in text and in code. However, that means any change could inadvertently break multiple pages.
Our tutorial site is currently hosted at https://applitools.com/tutorials/:

The tutorial site also has a sitemap file at https://applitools.com/tutorials/sitemap.xml:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
    <url>
        <loc>https://applitools.com/tutorials/</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>

    <url>
        <loc>https://applitools.com/tutorials/guides/advanced-tricks/troubleshooting-issues</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>
...

It would be very helpful to test this tutorial site with the semi-autonomous testing tool we just sketched out.

{
  "name": "auto-website-testing",
  "version": "1.0.0",
  "description": "A semi-autonomous testing project that visually tests all the pages in a website's sitemap",
  "main": "index.js",
  "scripts": {},
  "repository": {
     "type": "git",
     "url": "git+https://github.com/AutomationPanda/auto-website-testing.git"
   },
   "keywords": [],
   "author": "",
   "license": "ISC",
   "bugs": {
      "url": "https://github.com/AutomationPanda/auto-website-testing/issues"
   },
   "homepage": "https://github.com/AutomationPanda/auto-website-testing#readme",
   "devDependencies": {
     "@applitools/eyes-playwright": "^1.13.0",
     "@playwright/test": "^1.29.1",
     "ts-node": "^10.9.1"
   }
}

import { chromium } from '@playwright/test';
import { BatchInfo, Configuration, VisualGridRunner, BrowserType, Eyes, Target } from '@applitools/eyes-playwright';

(async () => {

     // Read environment variables
     const BASE_URL = process.env.BASE_URL;
     const SITE_NAME = process.env.SITE_NAME;
     const TEST_CONCURRENCY = Number(process.env.TEST_CONCURRENCY) || 1;

 // Validate environment variables
 if (!BASE_URL) {
    throw new Error('ERROR: BASE_URL environment variable is not defined');
 }
 if (!SITE_NAME) {
    throw new Error('ERROR: SITE_NAME environment variable is not defined');
 }

 // Parse the base and sitemap URLs
 const baseUrl = BASE_URL.replace(/\/+$/, '');
 const sitemapUrl = baseUrl + '/sitemap.xml';

 // Create Applitools objects

 let runner = new VisualGridRunner({ testConcurrency: TEST_CONCURRENCY });
 let batch = new BatchInfo({name: SITE_NAME});
 let config = new Configuration();
 let widthAndHeight = {width: 1600, height: 1200};
 let snapshotPromises: Promise<any>[] = [];

 // Set Applitools configuration
 config.setBatch(batch);
 config.addBrowser(1600, 1200, BrowserType.CHROME);
 // config.addBrowser(1600, 1200, BrowserType.FIREFOX);
 // config.addBrowser(1600, 1200, BrowserType.SAFARI);
 // config.addBrowser(1600, 1200, BrowserType.EDGE_CHROMIUM);

 // Set up a browser
 const browser = await chromium.launch();

 // Set up a sitemap context and page
 const sitemapContext = await browser.newContext({viewport: widthAndHeight});
 const sitemapPage = await sitemapContext.newPage();

 // Get the sitemap
 await sitemapPage.goto(sitemapUrl);
 const pageLinks = await sitemapPage.locator("loc").allTextContents();
 sitemapContext.close();

 // Capture a snapshot for each page
 for (const link of pageLinks) {
     snapshotPromises.push((async () => {

       // Open a new page
       const linkContext = await browser.newContext({viewport: widthAndHeight});
       const linkPage = await linkContext.newPage();

       // Open Eyes
       const eyes = new Eyes(runner, config);
       await eyes.open(
           linkPage,
           SITE_NAME,
           link.replace(baseUrl, ''),
           widthAndHeight
     );

       // Take the snapshot
       await linkPage.goto(link);
       await eyes.check(link, Target.window().fully());
       console.log(`Checked ${link}`);

       // Close Eyes
       await eyes.close(false);
       console.log(`Closed ${link}`);

    })());
 }

 // Close all Eyes
 console.log('Waiting for all snapshots to complete...');
 await Promise.all(snapshotPromises);

 // Close the browser
 await browser.close();
 console.log('Complete!');

})();

Running the tests

Let’s run the script to test it out. The website I’m going to use in this example is the Applitools tutorial site, which has about 60 pages. You can use any site you want as long as you set your environment variables. If you want to run this, you will need an Applitools account, which you can register for free with your email or GitHub account. Once you register an account, you’ll take your Applitools API key and set that as an environment variable.

We need to run our tests in the terminal. If this is your first time, you’ll need to run npm install to install the npm packages as well as npx playwright install to install the Playwright browsers. Once you’ve run those installers and set your environment variables, you just need to run npx ts-node autonomous.ts to run the script.

The script fetches the sitemap file, parsing out all of those links, and it’s firing off promises to start capturing visual snapshots for each one that’s happening asynchronously. The messages saying “Checked” link means those have now been initiated. What we’ll start to see is as they complete one by one in the Applitools Ultrafast Grid, we’ll see how the session becomes closed. Closed images code, JavaScript, closed mobile browser, that means one by one, the visual testing has been completed.

While the test is running, we can see results in the Applitools Eyes dashboard as a batch.

If I look at all the tests they’re popping in, we can see some of them are still running while others have passed.

If I want to see what the visual comparisons look like, I can compare them side by side. If there are no visual differences, the test for this page will pass.

You can see on the page it captures everything, the title bar, the sidebar table of contents, main body and footer. It will even scroll all the way to the bottom to make sure it gets the full page worth of contents.

No matter how long the page is, everything is being compared visually. It’s pretty cool to see just how many tests our autonomous testing solution can uncover. As I said before, the tutorial site right now has about 60 pages, which means that’s 60 tests that I didn’t have to explicitly write. Playwright plus Applitools Eyes took care of it all for me.

The post Implementing Autonomous Testing Today appeared first on Automated Visual Testing | Applitools.

Automating different types of tests

To illustrate our different types of test automation, we’ll be using the example project Cypress Heroes. In this full-stack TypeScript app, users can take the following actions:

• Log in with an email and password
• Like heroes, which increments the hero’s number of fans
• Hire heroes, which increments the hero’s number of saves
• Manage hero profile information like name, superpowers, and price

ICYMI: Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see Ely Lucas from Cypress demo the example project.

End-to-end testing

Cypress is traditionally known for end-to-end testing. You automate user interactions for specific scenarios from start to finish in the browser, and then run functional assertions to check the state of elements at each step. End-to-end tests are hidden in an actual web server and hit the site just like a user would.

Measurable stats for code coverage of your end-to-end testing can act as a health metric for your website or app. Adding coverage reports to your automation pipeline as commits can help ensure you’re testing all parts of your code.

Component testing

If you’re using a component-based framework like React or Angular or a design system like Storybook, you can also do component testing to test UI components. In this example, we have a button component with a few tests that pass, the hero card test, and a test for the login form. These components are being mounted in isolation outside of your typical web server.

Think of component tests as “UI unit” tests. While they don’t give end-to-end coverage, they’re quick and easy to run.

API testing

For your back end, you’ll need to automate API tests. The example project is using a community-built plugin called cypress-plugin-api. This plugin provides an interface inside the Cypress app to test APIs. It’s really cool and it’s super fun, and it allows you to write tests that you would have to do manually in a tool like Postman.

Fun fact: Cypress Ambassador Filip Hric developed the cypress-plugin-api. Check out Filip’s Test Automation University courses.

The API tests in our example are in the separate server project. We can use the command npx cypress open, and then we can run those tests in Chrome. We can see all of our results that we’re getting the response of the status codes. We can view a post request, the headers that were sent, the headers that were returned, and other stuff that you normally get from a tool like Postman.

And it’s just baked into the app, which is really nice. Cypress is basically a web app that tests a web app. And so, you could extend Cypress as an app with things like this to help you do your testing and to have it all seamlessly integrated.

Running a pipeline with GitHub Actions

The example project uses GitHub Actions to set up the test automation pipeline. When working on smaller projects, it’s easy to have CI interactions baked into your repository, all in one place.

Configuring your GitHub Action

With GitHub Actions, you declare everything you need in a YAML file in the .github/workflows folder. Your actions become part of your repository and are covered by version control. If you make any changes, you can review them easily with a simple line-by-line diff. GitHub Actions make it easy to automate processes alongside other interactions you make with your repository. For example, if you open a pull request, you can have it automatically kick off your tests and do linting. You can even perform static code analysis before merging changes.

Some environment variables are set at the top of the YAML file. The API URL is what the client app uses to communicate to the API. The example app is hooked up to send test results to the Cypress Cloud. Those results can then be used for analytics, diagnostics, reporting, and optimizing our test workflows. The Cypress cloud also requires a GitHub token, so it can do things like correctly identify what pull request is being merged.

For those new to GitHub Actions: You can define environment variables per step in a job, but declaring them at the top helps you update them painlessly.

Running each of our tests

To keep things simple, there is only one job right now in this GitHub Action. First, it checks out the code straight from GitHub. Next, it builds the project using the Cypress GitHub Action. The Cypress GitHub Action does a few things for you like building your application or npm installing or yarn installing the dependencies.

Building first means that subsequent jobs don’t have to build the app again. We’ve set run test to false, which is a parameter to the Cypress GitHub Action, because we don’t want to run the tests here. We’ll be running the tests separately below.

We have our component tests in our GitHub Action. We tell it to install false, since we installed it up above. And then we run our custom test command, which opens Cypress in run mode and initiates component testing. This record tells the GitHub Action to send the results to Cypress Cloud.

And then we have to start the client and server. For both end-to-end tests and API tests, the application must be up and running component tests. For the end-to-end test and API tests, the example app is hitting live servers.

This run command will start both the React app and the Node server, and then it will run the end-to-end test. We’re telling it again to not install the dependency, since it was already installed. Then, we’re running the command to start the end-to-end testing. The wait command will wait to make sure that both the client URL and the API URL are both up and running before it will start the test. If the test starts before both URLs get up and running, you’ll have some tests fail.

Another thing that the Cypress GitHub Action does is that you have the option to wait for these services to be live before the testing starts. By default, the npm run test commands are going to use the Chromium browser built into Electron. If you want to test on other browsers, you must make sure those browsers are installed on the runner. Cypress provides Docker images that you can add to your configuration to download the different browsers. However, downloading additional browsers increases the file size and makes the runs take longer.

Make sure that the Cypress binary itself is downloaded and installed. It’s going to run headlessly. This is because the command set up in these scripts is run mode, which is headless, whereas open mode is with the UI.

And then it will run the API test, which is very similar to end-to-end tests, except that since we’re not hitting the actual client app – only hitting the API app – we’re only waiting to make sure that the API URL is up and running.

If you write test cases per the local database for end-to-end testing before pushing to GitHub Actions, someone else running those test cases on their system could potentially fail. In whatever kind of test automation you develop, you’ll need to handle test data properly to avoid collisions. There are many different strategies you can follow. For more information on this and solving sample data dependency, watch my talk Managing the Test Data Nightmare.

How long do the test suites take?

When running your tests with Cypress and GitHub Actions, the results are uploaded to Cypress Cloud. You can go into Cypress Cloud and actually watch replays of all these tests that happened. The entire pipeline run in the example was 3 minutes and 50 seconds for all three test suites.

The individual test suites we ran took the following times:

• Component tests: 49 seconds
• End-to-end tests: 1 minute and 18 seconds
• API tests: 13 seconds

Improving test coverage with visual assertions

Since all the Cypress tests are run inside of the browser window, you can visually see them and inspect to make sure that they’re looking correctly. But this type of review is a manual step. If someone accidentally makes a change to the stylesheet, the site could no longer be running properly, but if we run the tests, they’ll pass.

We can use Applitools Eyes to fix this issue.

Visual testing is meant to automate the things that traditional automation is not so good at. For example, as long as particular IDs on your page are in the DOM somewhere, your traditional automation scripts with something like Cypress are still going to find and interact with the elements. Applitools Eyes uses visual AI to look at an app and be able to detect these kinds of visual differences that traditional assertions struggle to capture. Let’s add some visual snapshots to these end-to-end tests.

Adding Applitools to your project

First, you’ll need an Applitools account. You can register a free Applitools account with your GitHub username or your email, and you’ll be good to go. You’ll need your Applitools API key for when we run tests with Applitools.

Next, we’ll need to install the Applitools SDK using npm install @applitools/eyes-cypress.

It can be a dev dependency or it can be a regular dependency – whichever you prefer. In the example project, we use a dev dependency. In the example project, we’re using the Applitools Eyes SDK for Cypress, we have Applitools SDKs for basically every tool framework you got.
Next, we’ll need to create an Applitools configuration file. Where in Cypress projects, you have your cypress.config.js file, basically we want one that’s called applitools.config.js.

Configuring your Applitools runner

In the Applitools config file, we will specify the configuration for running visual tests. There’s a separation between declaring configuration and actually adding test steps.
One of the settings we want is called batchName, and we’re going to set that to “cy heroes visual tests” to reflect the name of our demo app. The batch name will appear in the Eyes Test Manager (or the Applitools “dashboard”) after we run our visual tests.

Next, we’ll set the browsers. This will be a list, with each item being an entry that specifies a browser configuration, including name, width, and height.

Typically, since Cypress runs inside of an Electron app, it can be challenging to test mobile browsers. However, the Applitools Ultrafast Grid enables us to render our visual snapshots on mobile devices. The settings for mobile devices are going to be a bit different than those for browsers. Instead of having a name, we’re going to have a device name.

Our applitools.config.js file is complete. When we run our tests – either locally or in the GitHub Action – Applitools will render the snapshots it captures on these four browser configurations in the Ultrafast Grid and report results using the batch name. Furthermore, the local platform doesn’t matter. Even if you run this test on Windows, the Ultrafast Grid can still render snapshots on Safari and mobile emulators. A snapshot is just going to be a capture of that full page. Applitools will do the re-rendering with the appropriate size, the appropriate browser configuration, and all that will happen in the cloud. Essentially you can do multi-browser and multi-platform testing with simple declarations.

Now that we have completed the configuration, let’s update the tests to capture visual snapshots, starting with the homepage.

Setting up our test suites

You need to make sure that your tests aren’t interfering with other tests. In these tests, we’re going through and modifying some of the heroes that are in the application. The state of the application changes per test, so to get around that, we’re creating a new hero just for working with our tests and deleting the hero after the tests.

In the example, we’re using Cypress tasks, which is code that actually runs on the Node process part of Cypress. It’s directly communicating with our database to add the hero, delete the hero, and all the other types of setup tasks that we want to do before we actually run our test.

So it’s going to happen for each of the tests, and then we’re visiting the homepage and getting access to the hero.

We get our new hero and then we call cy.deleteHero, which is going to call the database to delete the hero. From the describe block at the start of every test, we get our hero. And then, finally, we have the hero card by its name, and we find the button that has the right selectors, so we can actually select it and click the button.

This test is making sure that you’re logged in before you can like this hero. We’re making sure that the modal popped up, clicking the okay on the modal, and then making sure that modal disappears and does not exist anymore.

Down below we have another suite for when a normal user is logged in. And so we’re using a custom Cypress command to log in with this username and password. You can define these custom commands that are like making your own function, encapsulating a little bit of logic so that it could be reusable.

So what we’re doing to test the login is going to the homepage, running the login process, and verifying the login was actually successful. The cy.session is caching a session for us to restore the session later from cookies. This helps speed up your test so you’re not having to go through the whole flow of actually logging in again.

We have another suite here for when an admin user is logged in, because an admin user can edit users and delete heroes.

 In the example, negative login tests – where you use the wrong username and/or password – are under the component tests.

In the login form component test, when an email and password is invalid, an error should show. The example uses cy.intercept to mock the API request that goes to the cert, which goes to the off endpoint and returns a status code 401, which represents an invalid login.

You can either write a component test or an end-to-end test. In this case, a component test makes it easier to set up the mock data.

Adding a call to Applitools Eyes

With the test suites set up, we’re ready to add some visual snapshots here. We need to call an Eyes session using the Applitools Eyes SDK. The idea is that we open our eyes, and we can take visual snapshots. And then at the end of the test, we will close our eyes to say that we’ve captured all the snapshots for that session or for that test. And at that point, Applitools Eyes will upload the snapshots that are captured to the Applitools Eyes server, do all of the re-rendering of the things of those four browsers in the Ultrafast Grid. Then we can log into the Applitools dashboard and we can see exactly what happened with our testing.
To get the autocomplete for Eyes commands, we need to set up the Applitools Eyes stuff with the Cypress project. We already did npm install on the package, so we’ll need to run npx eyes-setup.

We’ll want to use the command cy.eyesOpen in the homepage describe block under the beforeEach method. We want to pass an app name and test name for logging and reporting purposes. You might also put their Cypress eyes open code in the beforeEach of the test cases, so the call doesn’t need to be duplicated.

Then, in the afterEach block, you’ll call cy.eyesClose.

In this test, you must log in, make sure that the modal pops up, log in, and then click okay in the modal and make sure the modal disappears, so we’ll need a snapshot when the modal is up and one when the modal goes away. In this case, we’ll capture the whole window.

If we didn’t want to capture everything, we could actually capture a region, like a div or even an individual element. On a small scale, using the region option does not make a measurable difference in execution speed, but it gives you a way to tune the type of snapshot we want.

For capturing the next step, we can basically copy the whole call there and paste it, changing the tag to homepage with the modal dismissed.

These snapshots are very straightforward to write, and something that we could consider is that some of those other assertions you might arguably be able to remove. The visual snapshot is going to capture everything on that window, so if it’s there and visible, we’re going to capture it and track it over time.

You would still need to keep all of your interactions, but you can remove most of your assertions checking visible elements. However, there are certainly things where if you want to check a very specific numeric value, you still want to keep those assertions.

Running the updated tests

All we need to do to run this test is make sure that we have our Applitools API key from our account saved as an environment variable of the Cypress application.

Note: If you happen to steal someone’s API key, it doesn’t really help you. It just means they’ll see your results, and you won’t. API keys should be kept secret and safe.

Using the Applitools Eyes dashboard

So to see the visual testing results, we will need to view them in the Applitools Eyes dashboard.

You can view your test results in a grid to see the UI quickly, or you can view your results in a list to see your configurations quickly.

On the left, you’ve got the batch name that was set. Then on the main part of the body, you’ll see there are actually four tests. We only wrote one test, but each test is run once per browser configuration we specified, providing cross-browser and cross-platform testing without additional steps.

If we open up the snapshots, you can see the two snapshots that we captured. These results are new, because this is the first time we’ve run the test.

We’ve established the snapshot as a baseline image, meaning anything in the future will be checked against that.

That’s where that visual aspect of the testing comes in. Your Cypress results will essentially tell you if it was bare bones basic functional, and then Eyes will tell you what it actually looked like. You get richer results together.

Resolving test results in the dashboard

Let’s see what this looks like if we make that visual change.

In the main file, we’ve updated the stylesheet and run the test again. There is no need to do anything in the Applitools Eyes dashboard before re-running the test.

The new test batch is in an unresolved state because Eyes detected a visual difference. In theory, a visual difference could be good or bad. You could be making an intentional change. Visual AI is basically a change detector that makes it obvious to you, the human, to decide what is good or bad. Then anytime Applitools Eyes sees the same kind of passing or failing behavior in the future, it’ll remember.

It’s important to note that the unresolved test results won’t stop your test automation job or your automation flow. Test automation would complete normally. You as the human tester would review visual test results in the Eyes Test Manager (the “dashboard”) afterwards. The pipeline would not wait for you to manually mark visual test results.

Let’s open up one of those snapshots so we can see it full screen.

In the upper left, below the View menu in the ribbon, there’s a dropdown to show both so that you can see the baseline and test side by side.

In the example, we had removed the stylesheet, so we can see very clearly that it’s very different. It’s not always this obvious. In this case, pretty much the whole screen is different. But if it were like a single button that was missing or something shunted a little bit, it would show that a specific area was different. That’s the power of the visual AI check.

Whenever Applitools detects a visual change, you can mark it as “passing” with a thumbs-up. Then that snapshot automatically becomes the new baseline against which future checkpoints are compared. Applitools will go to the background and track similar images. And it will automatically update those appropriately as well.

Note: If you ever want to “reset” snapshots, you can also delete the baselines and run your tests “fresh” as if for the first time. The snapshots they capture will automatically become new baseline images.

Once we’ve resolved all test results, we’ll need to save. And now if we were to rerun our test again, Applitools Eyes would see the new snapshots and pass tests as appropriate. If you have dynamic content or test data, you add region annotations, which will ignore anything in the region box.
It is possible to compare your production and staging environments. You can use our GitHub Integration to manage different branches or versions of your application. We also support different baselines for A/B testing.

Closing thoughts

That’s basically how you would do visual testing with Applitools and Cypress. There are two big points to remember if you want to add visual testing to your own test suites:

• To get these tests running in your pipeline, the only change you’d have to make is to inject the Applitools API key in those environment variables.
• We didn’t really add a fourth suite of tests. Visual testing is more of a technique or an aspect of testing, not necessarily its own category of tests. All you have to do is work in the SDK, capture some snapshots, and you’re good to roll.

We hope this guide has helped you to build out your test automation pipeline to be more reliable and scalable. If you liked the guide, check out our Applitools tutorials for other guides on building your test automation pipeline. Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see the full walkthrough. To keep up-to-date with test automation, you can peruse our latest courses taught by industry-leading testing experts on Test Automation University. Happy testing!

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

The post Future-Proofing Your Test Automation Pipeline appeared first on Automated Visual Testing | Applitools.

The hype

Gartner publishes reports related to the Hype Cycle of AI. You can read the detailed reports from 2019 and 2021.

The image below is from the Hype Cycle for AI, 2021 report.

Based on the report from 2021, we seem to be some distance away from the technology being able to satisfactorily answer the questions in my mind. 

But there are areas where we seem to be getting out of what the Gartner report calls “Trough of Disillusionment” and moving towards the “Slope of Enlightenment”.

Based on my research in the area, I agree with this.

Let’s explore this further.

The promising future

Recently I came across a lot of interesting buzz created by ChatGPT, and I had to try it out. 
Given I am a curious tester by heart, I signed up for it on https://chat.openai.com/ and tried to see the kind of responses generated by ChatGPT for use cases related to test strategy and automation.

The role and value of AI in testing

I took ChatGPT for a spin to see how it could help in creating test strategy and generate code to automate some tests. In fact, I also tried creating content using ChatGPT. Look out for my next blog for the details of the experiment.

I am amazed to see how far ahead we have come in making the use of AI technology accessible to the end-users.

The answers to a lot of the non-coding questions, though generic, were pretty spot on in the responses. Like the code generated by record-and-playback tools cannot be used directly and needs to be tuned and optimized for regular usage, the answers provided by ChatGPT can be a great starting point for anyone. You would get a high-level structure, with major areas identified, which you can then detail out based on context, that only you would be aware of.

But, I often wonder what is the role of AI in Testing? Is it just a buzzword gone viral on social media?

I did some research in the area, but most of the references I found were blog posts and articles about “how AI can help make testing better”.

Here is a summary from my research on trends and hopes from AI & Testing.

Test script generation

This aspect was pretty amazing to me. The code generated was usable, though in isolation. Typically in our automation framework, we have a particular architecture we conform to while implementing the test scripts. The code generated for specific use cases will need to be optimized and refactored to fit in the framework architecture.

Test script generation using Applitools Visual AI

Applitools Visual AI takes a very interesting approach at this. Let me explain this in more detail.

There are 2 main aspects in a test script implementation.

1. Implementing the actions/navigations/interactions with the product-under-test
2. Implementing various assertions based on the above interactions to validate the functionality is working as expected

We typically only write the obvious and important assertions in the test. This itself can make the code quite complex and also affects the test execution speed. 

The less important assertions are typically ignored – for lack of time, or also because they may not be directly related to the intent of the test.

For the assertions you have not added in the test script, you either need to implement different tests for validate those, or worse yet, they are ignored.

There is another category of assertions which are important and you need to implement in your test, but these are very difficult or impossible to implement in your test scripts (based on your automation toolset). Example – validating the colors, fonts, page layouts, overlapping content, images, etc. is very difficult (if at all possible) and complex to implement. For these types of validations, you usually end up relying on a human testing these specific details manually. Given the error-prone nature of manual testing, a lot of these validations are often (unintentionally) missed or ignored and the incorrect behavior of your application gets released to Production.

This is where Applitools Visual AI comes to the rescue.

Instead of implementing a lot of assertions, with Applitools Visual AI integrated in your UI automation framework, it can take care of all your functional and visual (UI and UX) assertions with a single line of code. With this single line of code, you are able to check the full screen automatically, thus exponentially increasing your test coverage. 

Let us see an example of what this means.

Below is a simple “invalid-login” test with assertions.

If you use Applitools Visual AI, the same test changes as below:

NOTE: In the above example with Applitools Visual AI, the lines eyes.open and eyes.closeAsync are typically called from your before and after test method hooks in your automation framework.

You can see the vast reduction in code required for the same “invalid-login” test as compared between the regular assertion-based code we are used to writing, versus using Applitools Visual AI. Hence, the time to implement the test has improved significantly.

The true value is seen when the test runs between different builds of your product which has changes in functionality.

The standard assertion-based test will fail at the first error it encountered. Whereas the test with Applitools is able to highlight all the differences between the 2 builds, which include:

• Broken functionality – bugs
• Missing images
• Overlapping content
• Changed/new features in the new UI

All the above is done without having to rely on locators, which could have changed as well, and caused your test to fail for a different reason. It is all possible because of the 99.9999% accurate AI algorithms provided by Applitools for visual comparison. 

Thus, your test coverage has increased, and it is impossible for bugs to escape your attention.

The Applitools AI algorithms can be used in any combination as appropriate to the context of your application to get the maximum validation as possible.

Based on the data from the survey done on “The impact of Visual AI on Test Automation”, we can see below the advantages of using Applitools Visual AI.

Visual AI: The Empirical Evidence

5.8x faster

Visual AI allows tests to be authored 5.8x faster compared to the traditional code-base approach.

5.9x more efficient

Test code powered by Visual AI increases coverage via open-ended assertions and is thus 5.9x more efficient per line of code.

3.8x more stable

Reducing brittle locators and labels via Visual AI means reduced maintenance overhead.

45% more bugs caught

Open-ended assertions via Visual AI are 45% more effective at catching bugs.

Increasing test coverage

AI technology is getting pretty good in suggesting test scenarios and test cases. 

There are many products that are already using AI technology under the hoods to provide valuable features to the users.

Increasing test coverage using Applitools Ultrafast Test Cloud

A great example of an AI-based tool is the Applitools Test Cloud which allows you to scale the test execution seamlessly without having to run the tests on other browsers and devices.

In the image below, you can see how easy it is to scale your web-based test execution across the devices and browsers of your choice as part of the same UI test execution using the Applitools Ultrafast Grid.

Similarly you can use the Applitools Ultrafast Native Mobile Grid for scaling the test execution for your iOS and Android native applications as shown below:.

Example of using Applitools Ultrafast Native Mobile Grid for Android apps:

Example of using Applitools Ultrafast Native Mobile Grid for iOS apps:

Test data generation

This is an area that is still to get better. My experiments showed that while actual test data was not generated, it indicated all the important areas we need to think about from a test data perspective. This is a great starting point for any team, while ensuring no obvious area is missed out.

Debugging

We use code quality checks in our code-base to ensure the code is of high quality. While the code quality tool itself usually provides suggestions on how to fix the flagged issues, there are times when it may be tricky to fix the problem. This is an area where I got lot of help in fixing the problem.

Example: In one of my projects, I was getting a sonar error related to “XML transformers should be secured”.

​​I asked ChatGPT this question:

The solution to the problem was spot-on and I was immediately able to resolve my sonar error.

Code optimization and fixing

Given a block of code, I was able to use ChatGPT to suggest a better way to implement that code, without me providing any context of my tech stack. This was mind-blowing!

Here is an example of this:

There were 5 very valuable suggestions provided by ChatGPT. All of these suggestions are very actionable, hence very valuable!

Analysis and prediction

This is a part I feel tools are currently still limited. There is an aspect of product context, learning, and using the product and team-specific data to come up with very contextual analysis and eventually get to the predictive and recommendations stage.

Testing AI systems

The tester in me is also thinking about another aspect. How would I test AI systems? How can I contribute to building and helping the technology and tools move to the Slope of Enlightenment in Gartner’s Hype Cycle? I am very eager to learn and grow in this exciting time!

Challenges of testing AI systems

Given I do not have much experience in this (yet), there are various challenges that come to my  mind to test AI systems. But by now I am getting lazy to type, so I asked ChatGPT to list out the challenges of testing AI systems – and it did a better job at articulating my thoughts … almost felt like it read my mind.

This answer seems quite to the point, but also generic. In this case, I also do not have any specific points I can think of, given my lack of knowledge and experience in this field. So I am not disappointed.

Why generic? Because I have certain thoughts, ideas, and context in my mind. I am trying to build a story around that. ChatGPT does not have an insight into “my” thoughts and ideas. It did a great job giving responses based on the data it was trained on, keeping the complexity, adaptability, transparency, bias, and diversity in consideration.

For fun, I asked ChatGPT: “How to test ChatGPT?”

The answer was not bad at all. In fact, it gave a specific example of how to test it. I am going to give it a try. What about you?

The post AI: The Magical Helping Hand in Testing appeared first on Automated Visual Testing | Applitools.

The post Let the Engineers Speak! Part 5: Audience Q&A appeared first on Automated Visual Testing | Applitools.

The discussion

Andy Knight (moderator):

So I’m going to ask another popcorn question here. And this one again comes with an audience poll. So if you’re in the audience, hop over to the polls and take a look here. Have you ever considered switching your test tool or framework? And if so, why? If so, why not?

Carter Capocaccia (Cypress):

Yeah, we considered it and did it. You know, I think you have to acknowledge it at a certain point that maybe what you’re doing isn’t working or the tool doesn’t fit your application or your architecture. And it’s a really hard decision to make, right? Because typically, by the time you get to that point, you’ve invested tens, hundreds, if not thousands of hours into a framework or an architecture. And making that switch is not an easy task.

Andy Knight (moderator):

What did you switch from and to?

Carter Capocaccia (Cypress):

Yeah, so [Selenium] WebDriver to Cypress. And you can already kind of see that architecturally they’re very, very different. But yeah, it was not an easy task, not an easy undertaking, but I think we had to admit to ourselves that WebDriver did not fit our needs. It didn’t operate in the way we wanted it to operate. At least for us, it was set up in a way that did not match our development cycles, did not match our UI devs’ patterns and processes.

So when we talk about, you know, kind of making a team of test engineers have additional help, well, one way to do that is [to align] your test automation practices with your UI dev practices. And so if we can have a JavaScript test framework that runs in a web app, that acts like a web app, that feels like a web app, well, we’ve now just added an entire team of UI developers that we’ve kind of made them testers, but we haven’t told them that yet. That’s kind of how that works. You adopt these people by just aligning with their tooling and all of a sudden they realize like, hey, I can do this too.

Andy Knight (moderator):

Mm-hmm. So I want to ask another question. Since you’re saying you used Selenium WebDriver before, in what language did you automate your test? Was that in JavaScript, just like first test, or was that in Java or Python or something else?

Carter Capocaccia (Cypress):

Yep, so it’s JavaScript. But when we look [at the] architecture of the app, right? So the architecture of WebDriver is very different than Cypress. [With] Cypress, you write a JavaScript function just like you write any other function. And most of the time, it’s going to operate just like you expect it to. I think that’s where, instead of them having to learn a WebDriver framework and what the execution order of things was and how the configurations were, Cypress is just pretty apparent in how it functions. When you look at like, well, how is the Cypress test executed? It uses Mocha under the hood as the test runner. So that’s just functions that are just kind of inside of describe and context and it blocks. So I think anybody that doesn’t know anything about testing but is a JavaScript developer can hop right into Cypress, look at it, and say, yeah, I think I know what’s going on here. And probably within the first 5 to 10 minutes, write an assertion or write a selector.

Andy Knight (moderator):

Awesome. That’s really cool. And so was that something that you and your team or org did recently? Was this like a couple years ago that you made this transition?

Carter Capocaccia (Cypress):

You know, I think I’d be lying to you if I told you the transition ever ends, right? I think it starts and you get to a point where you’ve scaled and you’ve gotten to where you’ve written a ton of tests on it, and then that’s great. But of course, it’s kind of the bell curve of testing. Like it starts really, really slow because you’re moving off the old framework, then you start to scale really quickly. You peak, and then at that point, you have these kind of outliers on the other side of the bell that are just running legacy applications or they just don’t want to move off of it. Or for whatever reason, they can’t move off of it. Let’s say they’re running like Node 10 still. It’s like okay, well, maybe we don’t want to invest the time there to move them off of it.

Yeah. So it’s a long journey to do a task like this. So it’s never done. By the time we’re finished, you know, there’s gonna be a new testing tool or Cypress will have a new version or something else will be new in there. You know, pipelines will [be through] some AI network at this point. Who knows? But I’m sure we’ll have to start the next task.

Andy Knight (moderator):

Sure. So how about others in our group of panelist speakers here? Have y’all ever transitioned from one framework or tool to another? Or in your current project, did you establish it and you were just super happy with what you chose? And tell us the reasons why.

Tally Barak (Playwright):

Yes. So, yeah, we also switched from WebdriverIO as I said at the beginning. We switched to Playwright, obviously. It was not that painful, to be honest, maybe because we kept the Cucumber side. So we had [fewer] scenarios than what we have today. So it was less to move. I think it went over a period [of] a few months. Unlike you, Carter, we actually moved completely away from WebdriverIO. There is no WebdriverIO, everything is with Playwright. But again, probably a smaller organization, so we were able to do that. And are we happy? I mean, [we are] thankful every single day for this switch.

Gleb Bahmutov (Cypress):

I can comment. At Mercari, we did switch from WebDriver in Python and mabl. So both were because the maintenance of tests and debugging of failed tests was [a] very hard problem, and everyone agreed that there is a problem. The transition was, again, faster than we expected, just because the previous tests did not work very well. So at some point we were like, well, we have 50 failing daily tests, right? It’s not like we can keep them. It’s not a sunken cost. It’s a cost of not working. So we have to recreate the test. But it was not that bad. So for us, the cost of debugging and maintaining the previous tool just became overwhelming, and we decided, let’s just do something else.

Jose Morales (WebdriverIO):

In my experience, we were using Selenium with JavaScript as a previous framework, and then we moved to WebdriverIO with JavaScript and Applitools. Because in Domino’s, we have different kind of products. We have another product that is called Next Generation of the stores. We’re using a reactive application there. This is also a website application. And we’re evaluating other tools for automation such as UiPath. And one thing that I like about UiPath is [it’s] a low-coding solution. That means you don’t need to really be a very proficient developer in order to create the scenarios or automate with UiPath, because it only [requires] drags and drops [to] create the scenarios. And it’s very easy to automate. And in the top of that, with UiPath, it’s a very enterprise solution.

We have different kinds of components like an orchestrator, we have a test manager, we have integration with Jenkins. We have robots that execute our test cases. So it’s an enterprise solution that we implemented for that particular project. And we have the same results as WebdriverIO with Applitools. We’re happy with the two solutions – the two frameworks – and we’re considering to remove WebdriverIO and Applitools and use UiPath for the main website for Domino’s. So I think in the future, the low-coding solution for automation is a way to go in the future. So definitely UiPath is a good candidate for us for our automation in the website.

Andy Knight (moderator):

How about you, Steve? I know you’re right now on a Selenium-based test project. Have you and your team thought about either moving to something else, or do you feel pretty good about the whole Selenium-ness?

Steve Hernandez (Selenium):

I think clearly there appears to be major speed-ups that you get from the JavaScript-based solutions. And I know you’ve evangelized a little bit about Playwright and then I’m hearing good things from others. I mean, we have a mature solution. We have 98% pass rates. You know, some of the limitation is more what our application can handle when we blast it with tests. I don’t know if your developers after each local build are tests running locally using some of these JavaScripts solutions. But I think one of the things that is appealing is the speed. And with Boa Constrictor 3, the roadmap is to – well, it has been modularized. So you can bring your own Rusty API client or potentially swap in Selenium and swap in something like a module for Playwright. So we would get that speed up but keep the same API that we’re using. So we’re toying with it, but if we can pull that off in Boa Constrictor, that would take care of that for us.

Andy Knight (moderator):

Yeah, man, definitely. So, I mean, it sounds like a big trend from a lot of folks has been, you know, step away from things like Selenium WebDriver to [a] more modern framework. You know, because Selenium WebDriver is pretty much just a low-level browser automator, which is still great. It’s still fine. It’s still awesome. But I mean, one of the things I personally love about Playwright – as well as things like Cypress – is that it gives you that nice feature richness around it to not only help you run your tests, but also help develop your tests.

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

The post Let the Engineers Speak! Part 4: Changing Frameworks appeared first on Automated Visual Testing | Applitools.

The discussion

Andy Knight (moderator):

Alrighty. So I mean, this, our conversation is now kind of going in so many good directions. This is awesome. We keep talking about integrations and how our frameworks and our tools can connect with other things. So I guess my next question I would like to ask as a big popcorn question would be: what are some of your favorite integrations with your test projects? I’m talking about things like CI hooks, special reports that y’all have, maybe things for visual testing, accessibility testing.

And not only is this a question for the panel speakers, but also for the audience. We have another Slido word cloud, so head over to the Slido tab. We’d love to hear what your favorite integrations are for your test projects.

But I’d like to now direct this question back to our panel and ask them to pick one. Tell me why you love it so much. Tell me how y’all use it and what value y’all get out of it.

Carter Capocaccia (Cypress):

Okay, I’ll hop in here. I think for me with Cypress, it’s just under the hood, it’s just JavaScript, right? So most anywhere is able to hook into that immediately. So when we talk about integrations, let’s just use GitHub for example here. If you wanted a GitHub Actions pipeline, you can just go ahead and start doing that immediately. And Cypress out of the box will provide you a Docker image to go ahead and use right inside of your GitHub Actions pipeline.

So for me, I guess I’m not going to focus on any one specific tool for integration here. I think the way that Cypress kind of makes itself agnostic to all external tools is what makes it really, really nice. You know, it doesn’t say, hey, you’ve gotta use us with this particular, like an Azure pipeline or any kind of other DevOps tool. It says, hey, under the hood, all we are is a JavaScript framework. If you want to execute our JavaScript inside of some kind of Docker image, create your own Docker image. Gleb knows all about this. He publishes a new Docker image like every day it seems.

And we have, you know, the ability to take all that, take the results, and the base of what comes out of Cypress is just JSON. We’ll shoot that over to an open-source dashboard. So there’s Report Portal out there if you wanted to take those results, and then put them into a portal. I see on here [the poll results] Allure reports. Well, okay, so you take it, you just put it into any kind of integration there.
So I think for me personally – I actually wrote a whole blog post about this where I was able to take Vercel, GitHub Actions, and Cypress and set up a CI/CD pipeline completely for free with automated deployments, preview environments, automated testing, the whole nine yards. And so it made it to where for my personal little website, all I’ve got to do is submit a pull request. It kicks off all my tests. As soon as it’s done, it merges it, and then it’s immediately deployed with Vercel. So that was a really cool thing. So if I have to name a tool, I’m gonna say Vercel and GitHub Actions.

Gleb Bahmutov (Cypress):

Can I second that? GitHub Actions are amazing. Honestly, I went through the whole, you know, self-running Jenkins to TeamCity, Buildkite, Circle CI. GitHub Actions are the best. It’s amazing how much you can do with just JavaScript on UCI. And I was the one writing all integrations for the Cypress team at the time.

I want to bring one more integration. I’ve written it. It’s [an] integration between Cypress and LaunchDarkly feature flags environment, because during the test, sometimes you want to test an experiment behind a feature flag. And for us now, it’s an open-source plugin. It’s like this, here’s my user, here are the experiment values I want to enable. Let’s see this functionality, and it’s working.

Tally Barak (Playwright):

[My] answer is GitHub Actions as well. I don’t have that much experience derived with different tools. I used to work with Travis and a bit with CircleCI. But GitHub Actions is just answering everything we need – and Playwright obviously – because it just requires a node for the JavaScript version. You just install it with Docker, and you are good to go and do whatever you want with the outputs and so on. So yeah, GitHub actions and I’m actually interested in the other one that you mentioned about the feature flag. That’s worth looking into.

And of course, Cucumber. I also have a repository that is a starter report for working with Cucumber and Playwright. I’ll post it on the channel if anyone is interested to try it out. If you want to stick with the Gherkin and the BDD-style test. Otherwise, go with the Playwright test runner.

Andy Knight (moderator):

Tally, I want to plus one of both things you said there. First of all, everyone knows that I love BDD, I love Cucumber, I love Gherkin, so yes. But also, I want to go back to what everyone is harping about GitHub Actions. Y’all in the audience, if you haven’t tried GitHub Actions or used to yet, definitely check it out. It is very straightforward, low-to-no cost CI baked into your repo. I mean, like anytime I’ve done open-source projects or example projects, just putting a GitHub Actions to automatically run your tests from a trigger, oh, it’s beautiful. It can be a little intimidating at first because you’ve got that YAML structure you’ve got to learn. But I have found it very easy to pick up. I feel like personally – and speakers back me up on this – I never sat down and read a manual on how to write a GitHub Action. I just kind of looked at GitHub Actions and copied them and tweaked them, and it worked.

Tally Barak:

And then you went to the marketplace and looked for some.

Andy Knight (moderator):

Yes, yes, yes. You know, and so you can find things that are out there already. And I’ve done it not just with Cypress and Playwright, but literally like every single test framework out there that I’ve touched. You know, if it’s like a Selenium thing, if it’s a Playwright thing, a Cypress thing. Any kind of test unit integration, end-to-end, you name it. Like, it’s awesome. But anyway, I just wanted a plus one to both of those and now I’ll yield to the rest of the speakers’ favored integrations and why.

Steve Hernandez (Selenium):

Can I give a plug for LivingDoc? We use a lot of the SpecFlow stuff in our stack, as you saw, like the runner. There’s a great SpecFlow+ Runner report that gets generated and can show how each of the threads are performing in your test suite so you can look at performance. But one of the things just really simply is the Azure Log Analytics ingestion API. So right in your test pipeline, you can just throw up a PowerShell script or some console application and send that data that you would have in your CI tool over into something like log analytics. I suppose there’s an equivalent for Amazon. But then you can correlate, you know, your telemetry from your application along with test data and start to see where performance issues are introduced.

But I guess coming back to my plug for LivingDoc is, I feel in a lot of cases our automated tests are actually the only documentation for some parts of the application as far as how they’re supposed to work because you’ve captured the requirements in a test. So they’re kind of like living documentation and LivingDoc literally is that. And it’s something that you can share with the business and the developers to see what you have for coverage and it’s human readable. I think that’s really important.

Carter Capocaccia (Cypress):

Well, now with GPT3, everything is human readable.

Andy Knight (moderator):

So I wanted to ask about visual testing and specifically with Applitools. I saw and heard that some of y’all had used visual testing with Applitools in your projects. In line with talking about integrations that you love, how do y’all use Applitools if you use it, and what value do you find you get out of it? Go ahead.

Carter Capocaccia (Cypress):

Yes, I want to bring up kind of why Applitools is a different breed of product when it comes to visual testing, though. So you really have two flavors of visual testing. One is a bitmap comparison where you take a screenshot of an image and it takes it and basically breaks it down into computer readable code and it compares a known standard against the new version. If there’s a certain percentage with diversions between the old version and the new version, it flags it.You can kind of already tell, well, if all I know about is just basically one to zeros and comparing the results here, it’s not really all that intelligent. Because I could have a piece of the page, let’s say it’s a headline. The headline updates every five minutes. Well, I don’t have time to just continually update this baseline screenshot.

Well, what a tool like Applitools does is it actually takes your screenshots and puts them through computer vision. And with computer vision you can now provide a lot more instruction to what do you want to compare? What’s important to you? Do you want to ignore certain parts of the page? When I compare X to Y, do I really consider, let’s say, a carousel where the images are being delivered by a CMS and they’re being changed on a daily basis? Do I really care that yesterday it was a picture of let’s say somebody’s front porch and today it’s a picture of a tree? So there’s a lot more intelligence that’s built into a tool like Applitools than there is with standard bitmap comparison. I think it takes the right kind of customer for Applitools. And when they do have that right kind of customer, it’s a pretty huge game changer, considering that it allows you to make a lot more informed decisions about the health of the visual appearance of your application versus, like I said, a bitmap comparison.

Basically, if you want to compare the two, one is analog, the other is digital. The analog is the map comparison. The digital way is Applitools. If we’re being honest with ourselves, the Applitools way is probably the best and right way to do this. I just know that there are a lot of teams that are making smaller scale applications that can get away with assuming higher risk in using a tool like a bitmap comparison tool.

Jose Morales (WebdriverIO):

I would like to add that we have been using Applitools for six months or so, and it was a great addition to our framework. It was reducing a lot of code because we were coding a lot of assertions. So for example, if you go to the Domino’s website, you have the store locator, and then the store locator basically based on your zip code automatically populates the city region and the store closest to you. There is a lot of functionality there. And I remember that we were validating every single text fill, every single button, every single drop and menu for the content. In English and in Spanish and the different languages. So for us it was very helpful to use Applitools because with WebdriverIO, we can simulate the location from the customer and then move the store locator for that particular zip code and tell Applitools, hey, take a screenshot for this one.

And in one single shot, we are validating all the colors, buttons, text fields, content, and we were very careful to see what is worth it for this scenario, right? So in all the framework, what we are doing is okay, let’s test this particular functionality for this scenario. Let’s use WebdriverIO to move the customer until this particular checkpoint and then take a screenshot for that particular state in our application. And in one single shot, we can validate this in all the devices. In Chrome, Firefox, Edge. Pixel, iPhones. So with that particular action, and because we are reducing a lot of assertions, we are writing less code, it’s easy to maintain, it’s easy to extend, and our test cases are more accurate and smaller. And I think for us Applitools was a great addition and I fell in love with Applitools and currently we’re trying to incorporate the mobile validations, meaning iOS and Android as well.

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

The post Let the Engineers Speak! Part 3: Favorite Test Integrations appeared first on Automated Visual Testing | Applitools.

We’re excited to announce the latest release of Applitools Eyes, which comes with a number of new enhancements that our customers have been asking for. Applitools Eyes 10.15 is now available and can be accessed in the dashboard.

Support For Monorepos in Git

Applitools now supports Monorepos for all major git providers, allowing teams to add Visual AI to large, complex, single-tenant code repositories using tags and PR titles to separate teams and logic inside Applitools. A monorepo is a popular method for repository organization in teams looking for maximum speed and collaboration across their codebase, but it can introduce complexity when it comes to tools that work with the repo. Applitools now has the ability to granularly run and test sections of the repo as if they were separated.

Support For Multiple Git Repos In One Account

Continuing on our Git hot streak, Applitools now also supports integrating multiple GitHub organizations into a single Applitools team. For partners, agencies, or just large organizations that separate organizations you can now work on multiple projects with one Applitools account.

Enhanced Support For Dynamic Region Validation

When using coded regions based on an element identifier, Applitools Eyes 10.15 can now adjust the region automatically and make sure it covers the most up-to-date element dimensions. This ignores irrelevant diffs and saves more of your time!

New REST API Endpoints

The Applitools REST API has a few new endpoints that enable teams to interact with Applitools at scale. In Applitools Eyes 10.15 we’ve added the ability to validate API keys and edit batches programmatically.

To try out these great new enhancements in the latest release of Applitools Eyes 10.15, get started with Applitools today.

The post Introducing Monorepo Support, New APIs, and more with Applitools 10.15 appeared first on Automated Visual Testing | Applitools.

There are many record and playback tools available, such as Selenium IDE, Testim, Katalon, and others.

Just recently Google has decided to launch its own Recorder tool embedded directly into Chrome. When Google joins the game it’s interesting to see. We decided to check it out.

The new tool is called Chrome DevTools Recorder.

What Is Chrome DevTools Recorder?

Chrome’s new Recorder tool allows you to record and replay tests from the browser, export them as a JSON file (and more), as well as measure test performance. The DevTools Recorder was released in November 2021, and you can read all about it here.

Right off the bat, we were excited to see that the tool is straightforward and simple. Since it is embedded in the browser we have the convenience of not having to context switch or deal with an additional external tool.

Let’s see what Google has in store with us with the tool and check out just how easily and quickly we can run our first test.

We’ll do so by recording a test on the Coffee cart website and exporting it as a Puppeteer Replay script. To top it off, we will be sprinkling some Applitools magic onto it and see how easy it is to integrate visual testing into the new tool. Let’s go!

First things first, let’s open up our new tool and record a test. 

How to Record a Test with Chrome DevTools Recorder

• Open this page. We will be using this demo page for recording
• Open Chrome DevTools
• Click on More options() > More tools > Recorder

• Click on the Start new recording button to begin
• Enter your recording name
• Click the Start a new recording button at the bottom of the recording window
• The recording is started (the panel is showing Recording… indicates the recording is in progress)
• Try to click around and order some coffees
• Every interaction with the webpage is now recorded – clicking on buttons, switching web pages, waiting for elements to load and much more

Once the recording is done, we have our first automation script ready to run.

Given the recording, we can see some options before us:

1. Replay the recording – playback what was recorded.
2. Measure performance – replays the tests and opens the new Performance insights panel or the Performance panel of the DevTools. This way we can analyze the performance of our test, the amount of time it took to load each resource.
3. Edit and add steps – we can manually edit our tests and add steps, even complex ones (e.g. wait until 9 images have been loaded and then continue the test). All without code.

Lastly, we have the option to export the test as a JSON file. This is a great feature as you can share the files with other users. 

You can also export it as a Puppeteer Replay script right away. It allows you to customize, extend and replay the tests with the Puppeteer Replay library, which makes the tool even more useful for more experienced users.

One of the main ‘weaknesses’ of Chrome’s Recorder tools is the very basic validation and a pretty standardized flow, with no option in the UI to add on top of it. 

The ability to quickly record a stable automated test and export it to make it more customizable is an incredible feature. It can help create tests quickly and efficiently.

Downloading the Puppeteer Script

• Click on the Export button in the Recorder panel
• Click on the Export as a @puppeter/replay script and save it as main.mjs (we will customize this file to add in Applitools visual testing)
• Also click on the Export as a JSON file as shown below (we will customize our script later to read from this JSON file)
  
  

Understanding the Puppeteer Replay script

Open the main.mjs file we exported just now. This is what the script looks like:

import url from 'url';
import { createRunner } from '@puppeteer/replay';

export const flow = {
    "title": "order-a-coffee",
    "steps": [
        {
            "type": "setViewport",
            "width": 380,
            "height": 604,
            "deviceScaleFactor": 1,
            "isMobile": false,
            "hasTouch": false,
            "isLandscape": false
        },
        ...
    ]
};

export async function run(extension) {
  const runner = await createRunner(flow, extension);
  await runner.run();
}

if (process && import.meta.url === url.pathToFileURL(process.argv[1]).href) {
  await run();
}

After we npm install all the dependencies, we can replay the script above with the node main.mjs command.

The Puppeteer Replay library provides us with an API to replay and stringify recordings created using the Chrome DevTools Recorder. 

The flow variable is our recorded test steps. It is a JSON object. You can replace the flow value to read from a JSON file instead. Here is an example:

/* main.mjs */
import url from 'url';
import fs from 'fs';
import { createRunner, parse } from '@puppeteer/replay';

// Puppeteer: read the JSON user flow
const recordingText = fs.readFileSync('./your-exported-file.json', 'utf8');
export const flow = parse(JSON.parse(recordingText));


export async function run(extension) {
  ...
}

...

Run the script again. It returns the same result.

Extend the Puppeteer Replay script

The Puppeteer Replay offers a way to customize how a recording is run using the PuppeteerRunnerExtension class, which introduces very powerful and simple hooks such as beforeEachStep and afterAllSteps. 

Puppeteer must be installed to customize the tests further. For example, the tests will launch in headless mode by default. In order for us to see how the browser runs the automated test we can turn it off.

Below you can see an example on extending this class and running in headful mode:

/* main.mjs */
...
import puppeteer from 'puppeteer';
import { PuppeteerRunnerExtension } from "@puppeteer/replay";

// Extend runner to log message in the Console
class Extension extends PuppeteerRunnerExtension {
  async beforeAllSteps(flow) {
    await super.beforeAllSteps(flow);
    console.log("starting");
  }

  async afterEachStep(step, flow) {
    await super.afterEachStep(step, flow);
    console.log("after", step);
  }
}

// Puppeteer: launch browser
const browser = await puppeteer.launch({ headless: false });
const page = await browser.newPage();

// Puppeteer: read the JSON user flow
..

// Puppeteer: Replay the script
if (process && import.meta.url === url.pathToFileURL(process.argv[1]).href) {
  // add extension
  await run(new Extension(browser, page));
}

// Puppeteer: clean up
await browser.close();

Now that we understand the code it’s time to kick it up a notch by adding Applitools Eyes to the mix to enable visual testing.

How to add Applitools Eyes to Chrome DevTools Recorder

Applitools Eyes is powered by Visual AI, the only AI-powered computer vision that replicates the human eyes and brain to quickly spot functional and visual regressions. Tests infused with Visual AI are created 5.8x faster, run 3.8x more stably, and detect 45% more bugs vs traditional functional testing.

Applitools also offers the Ultrafast Grid, which provides massively parallel test automation across all browsers, devices, and viewports. With the Ultrafast Grid, you run your test setup script once on your local machine, then the Applitools code takes a snapshot of the page HTML & CSS, and sends it to the grid for processing. This provides an out-of-the-box solution for cross-browser tests, so you don’t have to set up and maintain an in-house QA lab with multiple machines and devices.

Incorporating Applitools Eyes into Chrome DevTools Recorder only takes a few steps. Here’s an overview of the process, with the full details about each step below.

1. Install Applitools Puppeteer SDK
   using npm: npm i -D @applitools/eyes-puppeteer
2. Add the eyes-puppeteer dependency:
   const {Eyes, Target, VisualGridRunner, BrowserType, DeviceName} = require('@applitools/eyes-puppeteer')
   Eyes – the Eyes instance
   Target – Eyes Fluent API
   VisualGridRunner – using the Ultrafast Grid (UFG) with Eyes
   BrowserType – UFG browsers configuration
   DeviceName – UFG devices configuration
3. Initialize Eyes and set the desired configuration and browsers for the UFG
4. We must first open Eyes in order to perform visual validations – the browser must be defined for this step
5. Perform visual validation with Eyes in the afterEachStep hook
6. Finally, after closing the browser we can close Eyes and gather the test results
7. to run the test, run the command:
   node <path_to_test.js>
   

Step 1 – Install Applitools Puppeteer SDK

As indicated above, to install the Applitools Puppeteer SDK run the following command:

npm i -D @applitools/eyes-puppeteer

Step 2 – Dependencies 

/* main.mjs */
import { Eyes, Target, VisualGridRunner } from '@applitools/eyes-puppeteer';

/* applitools.config.mjs */
import { BrowserType, DeviceName } from '@applitools/eyes-puppeteer';

Step 3 – Instance and Configuration

We define an Eyes instance alongside a Visual Grid runner, which is used with Applitools Ultrafast Grid. We can use the runner at the end of the test to gather all the test results across all Eyes instances in the test, therefore, the runner is defined globally. Eyes is usually defined globally as well but may also be defined locally for a specific test case. The terminology for a test in Applitools is equivalent to opening Eyes, performing any number of visual validations, and closing Eyes when we’re done. This will define a batch in Applitools that will hold our test, meaning we can have multiple tests in a single batch.

/* main.mjs */

// Puppeteer: launch browser
...

// Applitools: launch visual grid runner & eyes
const visualGridRunner = new VisualGridRunner({ testConcurrency: 5 });
const eyes = new Eyes(visualGridRunner);

We then create a function, setupEyes, that will set our configuration to Eyes before starting the test and before opening Eyes.  

/* applitools.config.mjs */

import { BrowserType, DeviceName } from '@applitools/eyes-puppeteer';

export async function setupEyes(eyes, batchName, apiKey) {
  eyes.setApiKey(apiKey);

  const configuration = eyes.getConfiguration();

  configuration.setBatch({ name: batchName })
  configuration.setStitchMode('CSS');

  // Add browser
  configuration.addBrowser({ width: 1200, height: 800, name: BrowserType.CHROME });
  configuration.addBrowser({ width: 1200, height: 800, name: BrowserType.FIREFOX });
  configuration.addBrowser({ width: 1200, height: 800, name: BrowserType.SAFARI });
  configuration.addBrowser({ width: 1200, height: 800, name: BrowserType.EDGE_CHROMIUM });
  configuration.addBrowser({ width: 1200, height: 800, name: BrowserType.IE_11 });
  configuration.addBrowser({ deviceName: DeviceName.Pixel_2 });
  configuration.addBrowser({ deviceName: DeviceName.iPhone_X });

  eyes.setConfiguration(configuration);
};

Step 4 – Opening Eyes

In this step we open Eyes right after initializing the browser and defining the page. The page is required in order to communicate and interact with the browser.

/* main.mjs */

// Applitools: launch visual grid runner & eyes
...
const apiKey = process.env.APPLITOOLS_API_KEY || 'REPLACE_YOUR_APPLITOOLS_API_KEY';
const name = 'Chrome Recorder Demo';

await setupEyes(eyes, name, apiKey);
await eyes.open(page, {
  appName: 'Order a coffee',
  testName: 'My First Applitools Chrome Recorder test!',
  visualGridOptions: { ieV2: true }
});

This is a good opportunity to explain what a Baseline is – A Baseline is a set of images that represent the expected result of a specific test that runs on a specific application in a specific environment. A baseline is created the first time you run a test in a specific environment. This baseline will then be updated whenever you make changes to any of the pages in your app, and accept these changes in Applitools Eyes Test Manager. Any future run of the same test on the same environment will be compared against the baseline.

By default, creating a test on a specific browser for the first time (e.g. Firefox) will create a new Baseline, thus running the same test on a different browser (e.g. Chrome) will form a new baseline.

The baseline is a unique combination of the following parameters:

• OS
• Viewport Size
• Browser
• Application name
• Test name

This means that by default a new baseline will be created for every combination that was not used before.

Step 5 – Visual Validation

By calling eyes.check(), we are telling Eyes to perform a visual validation. Using the Fluent API we can specify which target we would like to capture. Here we are performing visual validation in an afterEachStep hook to validate each step of the replay along the way. The target is specified to capture the window (the viewport) without the fully flag, which will force a full-page screenshot.

/* main.mjs */
...
// Extend runner to take screenshot after each step
class Extension extends PuppeteerRunnerExtension {
  async afterEachStep(step, flow) {
    await super.afterEachStep(step, flow);
    await eyes.check(`recording step: ${step.type}`, Target.window().fully(false));
    console.log(`after step: ${step.type}`);
  }
}

Step 6 – Close Eyes and Gather Results

We must close Eyes at the end of our test, as not closing Eyes will result in an Applitools test running in an endless loop. This is due the fact that when Eyes are open, you may perform any amount of visual validations you desire.

By using the eyes.abortAsync functionality, we essentially tell Eyes to abort the test in case that Eyes were not properly closed for some reason.

/* main.mjs */
...

// Puppeteer: clean up
await browser.close();

// Applitools: clean up
await eyes.closeAsync();
await eyes.abortAsync(); // abort if Eyes were not properly closed

Finally, after Eyes and the browser are closed, we may gather the test results using the runner.

/* main.mjs */
...

// Manage tests across multiple Eyes instances
const testResultsSummary = await visualGridRunner.getAllTestResults()
for (const testResultContainer of testResultsSummary) {
  const testResults = testResultContainer.getTestResults();
  console.log(testResults);
}

You can find the full code in this GitHub repository.

Viewing Test Results in the Applitools Dashboard

After running the test, you’ll see the results populate in the Applitools dashboard. In this case, our baseline and our checkpoint have no visual differences, so everything passed.

Last but not Least – Export Cypress Tests from Google Chrome DevTools Recorder

As we have already mentioned, the ability to quickly record a stable automated test and export it to make it more customizable is an incredible feature. For advanced users, you may also customize how a recording is stringified by extending the PuppeteerStringifyExtension class.

For example, I’d like to introduce you to the Cypress Chrome Recorder library, where you can convert the JSON file into a Cypress test script with one simple command. The library is built on top of Puppeteer Replay’s stringified feature.

We can convert our JSON recording file to a Cypress test with the following CLI command:

npm install -g @cypress/chrome-recorder

npx @cypress/chrome-recorder <relative path to target test file>

The output will be written to the cypress/integration folder. If you do not have that folder, you can get it by installing Cypress with the npm install -D cypress in your project.

Once the test file is ready, we can simply run the test as we would run a standard Cypress test.

Conclusion

Although record and playback testing tools have their setbacks and challenges, this looks like a very simple and useful tool from Google. It can be a good solution for creating simple scenarios or quick tests, seeing how easy it is to use. 

What we loved most about the tool was its simplicity. Plain record and playback at the moment with no advanced features, it’s a great stepstone for beginners in testing or even non-code individuals. 

Like with any Record-Playback tool one of the challenges is validation. Combined with the ease and speed of adding and running Applitools Eyes you can start validating your UI in no time. Find all the visual regressions and make sure your application is visually perfect.

Applitools Eyes has many advanced features, including AI-powered auto-maintenance, which analyzes differences across all your tests and shows only distinct differences, allowing you to approve or reject changes that automatically apply across all similar changes in your test suite. Learn more about the Applitools platform and sign up for your own free account today.

The post Creating Your First Test With Google Chrome DevTools Recorder appeared first on Automated Visual Testing | Applitools.

Applitools is proud to announce its positioning in the Major Players category in the IDC MarketScape: Worldwide Cloud Testing 2022 Vendor Assessment — Empowering Business Velocity. Applitools was also positioned in the Major Players category in the IDC MarketScape: Worldwide Mobile Testing and Digital Quality 2022 Vendor Assessment — Enabling Multimodal Dynamism for Digital Innovation.

Written by Melinda-Carol Ballou, Research Director, Agile ALM, Quality & Portfolio Strategies at IDC Research, this IDC study uses the IDC MarketScape model to provide an assessment of 24 vendors for worldwide cloud testing and enterprise automated software quality (ASQ) SaaS solutions.

“Participating vendors needed to have sufficient cloud testing automated software quality capabilities available in key areas of concern (e.g., test infrastructure provisioning and configuration management; deep analytics for analysis of performance optimization, service virtualization, and architectural and other analysis to enable visibility into the health of applications deployed in native and hybrid cloud; readiness for software targeting the cloud; and/or delivery of their ASQ software solution in the cloud with partner integration for other capabilities) for IDC clients.”

Source: IDC

Applitools products considered as part of these vendor assessments include Applitools Eyes and Applitools Ultrafast Test Cloud (the combination of Applitools Eyes, Applitools Ultrafast Grid and Applitools Native Mobile Grid).

How Applitools Enables Automated Software Quality

As software development teams rapidly deliver new products and services to market through more frequent and shorter release cycles, they struggle to fully test the customer experience due to increasing application complexity and an explosion of device/browser combinations. When development teams are confident they can fix functional and visual bugs faster, they can push more high-quality code faster than ever before.

To enable automated software quality consistently, quickly, and at a fraction of the cost, Applitools extends to its customers the power of Applitools Visual AI—the only AI-powered computer vision that accurately mimics the human eye and brain to avoid undetected functional and visual bugs, minimizing false positive bug alerts.

Visual AI is trained on more than 1 billion images and supports analysis across custom regions, with advanced comparison modes/match levels and auto-maintenance of test results. Tests infused with Visual AI (via Applitools Eyes) are created 5.8x faster, run 3.8x more stable, and catch 45% more bugs vs. traditional functional testing. In addition, tests powered by Visual AI can take advantage of the ultrafast speed and stability of Applitools Ultrafast Test Cloud.

The Ultrafast Test Cloud can instantly validate entire application pages and detect issues on even the most complex and dynamic pages. It allows users to write and execute tests once locally with support for more than 50 test frameworks and programming languages including: Cypress, Storybook, Selenium Java, Selenium JavaScript, Selenium C#, Selenium Python, Selenium Ruby, Selenium IDE, Webdriver.IO, TestCafe, and more.

A single functional test run captures the DOM & CSS rules for every browser state, automatically rendering it in parallel across all browsers (Chrome, Firefox, Safari, Edge, and Internet Explorer) and viewports using the Ultrafast Grid. Screenshots are then instantly analyzed by Applitools Eyes to find functional and visual bugs. 

Applitools integrates with all the major CI/CD platforms, including GitHub, GitLab, Bitbucket, Jenkins, Azure DevOps, Travis CI, Circle CI, Semaphore, TeamCity, and Bamboo, as well as with defect- tracking/collaboration systems, including Jira, CA Rally, Microsoft Teams, and Slack. The Applitools Eyes dashboard also enables collaboration between design, product, development, testing, and DevOps teams.

Automated software testing powered by Visual AI can help developers test 18 times faster across the full test cycle including writing, running, analyzing, reporting, and maintaining tests. That’s because Visual AI-powered automated tests that leverage the Ultrafast Test Cloud can run 30-50x faster than traditional solutions, with 99.9999% accuracy.* 

“References contacted by IDC found that Applitools’ solution led to greater efficiency in testing operations. They reported cost savings of up to 95% by making the switch to Applitools in combination with adoption of containers — additional testing tools were not required. Another customer reported that it was able to increase the speed of testing to reduce approval time for moving websites to production from 29 days to 1.5 hours. This is especially significant for a company maintaining 2,400 websites — Applitools helped ensure that sites maintained the same digital experience look and feel across pages as they changed dynamically, supporting up to five browsers and four viewpoints”.

Source: IDC

Applitools Customers, Use Cases and Community 

Applitools is helping more than 400 of the world’s top digital brands release, test, and monitor flawless mobile, web, and native apps in a fully automated way. We help our customers modernize critical test automation use cases — functional and visual regression testing, web and mobile UI/UX testing, cross browser / cross device testing, localization testing, PDF testing, digital accessibility and legal/compliance testing — to transform the way their businesses deliver innovation at the speed of DevOps without jeopardizing brand integrity.

Learn about the impact Applitools is having on these organizations by reading the customer stories in our case study library: https://applitools.com/case-studies/.  

To support the community, Applitools also maintains and manages Test Automation University, a free online learning community with over 100,000 members, that hosts more than 50 courses on a wide range of test automation topics and best practices.

“Being recognized as a major player in the IDC MarketScape for worldwide cloud testing underscores our ability to bring speed and stability to automated software quality with Visual AI and Ultrafast Test Cloud. We’re the world’s fastest-growing software testing company because our customers are using the most advanced, yet simple way to ensure brand integrity across any digital end-user experience. Their success is our success.”

Moshe Milman, COO and co-founder, Applitools

Thank You

It takes a village. Our fantastic team has worked long and hard to achieve this recognition from IDC Research, but this is also the direct result of the feedback and collaboration we’ve had with our customers – we could not have done it without you. Because of our strong community and our valued partnerships, the industry has also recognized Applitools as the:

• Best Testing Service/Tool in the DevOps Dozen² Awards for 2021
• Leading software vendor in testing in the 2021 North American Testing Awards
• Fastest growing software testing company in North America on the 2021 Deloitte Technology Fast 500 list 
• Category leader in automated testing on GetApp and Capterra

Thank you for trusting Applitools to deliver flawless automated testing for you, and we’re excited to head into the future of testing together.

Learn More

For more, schedule a demo and see for yourself how Visual AI is helping industry leaders deliver visually perfect digital experiences.

*Modern Cross Browser Testing Through Visual AI Report https://applitools.com/modern-cross-browser-testing-report/ 

The post Applitools Recognized as ‘Major Player’ in IDC MarketScape: Worldwide Cloud Testing 2022 Vendor Assessment appeared first on Automated Visual Testing | Applitools.

Modern applications typically revolve around the idea of smaller components, where you may have pieces like a button, navigation, or an interactive widget that all fit together like puzzle pieces to make up a larger app experience.

When trying to test these components, we’re usually stuck having to load the entire application just for that one piece or using synthetic rendering. This allows us to simulate how interacting with a component would work, but it’s not necessarily the same as the real thing.

Instead, we can use Cypress’ newly released Component Test Runner which looks to help make testing components more reliable and easier.

What is the Cypress Component Test Runner?

The Cypress Component Test Runner is a new solution from Cypress that allows you run your component tests in the browser just like someone visiting your app would actually use it.

Check out the announcement from Front End Test Fest!

These tests would sit next to your typical end-to-end tests, where the goal is to create tests that are focused on each individual component rather than the full application.

Because component tests don’t require page routing or loading the rest of the application, these tests can run much faster and with less overhead.

Out of the box, Cypress supports popular component-oriented libraries like React and Vue along with bundlers like Webpack.

Adding vision to component tests with Applitools

Cypress does a great job being able to leverage the browser so that our tests are using real-world environments, but any assertion that you write is still going to be based on the code.

This starts to become a problem when we’re only able to test how our application is functionally working and not necessarily how someone can actually use it.

That’s where Applitools steps in! With the Cypress Eyes SDK, we’re able to provide comprehensive visual testing for each of our components..

How Applitools works with the Cypress Component Test Runner

We can use the same Applitools Cypress Eyes SDK that you can use to add visual testing to your Cypress end-to-end tests to add visual testing to your Cypress component tests.

Because Cypress uses the same APIs and plugin architecture, once the Eyes SDK is installed, running visual testing is as simple as opening your Eyes and capturing the component.

Inside of a Cypress component test, you can use the mount function to load your component:

mount(<MyComponent />);

As soon as it’s mounted, to run Applitools Eyes, include:

cy.eyesOpen({ appName: 'My App', testName: 'MyComponent' });
cy.eyesCheckWindow();
cy.eyesClose();

And Eyes will capture a DOM snapshot of your rendered component and send it to the Applitools cloud where it will check for visual differences.

Learn more about Applitools and the Cypress Component Test Runner

If you want to get started by trying out a working example, head over to this GitHub repository showing how to use Applitools and Cypress with Create React App.

https://github.com/colbyfayock/my-component-tests

To check out the launch announcement, make sure to watch the opening keynote from Front-End Test Fest.

Otherwise, learn more about how to use Applitools with Cypress with the Eyes Cypress SDK.

Get Started Today

Want to try Applitools Eyes for yourself? You can get started today at the link below – the account is free forever.

Functional Testing’s New Friend: Applitools Execution Cloud

Articles — 09.11.2023

AI-Powered Test Automation: How GitHub Copilot and Applitools Can Help

Articles — 08.22.2023

Introducing Applitools Execution Cloud Self-Healing Test Infrastructure

Articles — 05.04.2023

Top 10 Visual Testing Tools

Articles — 03.03.2023

The post Adding Vision to Cypress Component Test Runner with Applitools Eyes appeared first on Automated Visual Testing | Applitools.

Component libraries and design systems are important development and design tools that allow teams to focus on building consistent and high quality experiences. Storybook is a tool that helps us with those experiences, but as your library and system grows, it becomes more difficult to maintain that level of high quality.

With the Applitools Eyes Storybook SDK, we’re able to provide visual testing coverage for all stories in your Storybook. The cool thing is this works for whatever UI framework you choose! Whether you’re using Storybook with React, Vue, Angular, Svelte, or even React Native, Applitools is able to support any framework Storybook supports.

Let’s dig in and find out what exactly Storybook is, where it helps, and how Applitools can easily provide coverage for any number of stories that a Storybook library supports.

What is Storybook?

Storybook is a JavaScript-based tool that helps teams build component libraries, design systems, and beyond.

It does this by providing a library-like interface, that shows different components and pages in isolation allowing developers, designers, testers, and users of the library to consume and work in each “story” in a focused environment.

Each Storybook story is composed of a component or set of components that represent one piece of what someone would interact with or visualize in an application.

Testing and validating UI at the component level

While most stories don’t show an entire page built with these components, each of them are fit together in various ways to build the actual interfaces that are used by visitors of that app.

This gives the development team or users of the library an easy and flexible way to make sure each component works as expected from the start, where they can then have confidence to use it however they’d like in the application.

But the tricky part is making sure we’re providing proper coverage of each of the components in a way that will help us build that confidence along with finding a scalable solution as the library grows.

Solutions exist to provide code-based testing, where we might validate the string-based output of a component or test the components virtually mounted, but they’re not actually capturing what the users of the components are actually seeing.

Visual testing helps us solve that. We can capture what the user is seeing right in the browser, and use that to make sure our components and stories are working exactly like they should, giving us that needed confidence.

AI-Powered Visual Testing and Storybook

With the Applitools Eyes Storybook SDK, we’re able to easily provide coverage for any number of stories that our Storybook library supports.

After installing @applitools/eyes-storybook, all you need to do is simply run the following command:

npx eyes-storybook

Once run, Applitools will find all stories and start to capture each story much like other SDKs available for Eyes.

Because Storybook is ultimately web-based, where it renders any UI framework it supports right in a browser, the Eyes SDK is able to capture a DOM snapshot of that rendered output.

That means, the Eyes Storybook SDK can support any framework that Storybook can support!

The SDK uses Puppeteer, similar to the Eyes Puppeteer SDK, where each story is loaded in a new Puppeteer tab.

It then collects a DOM snapshot which is used to render each page cross-browser in the Applitools cloud, where you’ll then get the same AI-powered Visual Testing coverage that you would with any other Applitools integration.

This helps so that we can visually identify the issues with our application, based on what people are actually seeing. Saving us time when trying to solve where the issue occurred or if it’s a false positive in the first place.

Getting started with Applitools Eyes and Storybook

If you want to get started with integrating Applitools Eyes with your Storybook library, we have a variety of resources including a walkthrough for adding Applitools Eyes to Storybook along with dedicated tutorials for React, React using CSF, Vue, and Angular.

You can also learn more about the importance of design systems with my free webinar Bringing Quality Design Systems to Life with Storybook & Applitools.

The post How to Maintain High Quality Design Systems with Storybook by Leveraging Visual AI appeared first on Automated Visual Testing | Applitools.