We have a lot of great webinars and virtual events here at Applitools. I’m hoping posts like this give you a high-level summary of the key points with plenty of room for you to form your own impressions.

Dave Piacente

Curious if the software robots are here to take our jobs? Or maybe you’re not a fan of the AI hype train? During a recent session, The Future of AI-Based Test Automation, CTO Adam Carmi discussed—in practical terms—the current and future state of AI-based test automation, why it matters, and what you can do today to level up your automation practice.

• He describes how AI can be used to overcome common everyday challenges in end-to-end test automation, how the need for skilled testers will only increase, and how AI-based tooling can help supercharge any automated testing practice.
• He also puts his money where his mouth is by demonstrating how the neverending maintenance overhead of tests can be mitigated using AI-driven tooling which already exists today using concrete examples (e.g., visual validation and self-healing locators).
• He also discusses the role that AI will play in the future, including the development of autonomous testing platforms. These platforms will be able to automatically explore applications, add validations, and fill gaps in test coverage. (Spoiler alert: Applitools is building one, and Adam shows a bit of a teaser for it using a real-time in-browser REPL to automate the browser which uses natural language similar to ChatGPT.)

You can watch the full recording and find the session materials here, and I’ve included a quick breakdown with timestamps for ease of reference.

• Challenges with automating end-to-end tests using traditional approaches (02:34-10:22)
• How AI can be used to overcome these challenges (10:23-44:56)
• The role of AI in the future of test automation (e.g., autonomous testing) (44:57-58:56)
• The role of testers in the future (58:57-1:01:47)
• Q&A session with the speaker (1:01:48-1:12:30)

Want to see more? Don’t miss Future of Testing: AI in Automation.

The post AI and The Future of Test Automation with Adam Carmi | A Dave-reloper’s Take appeared first on Automated Visual Testing | Applitools.

Today’s software applications are getting more complicated, thus every testing team needs to focus on expanding test coverage. To achieve this goal, it is important to use a combination of testing types, such as unit testing, integration testing, system testing, and end-to-end testing, depending on the software application’s complexity and requirements.

End-to-End (E2E) testing is designed to ensure that all components of a software application are working together correctly and that the system as a whole meets the desired functionality, performance, and reliability requirements.

Cypress is a popular open-source end-to-end testing framework for web applications. It is designed to make the testing process easier and more efficient for developers. One of the unique features of Cypress is that it runs the tests within the browser, which means that it can provide better control and visibility over the application under test.
In this blog on End to End testing, we will deep dive into performing Cypress End to End testing on a local Cypress grid and will explain how to start automating visual tests with Applitools Eyes and the Ultrafast Grid using Cypress in JavaScript.

What is End to End Testing?

End-to-end (E2E) testing is a software testing strategy that verifies an application’s complete flow from beginning to end. It is a type of functional testing that tests the application’s behavior as a complete system, rather than testing individual components in isolation.

E2E testing simulates a real user scenario and covers all aspects of the application, including user interfaces, APIs, databases, and other integrations. It typically involves testing multiple components of an application to ensure that they work together as expected and fulfill the requirements of the business or end-users.

E2E testing is typically performed after other types of testing, such as unit testing and integration testing, have been completed. It is used to validate that the entire system works together seamlessly and to identify any issues that may have been missed in earlier stages of testing.

Why is end-to-end testing necessary?

End-to-end testing (E2E testing) is a type of software testing that tests the entire system or application from start to finish, simulating real-world user scenarios.

Unit testing alone is not enough to ensure the quality and reliability of software. While unit testing is an important part of the testing process, it only verifies the behavior of individual components or modules of the software in isolation. It does not guarantee that the software will work correctly when integrated with other components or modules.

This is where integration testing enters into the picture. Integration testing focuses on testing the interaction between two or more components of a system, to ensure that they work together correctly. However, even if all the individual components pass integration testing, there may still be issues with the overall system when all the components are put together. This is where end-to-end testing comes in – it tests the entire system from start to finish.

Cypress is a popular automation testing framework that is designed specifically for end-to-end testing. It runs tests directly in the browser, allowing it to provide an experience that is similar to how users interact with the application. This makes it easier to identify any issues that users might face, as the testing environment is as close to the real-world experience as possible.

To understand End to End testing, Let’s take a closer look at Mike Cohn’s test automation pyramid. We routinely do each level of testing listed in this pyramid while running automated Cypress testing.

Testing Pyramid Layers

The automation pyramid is a popular framework introduced by Mike Cohn that helps teams to plan and prioritize their testing efforts. It includes three levels of testing, which are:

1. Unit Tests: At the base of the pyramid are the unit tests, which test individual code components such as functions, methods, and classes. Unit tests are typically written by developers and are executed frequently during the development cycle. They are essential in ensuring that individual components of the application work as expected and can catch issues early in the development process.

2. Integration Tests: The middle layer of the pyramid consists of integration tests, which test how different components of the system work together. Integration tests ensure that the various parts of the application can communicate and interact with each other seamlessly. These tests are typically automated and are executed after the unit tests have passed.

3. End-to-End Tests: The top layer of the pyramid is end-to-end testing, which tests the entire application workflow from start to finish. These tests simulate real user scenarios and help ensure that the application functions as expected in a production environment. End-to-end tests are typically automated and are executed less frequently than the lower level tests.

Benefits of End-to-End Testing

There are several benefits of End to End testing. Some of the benefits of E2E testing include:

1. Increased Confidence: E2E testing provides a higher level of confidence in the software application by testing all components together. This testing approach ensures that all the components are integrated correctly and are working as expected.
2. Improved Quality: By testing the application from end-to-end, helps to identify and fix bugs earlier in the development process. This enhances the overall quality of the software.
3. Enhanced User Experience: E2E testing ensures that the application is working as expected for the end user. This helps to provide a better user experience and can lead to increased customer satisfaction.
4. Time and Cost Savings: E2E testing helps to identify issues early in the development cycle, which can save time and money by reducing the need for costly rework later in the process.
5. Better Collaboration: E2E testing promotes better collaboration between different teams working on the same application. This testing approach helps to identify issues that may be caused by a lack of communication between teams.
6. Increased Productivity: By automating the testing process, E2E testing can help to increase productivity by reducing the time and effort required to manually test the application.

Faster Time-to-Market: By catching defects earlier in the development process, end-to-end testing can help to reduce delays and accelerate the time-to-market of the application.

Frameworks for End to End testing

There are several popular frameworks for end-to-end testing, including:

Cypress

Cypress is a JavaScript-based end-to-end testing framework that provides a simple and intuitive API for testing web applications. Cypress supports modern web development technologies like React, Angular, Vue.js, and more. It provides a built-in test runner, and it runs tests in the browser, which makes it fast and reliable.

Cypress runs tests inside the browser; it also provides detailed information about what’s happening at every step of the test, including network requests, console output, and DOM changes. This makes it easier to identify and troubleshoot issues and helps ensure that the application is working as intended.

Cypress Trends on GitHub

The following information is taken from the official website of Cypress GitHub repository:

• Stars: 43.3k
• Forks: 2.8k
• Used By: 797k
• Releases: 303
• Contributors: 427

WebdriverIO

WebdriverIO is a popular open-source testing framework for Node.js that allows developers to automate web applications in a simple and efficient way. It uses the WebDriver API to communicate with browsers and supports a variety of testing frameworks, including Mocha, Jasmine, and Cucumber.

.WebdriverIO Trends on GitHub

The following information is taken from the official website of WebdriverIO GitHub repository:

• Stars: 8.1k
• Forks: 2.3k
• Used By: 50.5k
• Releases: 305
• Contributors: 491

Nightwatch.js

Nightwatch.js is an open-source Node.js-based end-to-end testing framework used to automate browser testing. It provides a simple and easy-to-use syntax for writing automated tests in JavaScript and allows you to run tests in real web browsers like Chrome, Firefox, and Safari.

Nightwatch.js uses the WebDriver protocol to communicate with the browser and control its behavior. It also includes a powerful built-in assertion library that makes it easy to write test assertions and helps you quickly identify issues with your web application.

Nightwatch.js Trends on GitHub

The following information is taken from the official website of Nightwatch.js GitHub repository:

• Stars: 11.4k
• Forks: 1.1k
• Used By: 142k
• Releases: 219
• Contributors: 112

Protractor

Protractor is an open-source end-to-end testing framework for Angular and AngularJS applications. It is built on top of WebDriverJS and uses Jasmine syntax for writing test scripts. Protractor is designed to simulate user interactions with the application and to verify that the application behaves as expected.

Protractor Trends on GitHub

The following information is taken from the official website of Protractor GitHub repository:

• Stars: 8.8k
• Forks: 2.4k
• Used By: 1.9m
• Contributors: 250

TestCafe

TestCafe is an open-source end-to-end testing framework that allows you to automate web testing without using browser plugins. TestCafe is built on top of Node.js and provides a simple and powerful API for testing web applications.

TestCafe Trends on GitHub

The following information is taken from the official website of TestCafe GitHub repository:

• Stars: 9.6k
• Forks: 677
• Used By: 12.3k
• Releases: 390
• Contributors: 117

Benefits End to End Testing Using Cypress

Here are some of the features of Cypress End to End testing:

1. Easy Setup: Cypress has a simple setup process that doesn’t require any additional drivers or libraries. You can get started with Cypress by installing a single package.
2. Automatic Waiting: Cypress automatically waits for elements to appear and become intractable before executing commands. This ensures that the tests are not affected by the timing of the application’s response.
3. Real-time Reloads: Cypress provides real-time reloads, which means that as you make changes to your code or tests, the application will automatically reload, and the tests will be re-run.
4. Interactive Debugging: Cypress provides an interactive test runner, which allows you to debug your tests by stepping through them, setting breakpoints, and viewing the application’s state at any point in time.
5. Time Travel: Cypress allows you to go back and forth in time to see what happened during the execution of a test. This feature is useful for debugging and understanding the behavior of your application.
6. Cross-browser Testing: Cypress allows you to run your tests on multiple browsers and viewports simultaneously. This helps you ensure that your application works correctly across different environments.
7. Network Traffic Control: Cypress allows you to control the network traffic of your application. You can stub, spy, and mock network requests to simulate different scenarios.
8. Automatic screenshots and videos: Cypress automatically takes screenshots and records videos of your tests, which makes it easy to see what went wrong when a test fails.

Frameworks for End to End testing

There are several popular frameworks for end-to-end testing, including:

Cypress

Cypress is a JavaScript-based end-to-end testing framework that provides a simple and intuitive API for testing web applications. Cypress supports modern web development technologies like React, Angular, Vue.js, and more. It provides a built-in test runner, and it runs tests in the browser, which makes it fast and reliable.

Cypress runs tests inside the browser; it also provides detailed information about what’s happening at every step of the test, including network requests, console output, and DOM changes. This makes it easier to identify and troubleshoot issues and helps ensure that the application is working as intended.

Cypress Trends on GitHub

The following information is taken from the official website of Cypress GitHub repository:

• Stars: 43.3k
• Forks: 2.8k
• Used By: 797k
• Releases: 303
• Contributors: 427

WebdriverIO

WebdriverIO is a popular open-source testing framework for Node.js that allows developers to automate web applications in a simple and efficient way. It uses the WebDriver API to communicate with browsers and supports a variety of testing frameworks, including Mocha, Jasmine, and Cucumber.

.WebdriverIO Trends on GitHub

The following information is taken from the official website of WebdriverIO GitHub repository:

• Stars: 8.1k
• Forks: 2.3k
• Used By: 50.5k
• Releases: 305
• Contributors: 491

Nightwatch.js

Nightwatch.js is an open-source Node.js-based end-to-end testing framework used to automate browser testing. It provides a simple and easy-to-use syntax for writing automated tests in JavaScript and allows you to run tests in real web browsers like Chrome, Firefox, and Safari.

Nightwatch.js uses the WebDriver protocol to communicate with the browser and control its behavior. It also includes a powerful built-in assertion library that makes it easy to write test assertions and helps you quickly identify issues with your web application.

Nightwatch.js Trends on GitHub

The following information is taken from the official website of Nightwatch.js GitHub repository:

• Stars: 11.4k
• Forks: 1.1k
• Used By: 142k
• Releases: 219
• Contributors: 112

Protractor

Protractor is an open-source end-to-end testing framework for Angular and AngularJS applications. It is built on top of WebDriverJS and uses Jasmine syntax for writing test scripts. Protractor is designed to simulate user interactions with the application and to verify that the application behaves as expected.

Protractor Trends on GitHub

The following information is taken from the official website of Protractor GitHub repository:

• Stars: 8.8k
• Forks: 2.4k
• Used By: 1.9m
• Contributors: 250

TestCafe

TestCafe is an open-source end-to-end testing framework that allows you to automate web testing without using browser plugins. TestCafe is built on top of Node.js and provides a simple and powerful API for testing web applications.

TestCafe Trends on GitHub

The following information is taken from the official website of TestCafe GitHub repository:

• Stars: 9.6k
• Forks: 677
• Used By: 12.3k
• Releases: 390
• Contributors: 117

Benefits End to End Testing Using Cypress

Here are some of the features of Cypress End to End testing:

1. Easy Setup: Cypress has a simple setup process that doesn’t require any additional drivers or libraries. You can get started with Cypress by installing a single package.
2. Automatic Waiting: Cypress automatically waits for elements to appear and become intractable before executing commands. This ensures that the tests are not affected by the timing of the application’s response.
3. Real-time Reloads: Cypress provides real-time reloads, which means that as you make changes to your code or tests, the application will automatically reload, and the tests will be re-run.
4. Interactive Debugging: Cypress provides an interactive test runner, which allows you to debug your tests by stepping through them, setting breakpoints, and viewing the application’s state at any point in time.
5. Time Travel: Cypress allows you to go back and forth in time to see what happened during the execution of a test. This feature is useful for debugging and understanding the behavior of your application.
6. Cross-browser Testing: Cypress allows you to run your tests on multiple browsers and viewports simultaneously. This helps you ensure that your application works correctly across different environments.
7. Network Traffic Control: Cypress allows you to control the network traffic of your application. You can stub, spy, and mock network requests to simulate different scenarios.
8. Automatic screenshots and videos: Cypress automatically takes screenshots and records videos of your tests, which makes it easy to see what went wrong when a test fails.

Set up Cypress For End to End Testing

To create a new project for Cypress automated testing, follow the steps listed below.

Step 1: Generate package.json.

• Create a project, let’s name it as cypress_applitools
• Use the npm init command to create a package.json file

Step 2: Install Cypress.

Install Cypress by running the command in the newly created folder:

npm install cypress –save-dev

OR

yarn add cypress –dev

Above command will install Cypress locally as a dev dependency for your project.

As shown below, Cypress version 12.11.0 is reflected after installation. The newest Cypress version at the time this blog was being written was 12.11.0.

Below is a diagram of Cypress’s default folder layout. The “e2e” folder is where test cases can be created.

About Project structure of Cypress

Cypress has built a default folder hierarchy when it opens for the first time, as can be seen in the screenshots. Each of these files and folders that Cypress created is described in detail below.

• e2e: All test cases are stored under this folder. This folder contains the actual test files, written in JavaScript, that define the tests to be run.
• Fixtures: This folder contains any data files that are needed for the tests, such as JSON or CSV files.
• Support: There are two files inside the support folder: commands.js and e2e.js
  • command.js: Is the file where your frequently used functions and unique commands are added. It has functions like the login function that you may use in various tests. You can alter some of the functions Cypress generated for you right here.
  • e2e.js: This file is executed before each and every spec file. This file is an excellent location for global configuration and behavior that alters Cypress in the same way as before or before. It just imports commands.js by default, but you can import or need more files to keep things organized.
• Node_Modules: The node_modules directory will have all the node packages installed and all test files will have access to them. When you install Node packages using NPM, they are downloaded and installed in the node_modules directory, which is located in the root directory of your project
• cypress.config.json: cypress.config.json is a configuration file used by Cypress to override the default configuration settings for a project. It is similar to cypress.json, but it is intended to be used as a per-environment configuration file.

Some examples of configuration options that can be set in cypress.config.json include:

• baseUrl: The base URL for the application being tested.
• testFiles: A list of test files to include or exclude from the test suite.
• video: Configuration options for Cypress video recording.
• screenshots: Configuration options for Cypress screenshots.

Basic constructs of Cypress

Cypress used Mocha’s syntax for developing test cases. Key constructs that are frequently used in Cypress test development are listed below.

• describe(): This method is used in Cypress (using Mocha’s syntax) to group together related test cases. It takes two arguments. It takes two arguments: A string that describes the group of test cases (e.g. “Login Page Tests”) and another argument, a callback function that contains the individual test cases (using the it() method).
• it(): This method is used to define an individual test case. It requires two arguments: a string that specifies the test scenario and a callback function that has the test code itself.
• before():  This method is used to run the code under before() block before any test case. The before() method takes one argument: a callback function that contains the setup code to be executed before any of the test cases
• after(): This method is used to run a cleanup once all the test cases are executed. The after() method takes one argument: a callback function that contains the cleanup code to be executed after all the test cases
• beforeEach(): This method is used to run the code under beforeEach() block beforeEach.The beforeEach() method takes one argument: a callback function that contains the code to be executed before each test case
• afterEach(): This method is used to run a cleanup function after each test case. The afterEach() function takes one argument: a callback function that contains the cleanup code to be executed after each test case
• .only(): It is used to run a specified suite or test exclusively, ignoring all other tests and suites. This can be useful when you’re debugging a specific test case or working on a specific suite of tests, and you want to focus on that specific test case or suite without running any others.
• .skip(): It is used to skip a specified suite or test, effectively ignoring it during test execution. This can be useful when you’re working on a test suite or test case that isn’t ready to be run yet, or when you want to temporarily disable a test without deleting it.

The post The Ultimate Guide To End-to-End Testing With Cypress appeared first on Automated Visual Testing | Applitools.

Challenges of frontend testing eCommerce apps

Traditional testing methods don’t always test your eCommerce applications in the same way that your customers shop. Properly capturing user scenarios based on how customers behave in your app is challenging, and writing stable tests that properly test these scenarios scale is tedious and can slow releases.

Data-rich testing scenarios

There’s a lot of data involved in eCommerce apps – product data, user data, app state, and so many different combinations of this data for different users as they shop in your virtual store. All of this data creates a lot of different potential user paths and testing scenarios to cover the different combinations of states based on product information and buyer information.

Multi-screen browsing

Shoppers may be on different browsers, on desktop or different mobile devices, or even using them in tandem to find the product they want – finding the product on their mobile device but completing the purchase on their laptop. A truly omnichannel experience requires that your app looks good and works on every combination of these screen sizes and operating systems.

A/B experimentation

Experimenting with the look and feel of your app can increase conversions and purchase. This puts a lot of pressure on QA to ensure the A/B experiments are properly tested, but these experiments often have complex logic on their own. And to test these A/B experiments, the test will have to mimic the complex experiment logic and understand the context of  the situation. Your product team is doing the work of a permanent test case for an experiment that may be fleeting and not in production for long if it fails.

Localization

There are a lot of global markets and vendors that sell online in many different countries across many different languages. Testing the languages supported is not only a requirement but also a challenge. Building the test cases and resourcing the right people who can test the implemented languages in context takes a lot of time and specific skill sets. To multiply the challenge, each time we add a language, we need to create and maintain new tests across browsers.

Post-production content

Your product or marketing teams may be updating or adding content that enters the app outside of the development process through a CMS, or analysts may be entering information about a new product into an ERP system. Something like adding a new headline or a new percentage to a product may truncate on a smaller screen, going from one line to two lines, which affect spacing elsewhere on the screen or even overlap onto other elements. You’ll need to create a set of tests that are able to be flexibly kicked off and monitored across small aspects of the content that may change without triggering a CI/CD build when changed.

Customer-generated content

Customer-generated content like reviews and product photos are not going to be in your development pipeline. With post-production content, you may have a style guide or requirements for copy and images, but customers aren’t adhering to a style guide. Ensuring that the content appears correctly within your templates for this content is important, and setting character limits and image resolution requirements can keep this content more consistent.

Methods for frontend testing

The traditional way to test frontend functionality includes hundreds of assertions of what the app should do in what we deem are the most important aspects of the experience – like product labels, buy-now buttons, or add-to-cart buttons. Even just testing the priority aspects of your app with these assertions creates technical debt, as things change and tests need to be maintained.

Common methods for frontend testing

We are big believers in doing testing in a layered approach to provide proper coverage of your app. No one way of testing can cover everything. Here are some common methods to include in frontend testing your eCommerce app or website.

Component testing

Component testing is essentially atomic unit tests on components of your frontend. Due to the repeated nature of eCommerce apps – with elements being used across product pages and category pages – testing components of a design system can save a lot of time over checking assertions for individual elements. These components still lack context of the entire application, which leaves room for errors to occur when these components are together in production.

Smoke testing

There’s a lot you can do from a validation standpoint for the web both locally and in test environments to continually make sure that all of your products are showing up. Smoke testing consists of fast tests that validate that the app is able to run without failure, missing bugs that aren’t mission critical. This can include simple crawling of URLs to make sure that they don’t return an error or that uniquely created query parameters are getting created as expected. These tests are the fastest way to get quick uptime coverage for multi-page application e-commerce sites.

End-to-end testing

A common final approach in testing is end-to-end testing around specific scenarios, but even those have challenges. You have to experience end-to-end scenarios to understand them, which means that you’re only able to test for what you know. There may be common bugs that come up in customer tickets or problem areas that you see a lot in QA tickets during your sprints. We can only run so many scenario tests, so you need to take time to establish testing priorities.

Test prioritization

When it comes to prioritizing what to test, it can help to follow the Pareto principle, where 80% of the consequences come from 20% of the causes. This means that you want to test the most impactful aspects of your eCommerce experience to cover the most impact. By this principle, around 20% of your templates are going to drive 80% of your actual revenue, so it’s really important to prioritize the most important 20% of your app or website.

For eCommerce, the most important parts of the app revolve around things like the:

• Home page: Unless your customer followed a direct link through social or paid media, the home page is likely the landing page for most of your customers. The home page is where you’ll get much of your traffic, so your homepage needs to be on brand and working properly.
• Category pages: Category pages are a chance for you to help guide your customers to what they want. These pages often build off templates for navigational consistency for users, so layout testing is an important part of covering these pages.
• Individual pages: Individual product pages are the most numerous, but when you’re using a design system with components, they’re going to be very common in nature. Like category pages, they are often built from templates and layout testing is key.
• Shopping cart: Your users may be browsing your site as they shop, so you need to test that cart data is maintained as a customer traverses the site. Your checkout system is where your users finalize their purchase, so if something goes wrong with it, you lose a sale.
• Search and filtering: A lot of customers will browse items by key words or filter by specific attributes, so making sure that your product metadata is tested can ensure the products show up in these results.

You start to see that you’re testing a lot of the same things. You just need to make sure that you can either use data-driven testing to make sure you’re getting the most scenarios or cross-browser testing of the most important parts across all the experiences that an end user is getting. Visual testing can kind of combine all those aspects – templated pages, localization, cross-browser testing – all at the same time in a much faster way.

How Applitools helps test eCommerce apps

Applitools is a test automation platform that uses AI to help teams ship flawless digital experiences without the hassle of the traditional testing practices.

With Applitools Eyes and our next-gen testing cloud, Ultrafast Grid, developers and QA engineers can run tests to quickly validate frontend functionality, accessibility, and visual correctness with unprecedented speed and accuracy.

Applitools enables you to test for dynamic content, but also unlock a variety of AI-powered visual testing capabilities:

• Visual regression testing: As your digital storefront changes, Applitools Eyes captures screenshots across browsers and devices and compares them to a previously accepted baseline image of your app. Visual AI in Eyes enables you to test layouts and dynamic content, ignore colors, and ignore specific regions of pages or test specifics with customizable match levels.
• Cross-browser testing: Applitools AI-powered automated visual testing can test visual elements across OS, browser, orientation, and resolution combinations. Just running the first baseline rendering and functional test on a single combination is sufficient to test results across the range of potential platforms.
• Automated analysis: Teams can save time with auto-grouping when testing across different browsers, devices, or screen sizes. Auto-grouping will show something like a missing navigation item as one singular bug.
• Automated maintenance: Auto-maintenance automates the creation of new baselines or failure status across the same groupings created by our auto-grouping feature. Auto-maintenance also enables users to set granular controls over what gets updated automatically between checkpoints, test runs, and more.
• Root cause analysis: When unexpected changes do occur on your site, root cause analysis helps teams pinpoint exactly what changed in the DOM. It surfaces the exact difference in DOM elements based on visual changes – eliminating back and forth hunting down bugs.
• Multi-baseline testing: Applitools Eyes supports visual testing of applications that are undergoing A/B testing by allowing checkpoints to be matched against a baseline that includes multiple variations of the baseline image per checkpoint.
• Accessibility testing: Applitools Contrast Advisor seamlessly integrates into your Applitools Eyes test automation workflow to let you instantly see how well your application complies with WCAG color contrast guidelines – you don’t even need to re-run existing tests.

The live demo in the webinar covered visual regression testing, accessibility testing, and multi-baseline testing. To check out the live demo of using Applitools Eyes to test an eCommerce site, you can view the on-demand webinar. If you’re ready to try it yourself, you can create a free account or reach out to our sales team. Happy testing!

The post Ensuring a Reliable Digital Shopping Experience appeared first on Automated Visual Testing | Applitools.

As a product manager at Applitools I am excited to announce an enriched and updated integration with Testim.io! This enhanced integration makes it easier for testers of any technical ability to use Applitools and our AI-powered visual testing platform by using Testim.io to easily create your test scripts.

What Is Testim.io Used For?

Testim.io is a cloud platform that allows users to create, execute, and maintain automated tests without using code.

It is a perfect tool for getting started with your first automated tests, if you do not have an existing automated testing framework or if you have not started to run tests yet. Testim.io allows you to integrate your own custom code into their steps so you can implement custom validations if you need to.

How Do Applitools and Testim.io Integrate?

The visual validation empowered by Applitools Eyes allows you to compare the visual differences between expected results (baseline) with actual results after creating the tests in Testim.io. By using Visual AI to compare snapshots Applitools Eyes can spot any unexpected changes and highlight them visually. This lets you expand your test coverage to include everything on a given page as well as visually verify your results quickly.

As part of the integration, you can modify test parameters to customize Eyes while working with the Testim UI.

This AI-based visual validation functionality is provided by Applitools and requires simple integration setup in the Eyes application. Learn more.

So, What’s New With Applitools and Testim.io?

This up-to-date integration provides access to Applitools’ latest and greatest capabilities, including Ultrafast Test Cloud, enabling ultrafast cross-browser and cross-platform functional and visual testing. Testim users also now have access to Root Cause Analysis and many more powerful Applitools features!

The new integration also greatly improves on the user experience of test creators adding Applitools Eyes checkpoints to their Testim.io tests. Visual validations can be added right inside Testim and the maintenance and analysis of test results is much simpler.

What Kind of Visual Validations Can You Do?

You can perform the following visual validations:

• Element Visualization – The Validate Element visualization step allows you to compare visual differences of a specific element between your baseline and your current test run.
• Viewport Visualization – The Validate Viewport allows you to compare the visual difference between your baseline and the current test run of your viewport.
• Full-page Visualization – Full-page validation allows you to compare the visual differences between your baseline and your current test run of your entire page.

What Are the New Visual Validation Settings?

Whether you select the element, viewport, or full-page visualization option you can always override the visual setting for that test or step.

The following Applitools Eyes settings can be accessed via the Testim.io UI:

• Add Environment (New) – allows you to select Ultrafast Test Cloud environments. You can select the same test to run on multiple environments: different browser types and viewports for web, Chrome emulation, or iOS simulation for mobile devices. Using Applitools Ultrafast Test Cloud you can now increase your coverage and accelerate your release cycles.
  

• Match Level – When writing a visual test, sometimes we will want to change the comparison method between our test and its baseline, especially when dealing with applications that consist of dynamic content. Here you can update the Applitools Eyes match level directly from Testim UI.
• Enable RCA [Root Cause Analysis] (New) – when this flag is on it will provide insights into the causes of visual mismatches so that when looking at the Eyes dashboard you will be able to see the DOM and CSS that generated with the image.
• Ignore displacement (New) – when this flag is on it will hide differences caused by element displacements. This feature is useful, for example, where content is added or deleted, causing other elements on the page to be displaced and generating additional differences. 
  

User Experience Improvements

In addition to exposing new features in the Testim UI, we have provided better visibility to Testim tests in Applitools Eyes:

• Testim test properties are passed to the Eyes Dashboard to allow better filtering and grouping with all Testim tests properties.
• Testim multi-step and test suites are now also grouped on the Applitools Eyes dashboard and are displayed as one batch to create a better user experience when moving between the two products.
• Testim Selenium and extension modes are supported.

Complete and Scalable AI-Powered UI Testing

Testim.io allows users to quickly create and maintain tests through record and playback. Adding Applitools visual testing with Ultrafast Test Cloud capabilities will make sure your release cycles are short and test analysis and maintenance are easier than ever!

Learn More about Testim.io-Applitools Integration

If you want to learn more about how you can integrate your codeless Testim tests with Applitools and benefit from the latest Applitools capabilities, head over to Testim.io documentation.

Contact us if you have any queries about Applitools!

Happy testing!

The post Codeless End-to-End AI-Powered Cross Browser UI Testing with Applitools and Testim.io appeared first on Automated Visual Testing | Applitools.

This post talks about an approach to Functional (end-to-end) Test Automation that works for a product available on multiple platforms. 

It shares details on the thought process & criteria involved in creating a solution that includes how to write the tests, and run it across the multiple platforms without any code change.

Lastly, the open-sourced solution also has examples on how to implement a test that orchestrates the simulation between multiple devices / browsers to simulate multiple users interacting with each other as part of the same test.

We will cover the following topics.

• Background
• Case Study
• Test Automation Approach
• Test Automation Implementation
  • Tech Stack
  • Test Intent Specification
    • Multi-User Scenarios
  • Configurable Framework
  • teswiz Framework Architecture
  • Visual Testing & Contrast Advisor
  • Reporting
• How Can You Get Started?

Background

How many times do we see products available only on a single platform? For example, Android app only, or iOS app only?

Organisations typically start building the product on a particular platform, but then they do expand to other platforms as well. 

Once the product is available on multiple platforms, do they differ in their functionality? There definitely would be some UX differences, and in some cases, the way to accomplish the functionality would be different, but the business objectives and features would still be similar across both platforms. Also, one platform may be ahead of the other in terms of feature parity. 

The above aspects of product development are not new.

The interesting question is – how do you build your Functional (End-2-End / UI / System) Test Automation for such products?

Case Study

To answer this question, let’s take an example of any video conferencing application – something that we would all be familiar with in these times. We will refer to this application as “MySocialConnect” for the remainder of this post.

MySocialConnect is available on the following platforms:

• All modern browsers (Chrome / Firefox / Edge / Safari) available on laptop / desktop computers as well as on mobile devices
• Android app via Google’s PlayStore
• iOS app via Apple’s App Store

In terms of functionality, the majority of the functionality is the same across all these platforms. Example:

• Signup / Login
• Start an instant call
• Schedule a call
• Invite registered users to join an on-going call
• Invite non-registered users can join a call
• Share screen
• Video on-off
• Audio on-off
• And so on…

There are also some functionality differences that would exist. Example:

• Safe driving mode is available only in Android and iOS apps
• Flip video camera is available only in Android and iOS apps

Test Automation Approach

So, repeating the big question for MySocialConnect is – how do you build your Functional (End-2-End / UI / System) Test Automation for such products?

I would approach Functional automation of MySocialConnect as follows:

1. The test should be specified only once. The implementation details should figure out how to get the execution happening across any of the supported platforms
2. For the common functionalities, we should implement the business logic only once
3. There should be a way to address differences in business functionality across platforms
4. The value of the automation for MySocialConnect is to simulate “real calls” – i.e. more than one user in the call – and interacting with each other

In addition, I need the following capabilities in my automation:

• Rich reports
  • With on-demand screenshots attached in the report
  • Details of the devices / browsers where the test 
  • Understand trends of test execution results
  • Test Failure analysis capabilities

• Support parallel / distributed execution of tests to get faster feedback
• Visual Testing support using Applitools Visual AI
  • To reduce the number of validations I need to write (less code)
  • Increase coverage (functional and UI / UX)
  • Contrast Advisor to ensure my product meets the WCAG 2.0 / 2.1 guidelines for Accessibility
• Ability to run on local machines or in the CI
• Ability to run the full suite or a subset of tests, on demand, and without any code change
• Ability to run tests across any environment
• Ability to easily specify test data for each supported environment 

Test Automation Implementation

To help implement the criteria mentioned above, I built (and open-sourced on github) my automation framework – teswiz. The implementation is based on the discussion and guidelines in [Visual] Mobile Test Automation Best Practices and Test Automation in the World of AI & ML. 

Tech Stack

After a lot of consideration, I chose the following tech stack and toolset to implement my automated tests in teswiz.

• cucumber-jvm – specify the test in simple, common business language
• AppiumTestDistribution – manages Android and iOS devices, and Appium
• Appium
• Selenium WebDriver
• reportportal.io
• Applitools Visual AI
• gradle

Test Intent Specification

Using Cucumber, the tests are specified with the following criteria:

• The test intent should be clear and “speak” business requirements
• The same test should be able to execute against all supported platforms (assuming feature parity)
• The clutter of the assertions should not pollute the test intent. That is implementation detail

Based on these criteria, here is a simple example of how the test can be written.

The tags on the above test indicates that the test is implemented and ready for execution against the Android apk and the web browser. 

Multi-User Scenarios

Given the context of MySocialConnect, implementing tests that are able to simulate real meeting scenarios would add the most value – as that is the crux of the product.

Hence, there is support built-in to the teswiz framework to allow implementation of multi-user scenarios. The main criteria for implementing such scenarios are:

• One test to orchestrate the simulation of multi-user scenarios
• The test step should indicate “who” is performing the action, and on “which” platform
• The test framework should be able to manage the interactions for each user on the specified platform.

Here is a simple example of how this test can be specified.

In the above example, there are 2 users – “I” and “you”, each on a different platform – “android” and “web” respectively.

Configurable Framework

The automated tests are run in different ways – depending on the context.

Ex: In CI, we may want to run all the tests, for each of the supported platforms

However, on local machines, the QA / SDET / Developers may want to run only specific subset of the tests – be it for debugging, or verifying the new test implementation.

Also, there may be cases where you want to run the tests pointing to your application for a different environment.

The teswiz framework supports all these configurations, which can be controlled from the command-line. This prevents having to make any code / configuration file changes to run a specific subset type of tests.

teswiz Framework Architecture

This is the high-level architecture of the teswiz framework.

Visual Testing & Contrast Advisor

Based on the data from the study done on the “Impact of Visual AI on Test Automation,” Applitools Visual AI helps automate your Functional Tests faster, while making the execution more stable. Along with this, you will get increased test coverage and will be able to find significantly more functional and visual issues compared to the traditional approach.

You can also scale your Test Automation execution seamlessly with the Applitools UltraFast Test Cloud and use the Contrast Advisor capability to ensure the application-under-test meets the accessibility guidelines of the WCAG 2.0 / 2.1 standards very early in the development stage.

Read this blog post about “Visual Testing – Hype or Reality?” to see some real data of how you can reduce the effort, while increasing the test coverage from our implementation significantly by using Applitools Visual AI.

Hence it was a no-brainer to integrate Applitools Visual AI in the teswiz framework to support adding visual assertions to your implementation simply by providing the APPLITOOLS_API_KEY. Advanced configurations to override the defaults for Applitools can be done via the applitools_config.json file. 

This integration works for all the supported browsers of WebDriver and all platforms supported by Appium.

Reporting

It is very important to have good and rich reports of your test execution. These reports not only make it valuable to pinpoint the reasons of the failing test, but also should be able to give an understanding of the trend of execution and quality of the product under test. 

I have used ReportPortal.io as my reporting tool – it is extremely easy to set up and use and allows me to also add screenshots, log files and other information that may seem important along with the test execution to make root cause analysis easy.

How Can You Get Started?

I have open-sourced this teswiz framework so you do not need to reinvent the wheel. See this page to get started – https://github.com/znsio/teswiz#what-is-this-repository-about

Feel free to raise issues / PRs against the project for adding more capabilities that will benefit all.

The post Automating Functional / End-2-End Tests Across Multiple Platforms appeared first on Automated Visual Testing | Applitools.

How do you reduce code maintenance of your end-to-end tests?

You spend significant effort to develop end-to-end software tests of your web and mobile apps. Do you think you spend too much time on test maintenance?

We assume you are reading this because you have built automation for your end-to-end application tests. When organizations ran these tests with manual testers, test maintenance included informing the testers about the expected behavior change for enhancements and eliminated features, as well as the behavior of new features. Testers might have been informed about appearance changes. Organizations depended on the skills and detail orientation of manual testers to catch any errors.

Because you invested in test automation, you specify your tests as part of your software development process.  Your test construction, however, becomes dependent on developers, because your test controls depend on artifacts your coders create.

Web element locators identify the fields for entering data to set a test condition, as well as the elements to click and initiate action. You might also use locators to validate your test output. Your end-to-end test maintenance effort depends, in large part, on ensuring that locators remain consistent.

Locators And End-To-End Test Maintenance

Application updates can cause your locators to fail. How do you uncover locator changes that can impact your ability to set test conditions or correctly validate output?

Ideally, developers agree not to touch locators. Once they’re set, that’s how they’ll be used. Realistically, locator changes occur and testers get surprised by failing tests.

Some advanced development teams set locators programmatically. You might not depend on statically-set locators. You might use a look-up table or other data structure to uncover the locator you need for a given task. Some programmatic approaches can reduce the failures caused by changing locators.

You incur maintenance costs when you upgrade your application and discover failing tests. If you have failing tests after an application upgrade, you will need to debug the test code. Ultimately, you have to figure out if the test failed due to broken application code or broken testing code.

Actually, you cannot count on the passing tests, either. When you upgrade your application, your test code might continue to pass, but parts of your application unusable. It is difficult and cumbersome to write locators that uncover visual errors.

Visuals As An Alternative To Locators

Visuals provide a great alternative to coded locators. Some test approaches require visual images to apply test conditions. Other approaches use visuals to validate the output of a test.

When you use visuals to control the test conditions, you need a way to ensure that your visual-based controls update consistently. If you update the application, your visuals might change, and you will lose the ability to control a test.

For visuals as outputs, you need a way to identify and approve expected changes and identify unexpected changes as bugs. Since visuals can apply to multiple pages (think an app header and footer), you also would like to group common changes across multiple pages and accept them in a bulk action.

For configuring test conditions using either visuals or locators, your will need to do maintenance work when your applications change the layout or naming of the control interfaces. For validation, however, visual is much more efficient. Visual validation can capture all the changes that affect a given page. When you

Applitools For End-to-End Tests

At Applitools, we focus on visuals to validate the output of any test condition you apply. You can use Applitools along with coded locators to ensure you catch visual errors. Or, you can choose to remove coded locators and replace them with a visual capture and validation. step

We recommend using Applitools to validate your end-to-end tests for several reasons.

• End-to-end tests validate the user experience, which is inherently visual.
• You likely already have functional coverage for your unit and system tests. Using those tests verbatim for end-to-end tests gives you no additional knowledge of your application behavior.
• End-to-end test code with visual assertions requires much less code maintenance.
• Applitools Auto-Maintenance and newly-announced Smart Assist make it easy to deploy and maintain your visual assertions.

Applitools uses an artificial intelligence (AI) engine to recognize individual visual elements on a page. We call this capability Applitools Visual AI. Rather than use pixels to determine visual differences, Applitools identifies and compares visually-distinct versions of elements. Visual AI distinguishes Applitools from all other visual comparison solutions because

By using Visual AI, you capture an entire page with a single assertion. Your captured page gets loaded into the Applitools Eyes server. The output comparison relies on fewer lines of code and reduces any fragility associated with using locators.

Applitools Reduces End-to-End Test Maintenance

Applitools makes it easy to maintain the visual data it collects. A great Visual AI feature is called “Automated Test Maintenance”, or Auto Maintenance. With Auto Maintenance, Applitools identifies common visual differences found on multiple pages. When you inspect a difference found by Applitools, you can choose to accept it or reject it. Once you have made the choice, Auto Maintenance lets you know about the other pages with the identical difference. With one click, you can choose to accept or reject the change for those pages as well.

For instance, you update the color scheme of your application logo. Ths logo is found in the same place on every page. You might think you have to go page by page and accept the change. Auto Maintenance makes it easy to accept all the update logos. You accept the first found logo change, then you are prompted to accept the changes everywhere else.

Simplifying Deployment With Smart Assist

Another useful feature in Applitools Visual AI is the recently-announced Smart Assist. Applitools allows you to choose among three useful visual comparison levels: Strict, Content, and Layout.

• In Strict mode, Applitools highlights any user-identifiable visual change.
• In Content mode, Applitools inspects content, not the color, font, or other visual difference. For example, you change CSS and want to ensure that nothing else has changed.
• Layout mode handles dynamic content on a page. Applitools compares the structure around the dynamic content, and that the dynamic content has no errors.

On its own, Applitools provides you lots of flexibility. You can choose the comparison level for a given page. If necessary, you can apply different comparison level to a specific region on a page.  For example, say you have third-party ads on your pages. You might define those regions as “Layout”, while the rest of the page is Strict.

With Smart Assist, Applitools suggests the appropriate comparison level as well as exceptions for individual regions within the screen images. Even if you choose Strict as the default page comparison level, Applitools will suggest appropriate exceptions for you. Strict, content, layout, or even ignore – Smart Assist can help you make the right call.

Conclusion

Visual AI eases test deployment and maintenance. You eliminate the need to validate pages based on DOM locators, which reduces your total lines of test code. You eliminate the need to update DOM locators in your test code, reducing test debug and maintenance time. And, your resulting test code evaluates your application based on what a user sees – not simply inspecting the DOM.

Applitools Smart Assist helps you ensure that you use the right level of comparison for your application visual tests. You can become an Applitools expert by letting Applitools Smart Assist be your guide.

Cover Photo by Peter G on Unsplash

For More Information

• Visual Testing: A Guide for Front End Developers
• Modern Cross Browser Testing with Cypress and Applitools
• How Do You Catch More Bugs In Your End-To-End Tests?
• Sign up for a free Applitools account.
• Request an Applitools demo.

The post How Do You Simplify End-To-End Test Maintenance? appeared first on Automated Visual Testing | Applitools.

As a quality engineer, what do you think about when you consider the return on investing an hour of your time? Or two hours? What if it could save you up to half the work hours you spent over the past year?

If you spend a lot of time writing or maintaining your end-to-end tests, our data shows that you’ll find lots of savings. What would you do with that time?

I have written previously about our Impact of Visual AI on Test Automation report. And, I have written about the benefits that many of your peers discovered by using Applitools Visual AI. They reduced the number of lines of code they had to write. Their code became more stable and easier to maintain. They caught more bugs with less code. And they learned how to do all this in about one to two hours.

So, what can an hour or two of your time give you?

ROI For A Quality Engineer Hour

Business people like me often talk about return on investment (ROI). If we invest this money today, what does the future payoff look like, and when does it come?

ROI offers a good model for thinking for everyone. Including a decision about how to spend your time to learn a new skill. If you spend an hour learning something now, what is the future benefit to you, and when do you get it?

The ROI of a quality engineer hour might be measured by:

• Increasing the number of test cases you can run
• Increasing the number of bugs you can uncover
• Decreasing the amount of low-value code you write.
• Decreasing your code maintenance effort.

So if you’re going to invest one or two hours into learning a new technology, like, say, Visual AI, you would like to see many hours worth of return on your time.

Visual AI frees you from writing inspection code in your end-to-end tests. Inspection code has the highest chance to contain errors, gets written selectively to provide limited coverage, and still can require a high degree of coding skill.  

You might think that your value as a software engineer comes from your coding skills, and that often devolves into the inane measure of the value of a software engineer by counting lines of code written. Truthfully, not all code has the same value. In fact, code in of itself has zero value. It’s what that code does to help you test more conditions that matters.

High-Value Code And Low-Value Code

Your end-to-end tests contain both high-value code and low-value code.  

The high-value code exercises your application. It sets the test cases. It runs execution variations. The high-value code correlates to source code and UI test coverage.

The low-value code inspects your results. And, if you’re using state-of-the-art coding, you’re inspecting the DOM for output web elements. Some you find with an ID. Others you find with a CSS selector. Some you find with a relative Xpath expression. Sometimes this code can involve incredible complexity. For example, writing all the assertions for reordering a table can involve elegant coding skills. And, yet…

If you review your end-to-end tests, much of your code determines whether the DOM contains the proper elements to let you conclude that the app passed. And, you have become good at triage. You cannot cover the state of every element in the DOM, so you become great at selecting code to inspect that tells you if your test passed or failed. Still, you write a lot of inspection code compared to execution code – and you only inspect a selected subset of the DOM.

But, the time you take writing this low-value code detracts from some of the high-value activity you can add to your end-to-end tests. For instance – you can write the preconditions for each test to ensure that they can be run independently – and, thus, in parallel. You can review the test conditions evaluated by each test to eliminate redundancy and improve automation accuracy.

What about maintenance? We didn’t even include code maintenance efforts you undertake between releases. That is yet more end-to-end coding effort you need – to validate and update existing tests, add new tests, and resolve failures – every time you modify the application. And, yes, some of that code provides new or modified test conditions. As well, some of that code needs to modify your inspection code.

Visual AI Replaces Low-Value Code

When we developed Visual AI, we recognized that a quality engineer makes trade offs. One engineer can only write a finite number of lines of code. An incomplete test has no value. So, every line of code needed to execute a test and validate the results makes a complete test.

We also recognized the limitations of DOM inspection for test results validation. Inspected thoroughly, the DOM represents the elements of the page to be rendered by the browser. And, one can write a detailed DOM inspection – and even account for variations in the DOM structure between releases. However, that depth of inspection involves complexity that rarely pays off for the coder. So, most coders spot-check their DOM – and can miss unexpected changes between releases unless validating the application through manual testing.

Visual AI uses one line of code to capture every visual element on your web page – and the state of the DOM that created the page. Once captured, Visual AI compares your captured version to your baseline. Then, Visual AI highlights for you the real user-noticeable differences between the baseline and the new capture. From there, you choose to accept the changes as expected new features or reject them as bugs. And, you can link those differences directly to the DOM code used to generate those differences.

Since inspection statements make up the bulk of your end-to-end test code, by adding Visual AI, you can eliminate the bulk of your inspection statements – letting you write code faster, making your code more stable, and allowing you more time to work on high-value test automation tasks.

How Long Does It Take To Learn Visual AI?

When we started the Applitools Visual AI Rockstar Hackathon, we directed participants to two courses on Test Automation University (TAU). TAU, offered exclusively by Applitools, offers classes on a range of technologies, including:

• Programming languages (like C and Python)
• Test frameworks (like Selenium and Cypress)
• Mobile testing
• API automation
• And, of course, Visual AI

We pointed participants to one course written by Raja Rao describing how to modernize your functional tests with Visual AI. Raja walked through the different test cases on the Hackathon in about an hour. We also pointed participants to a course by Angie Jones on how to add Visual AI to your test automation.  Each course took approximately an hour.

Hackathon participants got pretty amazing results. After an hour or two of classes, they applied their knowledge of Visual AI and found:

• Average test coverage jumped from 65% for coded inspection to 95% for Visual AI
• Test writing time, on average, dropped from a little over 7 hours to a little over 1 hour.
• The amount of code they wrote dropped significantly

So, for one or two hours of learning, Hackathon participants got faster test writing, more coverage, and less code.

Testing Visual AI Yourself

In the end, testing is a scientific activity. We run a series of tests and make observations – pass or fail.

In this blog, you’re reading a bunch of claims about Visual AI. These are based on data that we shared in our report about the Hackathon results.

What do these claims mean to you?

I recommend that you test these claims yourself. The claim – one or two hours of learning can help you write tests:

• With comprehensive inspection
• Requiring less code
• More easily maintained
• Let you focus your time on high-value test activity.

If true, would that be worth the investment?

It is up to you to find out.

Learn More

• Modern Functional Test Automation Through Visual AI
• Automated Visual Testing: A Fast Path To Test Automation Success

James Lamberti is CMO at Applitools

The post How Can 2 Hours Now Save You 1,000 Hours In The Next Year? appeared first on Automated Visual Testing | Applitools.