On February 1, 2022, I gave a webinar entitled Cross Browser Testing with Cypress, in which I explained how to run Cypress tests against any browser using Applitools Visual AI and the Ultrafast Test Cloud. We had many attendees and lots of great questions – so many questions that we couldn’t answer them all during the event. In this article, I do my best to provide answers to as many remaining questions as possible.
Questions about the Webinar
Is there a recording for the webinar?
Yes, indeed! The recording is here.
Where is the repository for the example code?
Here it is: https://github.com/applitools/workshop-cbt-cypress. The repository also contains a full walkthrough in the WORKSHOP.md file.
Questions about Cypress
Can we run Cypress tests through the command line?
Yes! The npx cypress open command opens the Cypress browser window for launching tests, while the npx cypress run command launches tests purely from the command line. Use npx cypress run for Continuous Integration (CI) environments.
So, in a Cypress test case, we don’t need to create any driver object for opening a browser?
Correct! When you initialize a Cypress project, it sets up all the imports and references you need. Just call the cy object. To navigate to our first page, call cy.visit(…) and provide the URL as a string.
Can Cypress handle testing with iFrames?
Yes! Check out this cookbook recipe, Working with iFrames in Cypress.
How does cy.contains(…) work?
The cy.contains(…) call selects elements based on their text. For example, if a button has the text “Add Account”, then cy.contains(“Account”) would locate it. Check the Cypress docs for more detailed information.
Can Cypress access backend intranet APIs?
I haven’t done that myself, but it looks like there are ways to set up Windows Authentication and proxies with Cypress.
Questions about Applitools
How do I establish baseline snapshots?
The first time you take a visual snapshot with Applitools Eyes, it saves the snapshot as the baseline. The next time the same snapshot is taken, it is treated as a checkpoint and compared against the baseline.
Does Applitools Eyes fail a test if a piece of content changes on a page?
Every time Applitools Eyes detects a change, it asks the tester to decide if the change is good (“thumbs up”) or bad (“thumbs down”). Applitools enables testers to try different match levels for comparisons. For example, if you want to check for layout changes but ignore differences in text and color, you can use “layout” matching. Alternatively, if the text matters to you but layout changes don’t, you can use “content” matching. Applitools also enables testers to ignore regions of the snapshots. For example, a timestamp field will be different for each snapshot, so those could easily be ignored.
How do we save a new baseline snapshot if pages change during development?
When a tester marks a change as “good,” the new snapshot is automatically saved as the new baseline.
What happens if we have thousands of tests and developers change the UI? Will I need to modify thousands of baselines?
Hopefully not! Most UI changes are localized to certain pages or areas of an app. In that case, only those baselines would need updates. However, if the UI changes affect every screen, such as a theme change, then you might need to refresh all baselines. That isn’t as bad as it sounds: Applitools has AI-powered maintenance capabilities. When you accept one new snapshot as a baseline, Applitools will automatically scan all other changes in the current batch and accept similar changes. That way, you don’t need to grind through a thousand “thumbs-up” clicks. Alternatively, you could manually delete old baselines through the Applitools dashboard and rerun your tests to establish fresh ones. You could also establish regions to ignore on snapshots for things like headers or sidebars to mitigate the churn caused by cross-cutting changes.
Does Applitools Eyes wait for assets such as images, videos, and animations to load before taking snapshots?
No. The browser automation tool or the automation code you write must handle waiting.
(* Actually, there is a way when not using the Ultrafast Test Cloud. The classic SDKs include a parameter that you can set for controlling Eyes snapshot retries when there is no match.)
Can we accept some visual changes while rejecting others for one checkpoint?
Yes, you can set regions on a snapshot to use different match levels or to be ignored entirely.
Can we download the snapshot images from our tests?
Yes, you can download snapshot images from the Applitools Eyes dashboard.
Does Applitools offer any SDKs for visually testing Windows desktop apps?
Yes! Applitools offers SDKs for Windows CodedUI, Windows UFT, and Windows Apps. Applitools also works with Power Automate Desktop.
Does the Applitools Ultrafast Grid use real or emulated mobile devices?
Presently, it uses emulated mobile devices.
Can I publicly share my Applitools API key?
No, do NOT share your API key publicly! That should be kept secret. Don’t let others run their tests using your account!
Questions about Applitools with Cypress
How do I set up Applitools to work with Cypress?
Follow the Applitools Cypress Tutorial. You’ll need to:
- Register an Applitools account.
- Install the @applitools/eyes-cypress package.
- Run npx eyes-init to set up Applitools Eyes.
- Set the APPLITOOLS_API_KEY environment variable to your API key.
Cypress cannot locally run tests against Safari, Internet Explorer, or mobile browsers. Can Cypress tests check snapshots on these browsers in the Applitools Ultrafast Test Cloud?
Yes! Snapshots capture the whole page, not just pixelated images. Applitools can render snapshots using any browser configuration, even ones not natively supported by Cypress.
Can Applitools Eyes focus on specific elements instead of an entire page?
Yes! You can check a specific web element as a “region” of a page like this:
cy.eyesCheckWindow({
target: 'region',
selector: {
type: 'css',
selector: '.my-element'
}
});
Can we run visual tests with Cypress using a free Applitools account?
Yes, but you will not be able to access all of Applitools’ features with a free account, and your test concurrency will be limited to 1.
Can we perform traditional assertions together with visual snapshots?
Sure! Visual testing eliminates the need for most traditional assertions, but sometimes, old-school assertions can be helpful for checking things like text formatting. Cypress uses Chai for assertions.
Questions about Testing
If a project has little-to-no test automation in place, should we start writing visual tests right away, or should we start with traditional functional tests and add visual tests later?
Visual tests are arguably easier to automate than traditional functional tests because they simplify assertions. Apply the 80/20 rule: start with a small “smoke” test suite that simply captures snapshots of different pages in your web app. Run that suite for every code change and see the value it delivers. Next, build on it by covering more interactions than navigation. Then, once those are doing well, try to automate more complicated behaviors. At that point, you might need some traditional assertions to complement visual checkpoints.
Can we compare snapshots from a staging environment to a production environment?
Yes, you can compare results across test environments as long as the snapshots have the same app, test, and tag names.
Can we schedule visual tests to run every day?
Certainly! Both Applitools and Cypress can integrate with any Continuous Integration (CI) system.
Does visual testing have any disadvantages when compared to traditional functional testing?
Presently, visual testing does not check specific text formatting, such as dates or currencies. You’ll need to use traditional assertions for that type of pattern matching. Nevertheless, you can use visual testing together with traditional techniques to automate the best functional tests possible.
How do we test UX things like heading levels, fonts, and text sizes?
If you take visual snapshots of pages, then Applitools Eyes will detect differences like these. You could also automate traditional assertions to verify specific attributes such as a specific heading number or font name, but those kinds of assertions tend to be brittle.
What IDE should we use for developing Cypress tests with Applitools?
Any JavaScript editor should work. Visual Studio Code and JetBrains WebStorm are popular choices.
What tool or framework should we use for API testing?
Cypress has built-in API support with the cy.request(...) method, making it easy to write end-to-end tests that interact with both the frontend and backend. However, if you want to automate tests purely for APIs, then you should probably use a tool other than Cypress. Postman is one of the most popular tools for API testing. If you want to stick to JavaScript, you could look into SuperTest and Nock.
Can we do load testing with cross-browser testing?
Load testing is the practice of adding different intensities of “load” to a system while running functional and performance tests. For web apps, “load” is typically a rate of requests (like 100 requests per second). As load increases, performance degrades, and functionality might start failing. You can do load testing with cross-browser testing, but keep in mind that any failures due to load would probably happen the same way for any browser. Load hits the backend, not the frontend. Repeating load tests for a multitude of different browser configurations may not be worthwhile.
