Learn how you can run cross-browser Cypress tests against any browser, including Safari, IE and mobile browsers.
Ah, Cypress – the darling end-to-end test framework of the JavaScript world. In the past few years, Cypress has surged in popularity due to its excellent developer experience. It runs right in the browser alongside web apps, making it a natural fit for frontend developers. Its API is both concise and powerful. Its interactions automatically handle waiting to avoid any chance of flakiness. Cypress almost seems like a strong contender to dethrone Selenium WebDriver as the king of browser automation tools.
However, Cypress has a critical weakness: it cannot natively run tests against all browser types. At the time of writing this article, Cypress supports only a limited set of browsers: Chrome, Edge, and Firefox. That means no support for Safari or IE. Cypress also doesn’t support mobile web browsers. Ouch! These limitations alone could make you think twice about choosing to automate your tests in Cypress.
Thankfully, there is a way to run Cypress tests against any browser type, including Safari, IE, and mobile browsers: using Applitools Ultrafast Test Grid. With the help of Applitools, you can achieve full cross-browser testing with Cypress, even for large-scale test suites. Let’s see how it’s done. We’ll start with a basic Cypress test, and then we’ll add visual snapshots that can be rendered in any browser in the Applitools cloud.
Defining a Test Case
Let’s define a basic web app login test for the Applitools demo site. The site mimics a basic banking app. The first page is a login screen:
You can enter any username or password to login. Then, the main page appears:
Nothing fancy here. The steps for our test case are straightforward:
Scenario: Successful login
Given the login page is displayed
When the user enters their username and password
And the user clicks the login button
Then the main page is displayed
These steps would be the same for the login behavior of any other application.
Automating a Cypress Test
Let’s automate our login test using Cypress. Create a JavaScript project and install Cypress. Then, create a new test case spec: cypress/integration/login.spec.js. Add the following test case to the spec file:
describe('Login', () => {
beforeEach(() => {
cy.viewport(1600, 1200)
})
it('should log into the demo app', () => {
loadLoginPage()
verifyLoginPage()
performLogin()
verifyMainPage()
})
})
Cypress uses Mocha as its core test framework. The beforeEach call makes sure the browser viewport is large enough to show all elements in the demo app. The test case itself has a helper function for each test step.
The first function, loadLoginPage, loads the login page:
function loadLoginPage() {
cy.visit('https://demo.applitools.com')
}
The second function, verifyLoginPage, makes sure that the login page loads correctly:
function verifyLoginPage() {
cy.get('div.logo-w').should('be.visible')
cy.get('#username').should('be.visible')
cy.get('#password').should('be.visible')
cy.get('#log-in').should('be.visible')
cy.get('input.form-check-input').should('be.visible')
}
The third function, performLogin, actually does the interaction of logging in:
function performLogin() {
cy.get('#username').type('andy')
cy.get('#password').type('i<3pandas')
cy.get('#log-in').click()
}
The fourth and final function, verifyMainPage, makes sure that the main page loads correctly:
function verifyMainPage() {
// Check various page elements
cy.get('div.logo-w').should('be.visible')
cy.get('div.element-search.autosuggest-search-activator > input').should('be.visible')
cy.get('div.avatar-w img').should('be.visible')
cy.get('ul.main-menu').should('be.visible')
cy.contains('Add Account').should('be.visible')
cy.contains('Make Payment').should('be.visible')
cy.contains('View Statement').should('be.visible')
cy.contains('Request Increase').should('be.visible')
cy.contains('Pay Now').should('be.visible')
// Check time message
cy.get('#time').invoke('text').should('match', /Your nearest branch closes in:( \d+[hms])+/)
// Check menu element names
cy.get('ul.main-menu li span').should(items => {
expect(items[0]).to.contain.text('Card types')
expect(items[1]).to.contain.text('Credit cards')
expect(items[2]).to.contain.text('Debit cards')
expect(items[3]).to.contain.text('Lending')
expect(items[4]).to.contain.text('Loans')
expect(items[5]).to.contain.text('Mortgages')
})
// Check transaction statuses
const statuses = ['Complete', 'Pending', 'Declined']
cy.get('span.status-pill + span').each(($span, index) => {
expect(statuses).to.include($span.text())
})
}
The first three functions are fairly concise, but the fourth one is a doozy. The main page has so many things to check, and despite its length, this step doesn’t even check everything!
Run this test locally to make sure it works (npx cypress open). It should pass using any local browser (Chrome, Edge, Electron, or Firefox).
Introducing Visual Snapshots
You could run this login test on your local machine or from your Continuous Integration (CI) service, but in its present form, it can’t run on those extra browsers (Safari, IE, mobile). To do that, we need the help of visual testing techniques using Applitools Visual AI and the Ultrafast Test Cloud.
Visual testing is the practice of inspecting visual differences between snapshots of screens in the app you are testing. You start by capturing a “baseline” snapshot of, say, the login page to consider as “right” or “expected.” Then, every time you run the tests, you capture a new snapshot of the same page and compare it to the baseline. By comparing the two snapshots side-by-side, you can detect any visual differences. Did a button go missing? Did the layout shift to the left? Did the colors change? If nothing changes, then the test passes. However, if there are changes, a human tester should review the differences to decide if the change is good or bad.
Manual testers have done visual testing since the dawn of computer screens. Applitools Visual AI simply automates the process. It highlights differences in side-by-side snapshots so you don’t miss them. Furthermore, Visual AI focuses on meaningful changes that human eyes would notice. If an element shifts one pixel to the right, that’s not a problem. Visual AI won’t bother you with that noise.
If a picture is worth a thousand words, then a visual snapshot is worth a thousand assertions. We could update our login test to take visual snapshots using Applitools Eyes SDK in place of lengthy assertions. Visual snapshots provide stronger coverage than the previous assertions. Remember how verifyMainPage had several checks but still didn’t cover all the elements on the page? A visual snapshot would implicitly capture everything with only one line of code. Visual testing like this enables more effective functional testing than traditional assertions.
But back to the original problem: how does this enable us to run Cypress tests in Safari, IE, and mobile browsers? That’s the magic of snapshots. Notice how I said “snapshot” and not “screenshot.” A screenshot is merely a grid of static pixels. A snapshot, however, captures full page content – HTML, CSS, and JavaScript – that can be re-rendered in any browser configuration. If we update our Cypress test to take visual snapshots of the login page and the main page, then we could run our test one time locally to capture the snapshots, Then, the Applitools Eyes SDK would upload the snapshots to the Applitools Ultrafast Test Cloud to render them in any target browser – including browsers not natively supported by Cypress – and compare them against baselines. All the heavy work for visual checkpoints would be done by the Applitools Ultrafast Test Cloud, not by the local machine. It also works fast, since re-rendering snapshots takes much less time than re-running full Cypress tests.
Updating the Cypress Test
Let’s turn our login test into a visual test. First, make sure you have an Applitools account. You can register for a free account to get started.
Your account comes with an API key. Visual tests using Applitools Eyes need this API key for uploading results to your account. On your machine, set this key as an environment variable.
On Linux and macOS:
$ export APPLITOOLS_API_KEY=<value>On Windows:
> set APPLITOOLS_API_KEY=<value>Time for coding! The test case steps remain the same, but the test case must be wrapped by calls to Applitools Eyes:
describe('Login', () => {
it('should log into the demo app', () => {
cy.eyesOpen({
appName: 'Applitools Demo App',
testName: 'Login',
})
loadLoginPage()
verifyLoginPage()
performLogin()
verifyMainPage()
})
afterEach(() => {
cy.eyesClose()
})
})
Before the test begins, cy.eyesOpen(...) tells Applitools Eyes to start watching the browser. It also sets names for the app under test and the test case itself. Then, at the conclusion of the test, cy.eyesClose() tells Applitools Eyes to stop watching the browser.
The interaction functions, loadLoginPage and performLogin, do not need any changes. The verification functions do:
function verifyLoginPage() {
cy.eyesCheckWindow({
tag: "Login page",
target: 'window',
fully: true
});
}
function verifyMainPage() {
cy.eyesCheckWindow({
tag: "Main page",
target: 'window',
fully: true,
matchLevel: 'Layout'
});
}
All the assertion calls are replaced by one-line snapshots using Applitools Eyes. These snapshots capture the full window for both pages. The main page also sets a match level to “layout” so that differences in text and color are ignored.
The test code changes are complete, but you need to do one more thing: you must specify browser configurations to test in the Applitools Ultrafast Test Cloud. Add a file named applitools.config.js to the root level of the project, and add the following content:
module.exports = {
testConcurrency: 5,
apiKey: 'APPLITOOLS_API_KEY',
browser: [
// Desktop
{width: 800, height: 600, name: 'chrome'},
{width: 700, height: 500, name: 'firefox'},
{width: 1600, height: 1200, name: 'ie11'},
{width: 1024, height: 768, name: 'edgechromium'},
{width: 800, height: 600, name: 'safari'},
// Mobile
{deviceName: 'iPhone X', screenOrientation: 'portrait'},
{deviceName: 'Pixel 2', screenOrientation: 'portrait'},
{deviceName: 'Galaxy S5', screenOrientation: 'portrait'},
{deviceName: 'Nexus 10', screenOrientation: 'portrait'},
{deviceName: 'iPad Pro', screenOrientation: 'landscape'},
],
batchName: 'Modern Cross-Browser Testing Workshop'
}
This config file contains four settings:
testConcurrencysets the level of parallel execution in the Applitools Ultrafast Test Cloud. (Free accounts are limited to 1 concurrent test.)apiKeysets the environment variable name for the Applitools API key.browserdeclares a list of browser configurations to test. This config file provides ten total configs: five desktop, and five mobile. Notice that Safari and IE11 are included. Desktop browser configs include viewport sizes, while mobile browser configs include screen orientations.batchNamesets a name that all results will share in the Applitools Dashboard.
Done! Let’s run the updated test.
Running our Cypress Cross Browser Test
Run the test locally to make sure it works (npx cypress open). Then, open the Applitools dashboard to view visual test results:
Notice how this one login test has one result for each target configuration. All results have “New” status because they are establishing baselines. Also, notice how little time it took to run this batch of tests:
Running our test across 10 different browser configurations with 2 visual checkpoints each at a concurrency level of 5 took only 36 seconds to complete. That’s ultra fast! Running that many test iterations locally or in a traditional Cypress parallel environment could take several minutes.
Run the test again. The second run should succeed just like the first. However, the new dashboard results now say “Passed” because Applitools compared the latest snapshots to the baselines and verified that they had not changed:
This time, all variations took 32 seconds to complete – about half a minute.
Passing tests are great, but what happens if a page changes? Consider an alternate version of the login page:
This version has a broken icon and a different login button. Modify the loadLoginPage function to test this version of the site like this:
function loadLoginPage() {
cy.visit('https://demo.applitools.com/index_v2.html')
}
Now, when you rerun the test, results appear as “Unresolved” in the Applitools dashboard:
When you open each result, the dashboard will display visual comparisons for each snapshot. If you click the snapshot, it opens the comparison window:
The baseline snapshot appears on the left, while the latest checkpoint snapshot appears on the right. Differences will be highlighted in magenta. As the tester, you can choose to either accept the change as a new baseline or reject it as a failure.
Taking the Next Steps
Even though Cypress can’t natively run tests against Safari, IE, or mobile browsers, it can with visual testing with Applitools Ultrafast Test Cloud. You can use Cypress tests to capture snapshots and then render them under any number of different browser configurations to achieve true cross-browser testing with visual and functional test coverage.
Want to see the full code? Check out this GitHub repository: applitools/workshop-cbt-cypress.
Want to try visual testing for yourself? Register for a free Applitools account.
Want to see more examples? Check out other articles here, here, and here.
