Wondering how to decide between Cypress and Playwright for your test automation? Check out the results of our head to head battle of Cypress vs Playwright and see who comes out on top.

On the 26th of May Applitools hosted another “Let the Code Speak!” event. This time it was focused on two rising stars in web test automation – Cypress and Playwright. Both of these tools have huge fan bases and users that have reasons to either love or doubt these tools. The event had more than 500 attendees that decided on the ultimate winner. To determine the best framework, I (Filip Hric) and Andrew Knight a.k.a Automation Panda shared short code snippet solutions on various different testing problems in front of an online audience. Right after each example, the audience got to vote for the better solution in each of the 10 rounds.

Why Compare Playwright vs Cypress?

Cypress and Playwright both brought some novelties to the world of web test automation. These often work well with modern web apps that are full of dynamic content, often fetched from the server by REST API. For cases like these, automatic waiting is a must. But while these tools carry some similarities, they also differ in many aspects. These stem from the basic architecture of these tools, choice of supported languages, syntax they use, and more.

There is no silver bullet in test automation. All these differences can make a tool super useful, or inefficient, based on the project you are working on. It is comparisons like these that aim to help you decide, and have a little fun along the way. 

If you missed the event, there’s no need to worry. The whole recording is available online, and if you want to check out the code snippets that were used, I recommend you to take a look into the GitHub repository that we have created for this event.

Cypress vs Playwright Head to Head – Top 10 Features Compared

Round 1: How to Interact with Elements

We started off with the basics, interacting with a simple login form. The goal was to compare the most simple flow, and the one you usually start your test automation with. 

At first sight these two code samples don’t look too different from one another. But the crowd decided that the Cypress syntax was slightly more concise and voted 61% in its favor.

Round 2: How to Test iframes

Although iframes are not as common as they used to be, they can still present a challenge to QA engineers. In fact, they need an additional plugin in Cypress, which was probably why it lost this round. Playwright has native API to switch to any given iframe that takes away the extra leg work of installing a plugin.

Round 3: How Cypress and Playwright Handle Waiting and Retrying

With the nature of modern apps, waiting for changes is essential. Single page applications re-render their content all the time. This is something testing tools need to account for nowadays. Built in waiting and retrying capabilities prove to give the edge to modern testing tools like Cypress and Playwright. 

Taking a look at the code, this could be anyone’s win, but this round went to Cypress with a 53% of audience vote.

Round 4: How to Test Native Browser Alerts in Cypress vs Playwright

Given the different design of each tool, it was interesting to see how each of them deal with native browser events. Playwright communicates with the browser using a websocket server, while Cypress is injected inside the browser and automates the app from there. Handling native browser events might be more complicated then and it has proven to be the case in this round. While Playwright showed a consistent solution for alerts and prompts, Cypress had its own solution for all three cases, which caused a sweeping 91% victory by Playwright in this round.

Round 5: Navigation to New Windows

In the next example, we attempted to automate a page that opens a new window. The design of each of the tools proved to be a deciding factor once again. While Playwright has an API to handle a newly opened tab, Cypress reaches for a hack solution that removes the target attribute from a link and prevents opening of a new window entirely. While I argued that this is actually a good enough solution, testers in the audience did not agree and out-voted Cypress 80:20 in favor of Playwright.

Round 6: Handling API Requests

Being able to handle API requests is an automation superpower. You can use them to setup your application, seed data, or even log in. Or you can decide to create a whole API test suite! Both Cypress and Playwright handle API requests really well. In Playwright, you create a new context and fire API request from that context. Cypress uses its existing command chain syntax to both fire a request and to test it. Two thirds of the audience liked the Cypress solution better and gave it their vote.

Round 7: Using the Page Objects Pattern

Although page objects are generally not considered the best option for Cypress, it is still a popular pattern. They provide necessary abstraction and help make the code more readable. There are many different ways to approach page objects. The voting was really close here. During the live event it actually seemed like Playwright won this one, after the show we found out that this round ended up with a tie. 

Round #8 – Cypress and Playwright Language Support

The variety of languages that testers use nowadays is pretty big. That’s why Playwright’s wider support of languages seems like a clear winner in this round. Cypress however tries to cater to the developer’s workflow, where the support of JavaScript and TypeScript is good enough. However, this may feel like a pain point to testers that come from different language backgrounds and are not used to writing their code in these languages. It seemed that the audience agreed that wider languages support is better and voted 77% in favor of Playwright.

Round 9: Browser Support in Cypress and Playwright

Although Chrome is the most popular browser and has become dominant in most countries, browser support is still important when testing web applications. Both tools have good support for various browsers, although Cypress currently lacks support for Safari or WebKit. Maybe this helped the audience decide on the final round win of Playwright.

Round 10: Speed and Overall Performance

Last round of the event was all about the speed. Everyone likes their tests to run fast, so they can get the feedback about the state of their application as soon as possible. Playwright was a clear winner this time, as its execution time was 4x faster than Cypress. Some latest improvements on Cypress’ side have definitely helped, but Playwright is still king in terms of speed.

And the (real) winner of Cypress vs Playwright is…

The whole code battle ended up in 7:3 favor of Playwright. After the event, we met for a little aftershow, discussed the examples in more depth and answered some questions. This was a great way to provide some more context to the examples and discuss things that had not been said.

I really liked a take from someone on Twitter who said that the real winners were the testers and QA engineers that get to pick between these awesome tools. I personally hope that more Cypress users have tried Playwright after the event and vice versa.

This event was definitely fun, and while it’s interesting to compare code snippets and different abilities of the tools, we are well aware that these do not tell the whole story. A tester’s daily life is full of debugging, maintenance, complicated test design decisions, considering risks and effectiveness of automation… Merely looking at small pieces of code will not tell us how well these tools perform in real life. We’d love to take a look into these aspects as well, so we are planning a rematch with a slightly different format. Save the date of September 8th for the battle and stay tuned to this page for more info on the rematch. We’ll see who’s the winner next time! ?

The post Cypress vs Playwright: Let the Code Speak Recap appeared first on Automated Visual Testing | Applitools.

We’re approaching the end of May, which means we’re just a handful of weeks the midpoint of 2022 already. If you’re like me, you’re wondering where the year has gone. Maybe it has to do with life in the northeastern US where I live, where we’ve really just had our first week of warm weather. Didn’t winter just end?

As always, the year is flying by, and it can be hard to keep up with all the great videos or events you might have wanted to watch or attend. To help you out, we’ve rounded up some of our most popular test automation videos of 2022 so far. These are all top-notch workshops or webinars with test automation experts sharing their knowledge and their stories – you’ll definitely want to check them out.

Cross Browser Test Automation

Cross-browser testing is a well-known challenge to test automation practitioners. Luckily, Andy Knight, AKA the Automation Panda, is here to walk you through a modern approach to getting it done. Whether you use Cypress, Playwright, or are testing Storybook components, we have something for you.

Modern Cross Browser Testing with Cypress

For more, see this blog post: How to Run Cross Browser Tests with Cypress on All Browsers (plus bonus post specifically covering the live Q&A from this workshop).

Modern Cross Browser Testing in JavaScript Using Playwright

For more, see this blog post: Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser.

Modern Cross Browser Testing for Storybook Components

For more, see this blog post: Testing Storybook Components in Any Browser – Without Writing Any New Tests!

Test Automation with GitHub or Chrome DevTools

GitHub and Chrome DevTools are both incredibly popular with the developer and testing communities – odds are if you’re reading this you use one or both on a regular basis. We recently spoke with developer advocates Rizel Scarlett of GitHub and Jecelyn Yeen of Google as they explained how you can leverage these extremely popular tools to become a better tester and improve your own testing experience. Click through for more info about each video and get watching.

Make Testing Easy with GitHub

For more, see this blog post: Using GitHub Copilot to Automate Tests.

Automating Tests with Chrome DevTools Recorder

For more, see this blog post: Creating Your First Test With Google Chrome DevTools Recorder.

Test Automation Stories from Our Customers

When it comes to implementing and innovating around test automation, you’re never alone, even though it doesn’t always feel that way. Countless others are struggling with the same challenges that you are and coming up with solutions. Sometimes all it takes is hearing how someone else solved a similar problem to spark an idea or gain a better understanding of how to solve your own.

Accelerating Visual Testing

Nina Westenbrink, Software Engineer at a leading European telecom, talks about how the visual time to test the company’s design system was decreased and simplified, offering helpful tips and tricks along the way. Nina also speaks about her career as a woman in testing and how to empower women and overcome biases in software engineering.

Continuously Testing UX for Enterprise Platforms

Govind Ramachandran, Head of Testing and Quality Assurance for Asia Technology Services at Manulife Asia, discusses challenges around UI/UX testing for enterprise-wide digital programs. Check out his blueprint for continuous testing of the customer experience using Figma and Applitools.

This is just a taste of our favorite videos that we’ve shared with the community from 2022. What were yours? You can check out our full video library here, and let us know your own favorites @Applitools.

The post Our Best Test Automation Videos of 2022 (So Far) appeared first on Automated Visual Testing | Applitools.

On February 1, 2022, I gave a webinar entitled Cross Browser Testing with Cypress, in which I explained how to run Cypress tests against any browser using Applitools Visual AI and the Ultrafast Test Cloud. We had many attendees and lots of great questions – so many questions that we couldn’t answer them all during the event. In this article, I do my best to provide answers to as many remaining questions as possible.

Questions about the Webinar

Is there a recording for the webinar?

Yes, indeed! The recording is here.

Where is the repository for the example code?

Here it is: https://github.com/applitools/workshop-cbt-cypress. The repository also contains a full walkthrough in the WORKSHOP.md file.

Questions about Cypress

Can we run Cypress tests through the command line?

Yes! The npx cypress open command opens the Cypress browser window for launching tests, while the npx cypress run command launches tests purely from the command line. Use npx cypress run for Continuous Integration (CI) environments.

So, in a Cypress test case, we don’t need to create any driver object for opening a browser?

Correct! When you initialize a Cypress project, it sets up all the imports and references you need. Just call the cy object. To navigate to our first page, call cy.visit(…) and provide the URL as a string.

Can Cypress handle testing with iFrames?

Yes! Check out this cookbook recipe, Working with iFrames in Cypress.

How does cy.contains(…) work?

The cy.contains(…) call selects elements based on their text. For example, if a button has the text “Add Account”, then cy.contains(“Account”) would locate it. Check the Cypress docs for more detailed information.

Can Cypress access backend intranet APIs?

I haven’t done that myself, but it looks like there are ways to set up Windows Authentication and proxies with Cypress.

Questions about Applitools

How do I establish baseline snapshots?

The first time you take a visual snapshot with Applitools Eyes, it saves the snapshot as the baseline. The next time the same snapshot is taken, it is treated as a checkpoint and compared against the baseline.

Does Applitools Eyes fail a test if a piece of content changes on a page?

Every time Applitools Eyes detects a change, it asks the tester to decide if the change is good (“thumbs up”) or bad (“thumbs down”). Applitools enables testers to try different match levels for comparisons. For example, if you want to check for layout changes but ignore differences in text and color, you can use “layout” matching. Alternatively, if the text matters to you but layout changes don’t, you can use “content” matching. Applitools also enables testers to ignore regions of the snapshots. For example, a timestamp field will be different for each snapshot, so those could easily be ignored.

How do we save a new baseline snapshot if pages change during development?

When a tester marks a change as “good,” the new snapshot is automatically saved as the new baseline.

What happens if we have thousands of tests and developers change the UI? Will I need to modify thousands of baselines?

Hopefully not! Most UI changes are localized to certain pages or areas of an app. In that case, only those baselines would need updates. However, if the UI changes affect every screen, such as a theme change, then you might need to refresh all baselines. That isn’t as bad as it sounds: Applitools has AI-powered maintenance capabilities. When you accept one new snapshot as a baseline, Applitools will automatically scan all other changes in the current batch and accept similar changes. That way, you don’t need to grind through a thousand “thumbs-up” clicks. Alternatively, you could manually delete old baselines through the Applitools dashboard and rerun your tests to establish fresh ones. You could also establish regions to ignore on snapshots for things like headers or sidebars to mitigate the churn caused by cross-cutting changes.

Does Applitools Eyes wait for assets such as images, videos, and animations to load before taking snapshots?

No. The browser automation tool or the automation code you write must handle waiting.

(* Actually, there is a way when not using the Ultrafast Test Cloud. The classic SDKs include a parameter that you can set for controlling Eyes snapshot retries when there is no match.)

Can we accept some visual changes while rejecting others for one checkpoint?

Yes, you can set regions on a snapshot to use different match levels or to be ignored entirely.

Can we download the snapshot images from our tests?

Yes, you can download snapshot images from the Applitools Eyes dashboard.

Does Applitools offer any SDKs for visually testing Windows desktop apps?

Yes! Applitools offers SDKs for Windows CodedUI, Windows UFT, and Windows Apps. Applitools also works with Power Automate Desktop.

Does the Applitools Ultrafast Grid use real or emulated mobile devices?

Presently, it uses emulated mobile devices.

Can I publicly share my Applitools API key?

No, do NOT share your API key publicly! That should be kept secret. Don’t let others run their tests using your account!

Questions about Applitools with Cypress

How do I set up Applitools to work with Cypress?

Follow the Applitools Cypress Tutorial. You’ll need to:

1. Register an Applitools account.
2. Install the @applitools/eyes-cypress package.
3. Run npx eyes-init to set up Applitools Eyes.
4. Set the APPLITOOLS_API_KEY environment variable to your API key.

Cypress cannot locally run tests against Safari, Internet Explorer, or mobile browsers. Can Cypress tests check snapshots on these browsers in the Applitools Ultrafast Test Cloud?

Yes! Snapshots capture the whole page, not just pixelated images. Applitools can render snapshots using any browser configuration, even ones not natively supported by Cypress.

Can Applitools Eyes focus on specific elements instead of an entire page?

Yes! You can check a specific web element as a “region” of a page like this:

cy.eyesCheckWindow({
  target: 'region',
  selector: {
    type: 'css',
    selector: '.my-element'
  }
});

Can we run visual tests with Cypress using a free Applitools account?

Yes, but you will not be able to access all of Applitools’ features with a free account, and your test concurrency will be limited to 1.

Can we perform traditional assertions together with visual snapshots?

Sure! Visual testing eliminates the need for most traditional assertions, but sometimes, old-school assertions can be helpful for checking things like text formatting. Cypress uses Chai for assertions.

Questions about Testing

If a project has little-to-no test automation in place, should we start writing visual tests right away, or should we start with traditional functional tests and add visual tests later?

Visual tests are arguably easier to automate than traditional functional tests because they simplify assertions. Apply the 80/20 rule: start with a small “smoke” test suite that simply captures snapshots of different pages in your web app. Run that suite for every code change and see the value it delivers. Next, build on it by covering more interactions than navigation. Then, once those are doing well, try to automate more complicated behaviors. At that point, you might need some traditional assertions to complement visual checkpoints.

Can we compare snapshots from a staging environment to a production environment?

Yes, you can compare results across test environments as long as the snapshots have the same app, test, and tag names.

Can we schedule visual tests to run every day?

Certainly! Both Applitools and Cypress can integrate with any Continuous Integration (CI) system.

Does visual testing have any disadvantages when compared to traditional functional testing?

Presently, visual testing does not check specific text formatting, such as dates or currencies. You’ll need to use traditional assertions for that type of pattern matching. Nevertheless, you can use visual testing together with traditional techniques to automate the best functional tests possible.

How do we test UX things like heading levels, fonts, and text sizes?

If you take visual snapshots of pages, then Applitools Eyes will detect differences like these. You could also automate traditional assertions to verify specific attributes such as a specific heading number or font name, but those kinds of assertions tend to be brittle.

What IDE should we use for developing Cypress tests with Applitools?

Any JavaScript editor should work. Visual Studio Code and JetBrains WebStorm are popular choices.

What tool or framework should we use for API testing?

Cypress has built-in API support with the cy.request(...) method, making it easy to write end-to-end tests that interact with both the frontend and backend. However, if you want to automate tests purely for APIs, then you should probably use a tool other than Cypress. Postman is one of the most popular tools for API testing. If you want to stick to JavaScript, you could look into SuperTest and Nock.

Can we do load testing with cross-browser testing?

Load testing is the practice of adding different intensities of “load” to a system while running functional and performance tests. For web apps, “load” is typically a rate of requests (like 100 requests per second). As load increases, performance degrades, and functionality might start failing. You can do load testing with cross-browser testing, but keep in mind that any failures due to load would probably happen the same way for any browser. Load hits the backend, not the frontend. Repeating load tests for a multitude of different browser configurations may not be worthwhile.

The post Cross Browser Testing with Cypress Workshop Q&A appeared first on Automated Visual Testing | Applitools.

80+ questions and answers about one of the
hottest new testing frameworks around, Playwright!

Playwright is a relatively new open source cross-browser automation framework for end-to-end testing, developed and maintained by Microsoft. It tests across all modern browsers and is designed to be a framework that solves the needs of testing for today’s web apps.

Andrey Lushnikov, Principal Engineer at Microsoft, recently spoke with us in a webinar covering the inspiration for the development of Playwright, numerous features and efficiency tricks, as well as its cross-browser availability and support for multiple languages.

You can check out the full webinar, Playwright: A New Test Automation Framework for the Modern Web, at the link below.

There was a lively Q&A component to the webinar, and there wasn’t time to answer all the questions live. So the team at Microsoft looked at the remaining questions (all 80 of them!) and answered them. We’ve collected them all here for you so that, between the webinar and this extended Q&A, you can get all your Playwright questions answered in one place.

If you have a question that’s not answered here, you can also feel free to drop it on the Playwright Slack. Let us know about it @Applitools and we’ll be sure to add it to this post.

Extended Q&A: Your 80 Questions about Playwright

Click to skip to a section:

• Features Questions
• Integration Questions
• Migration Questions
• Platform Questions
• Availability Questions
• Comparison Questions
• Design Questions
• Support Questions

Features Questions

Q: Is there any built in reporting in Playwright?

A: No, reporting is typically used on the test runner level. You can use existing reporting tools along with the existing test runners with Playwright. Having said that, we are working on our own test runner for Node to address some of the existing e2e testing pain points.

Q: Can Playwright discover UI performance issues, for example, if a drop down list takes some time to load the mouse movement is not being followed on the items in the list, is that something that Playwright can help with discovering?

A: Here’s a quick overview of our performance story:

• On the high level, you can measure the time your calls took. We are working on a playwright trace that would have this information in a serialized and machine-processable form. That’ll happen ~v1.12, in a month.
• For Chromium, you can use tracing to drop this file into devtools and get in-depth performance story: https://playwright.dev/docs/next/api/class-browser#browserstarttracingpage-options this link is to the new doc, but the method has been there since forever.
• We expose resource timing https://playwright.dev/docs/api/class-request#requesttiming

I think Java will only catch up with chrome tracing API starting with v1.11. JavaScript and Python already have it.

Q: Does it support mocking? Like cy.intercpet? Can we get a coverage report from playwright? And can we wait like cy.wait()? And can we run a webkit test in container as part of pipeline?

A: The answer is yes – you can see more here: https://playwright.dev/docs/network#handle-requests

Q: Does Playwright provide access to windows displayed by the browser itself, not by web pages? (E.g. File Download/Upload windows)

A: Yes! Here’s more about Downloads / Uploads. Playwright exposes APIs to accept / decline / process downloads as well as for uploads. You can handle file upload dialog as well as set files into the input element.

Q: When a page is opened using browser context, is the page rendered in GUI or opened headlessly? In other words – what is the chance that rendered page will differ for the human when compared to test code?

A: There is no difference between headless and headful in terms of rendering.

Q: How does Playwright handle Salesforce iFrames?

A: If this is about new out-of-process iFrames in Chromium-based browsers, yes, we support them!

Q: How long does Playwright wait before giving up on waiting for an element?

A: It waits for a given timeout (can be local or global, 30 sec default). Check out this link for more: https://playwright.dev/docs/actionability

Q: Can Playwright identify elements by ID?

A: Yes, and many more – see more about our selectors here: https://playwright.dev/docs/selectors

Q: Can the tests run in parallel?

A: Yes, test runners like pytest and Jest will parallelize Playwright-based tests.

Q: Is it possible to show the cursor on a video recording?

A: Not at this moment, but please request it via GitHub.

Q: Are these assertions [demonstrated in the webinar] native to Playwright? or are they from Jest library?

A: Examples are shown using expect library (Jest). But that’s really up to you.

Q: How do you implement Data Driven methods? How do you get external data into script and drive multiple iterations with different data sets?

A: That’s more of a test runner question. Different test runners offer different APIs for that.

Q: Does Playwright offer a special functionality to handle canvas?

A: Nothing special for canvas. You can click with position to do free hand input there, but that’s not specific to canvas.

Q: What if I want to click a button that is disabled? What is Playwright’s wait strategy – does the test run into a timeout?

A: It’ll wait until the timeout: https://playwright.dev/docs/actionability.

Q: How do you run a set of test cases on different files?

A: That would be defined by the test runner you use: jest-playwright, cucumber, etc. We are working on a test runner for Playwright ourselves, which we will talk about a little later.

Q: Does an element need to be in the viewport to be active?

A: We will scroll to the element. And make sure it is not occluded by the sticky header.

Q: Do we get verbose logs saying what Playwright was trying to do before any test failure?

A: Yes you do – you can read more about verbose logs in Playwright here.

Q: Can we see what errors look like when a test fails?

A: They will be in the form of exceptions, but those exceptions contain verbose logs of activities.

Q: Does playwright provide some support to easily achieve multithreading?

A: Nothing special for multithreading. The test runner will shard testing processes to achieve concurrency.

Q: On Auto-waiting – how does this work? If, for example, a button gets enabled after a request comes in it needs to check this continuously? So you would need some kind of probing mechanisms that check every x seconds and timing out after y seconds?

A: Something like this. It will be rAF on log time scale probes, but happening on the browser side for perf reasons.

Q: In the demo [shown in the webinar], there is a waitForNavigate method. What exactly is implemented in waitForNavigate?

A: Put simply, waitForNavigation will wait for any navigation to happen.

Q: Why is waitForNavigation needed if Playwright has an auto-wait mechanism?

A: It is there for when you might need it, for example to make it absolutely clear that you are expecting a navigation. Say for example you’re filling a multi-page form and each page has a “next” button that brings you to the next page. If you want to click “next” on the first two pages, you might use waitForNavigation after the first click and before the second one to make sure you don’t click the same first button twice.

Q: What about automation animated elements, for instance pages with WebGL. Does Playwright support that kind of interaction?

A: As far as Playwright is concerned, WebGL is just a canvas element, you can interact with it using the mouse and script it using JavaScript. There is no specific support for WebGL vs regular canvas.

Q: Do you plan to include something for component testing at some point?

A: That’s a very interesting and hot topic. For those interested in this, please file an issue here for us to learn more about your use case.

Q: How easy is it to run parallel test against other tools like Cypress?

A: Test runners will do that seamlessly, you don’t need to do anything.

Q: Do you recommend Playwright for API testing?

A: That depends. We are seeing customers using Playwright for API and load testing when they need real user patterns, but in most of the cases you are better of with the dedicated API testing tools.

Q: Is it mandatory to use geolocation? If so, how do we get a position?

A: No, you can set it to test geo behavior, but you don’t need to.

Q: How does Playwright handle (or bypass) any SSL cert/protocol error?

A: We hook into the browsers and tell them to trust any TLS/certificates you wish. Check out the docs: https://playwright.dev/docs/api/class-browser?_highlight=ignorehttps#browsernewcontextoptions

Q: Does Playwright auto detect page ready?

A: There is no such thing as a page ready in general terms. We encourage you to keep things visual, you say page.click('text=Submit') and you don’t really need to wait for page to be ready, all you need to wait for is an interactable ‘Submit’ button. That makes the life of test writing so much easier.

Q: Can I wait for, intercept and parse HTTP requests (e.g. API calls) happening in the background, when page is loaded or is changed?

A: Absolutely: https://playwright.dev/docs/network#handle-requests

Q: Is multi-user testing (e.g. shared Excel access at the same time) with updating/data integration between two users possible?

A: I think I misunderstood this question during the live session. If two different users that have the same web page open, then absolutely, you can modify page on behalf of one user and see results in the other.

Q: Does Playwright by default work asynchronously? That is, do we not have to use async/await explicitly?

A: That depends on the language. Our Java is sync API, our Python can be both sync and async, our JavaScript is async. We try to make it fit the ecosystems and be idiomatic.

Q: How do you use Playwright Inspector when working in Visual Studio Code?

A: We would expect that you would typically use one or the other, depending on the use case.

Q: What’s an alternative of cy.click({force:true}) in Playwright ? Is it integrated in the Auto Wait feature?

A: It’s page.click({ force: true }). For more, see: https://playwright.dev/docs/api/class-page#pageclickselector-options

Q: Sometimes you need to wait for page to be fully loaded before taking a screenshot. For SPA, waitForNavigation and waitForLoadState do not work well. Is waitForSelector the best choice for this situation?

A: Yes, you nailed it, there is nothing better than a manual page ready criteria based on something visual.

Integration Questions

Q: What about reporting in Playwright? Does it have a CLI to aggregate reports from multiple runs? A reporting dashboard?

A: We don’t (yet!) ship a test runner with Playwright. We assume you will use your favorite runner and utilize its reporter.

Q: How does Playwright work with Continuous Integration?

A: You can run Playwright with your favorite test runner on your favorite CI: https://playwright.dev/docs/ci

Q: Can we integrate unit test libraries such as JUnit and TestNg with Playwright?

A: You can use Playwright with any of these unit testing frameworks! That’s in fact what most of our customers do.

Q: How easy it is to integrate parallel testing solutions like zalenium?

A: I am not familiar with zalenium, but parallel testing is a priority for us. We make sure all test runners can run Playwright in parallel.

Q: Can we integrate Playwright with visual tools?

A: Applitools Visual testing has a SDK for Playwright here. [Editor’s Note: We’ve also got a blog post with a great example of using Playwright with visual testing here]

Q: Will there be an integration with Cucumber?

A: There already is one – you can read more about it here: https://tally-b.medium.com/e2e-testing-with-cucumber-and-playwright-9584d3ef3360

Q: Will there be integration with a test management tool, such as TestRail?

A: That would need to happen on the higher level (test runner level).

Q: Does playwright come with a test explorer for Visual Studio Code?

A: That would need to happen on the test runner level too.

Q: Is there a GitHub Action for Playwright cloud execution?

A: Yes! You can read more about Playwright and GitHub Actions here: https://playwright.dev/docs/ci#github-actions

Q: Are Playwright assertions unique? Does it depend on a separate assertion library like Chai?

A: Playwright does not assert, you use it with your preferred library. You can use Expect or Chai, or anything else.

Migration Questions

Q: In our organization we use Jest-Puppeteer-Cucumber (TypeScript). Is there anything in Playwright that could help ease a possible migration from that tech stack?

A: There is no migration tool / guide, but the APIs for basic operations are very similar. You’ll end up deleting a bunch of waitForSelectors and then it’ll work as is.

Q: In terms of migration, would we need to rework our current Selenium WebDriver-based framework and redo a new one based on Playwright?

A: If you are using modern CI/CD, Playwright will work out of the box, for example using GitHub Actions. But if you have on-premise Selenium grid, you would need your current framework to be extended to allow pointing Playwright clients to your browser farm. Then you can gradually migrate.

Platform Questions

Q: Does Playwright have any plans for mobile apps? Native and Hybrid?

A: Experimental support for Android is available: https://playwright.dev/docs/api/class-android, both browser and WebViews. Nothing for iOS so far. Please star the issue on GitHub and share your requirements there.

Q: Is there an open source scale-out tool like Selenium Grid that will work with Playwright? If not, are there plans to build one?

A: Our APIs allow third parties to run services providing Selenium Grid-like features with Playwright. We are not commenting on whether we are building one.

Q: How does Playwright talk to the browser? With an HTTP request?

A: Playwright uses browser remote debugging protocols (e.g. CDP for Chromium) over the process pipe.

Q: Does Playwright support the Safari browser?

A: Yes, Safari is based on WebKit, one of the three rendering engines we support. Learn more at https://playwright.dev/docs/browsers.

Q: Are there any browser version restrictions for Playwright?

A: Playwright supports the latest versions of the browsers: their stable, beta channels for Chromium and Firefox-based browsers. Our WebKit corresponds to the Safari Technology Preview version. Learn more at https://playwright.dev/docs/browsers.

Q: How does Playwright manage multiple testing over multiple browser versions ? Ie, chrome 88, 87, etc.

A: Google Chrome and Microsoft Edge are trying hard to not allow downloads of the stale browsers and force-update their users. We keep Playwright in sync with the latest, but you can always fetch an older version of Playwright that works with say 86.

Q: Why is the Chromium browser launched in incognito mode?

A: For browser context isolation, we are using an internal concept called off-the-record-profile. It is rendered as incognito, because it is, well, off the record

Q: Is there any target to have a complete framework that covers web, mobile and desktop testing? And what about cloud testing?

A: We are only focusing on web everywhere at this point. Playwright does also have experimental Android and experimental Electron.

Availability Questions

Q: Is Playwright an open source or license-based library?

A: It’s open source.

Q: Does Playwright support Node.js and JavaScript right now?

A: Playwright supports JavaScript, TypeScript, Java, Python, Python asyncio, and C# in alpha. Learn more at https://playwright.dev/docs/languages

Q: Can you tell me more about the Playwright C# implementation?

A: You can find more here: https://github.com/microsoft/playwright-sharp. Again, it’s in alpha state as of now.

Q: Is there anyway to get the codegen to write ES6 JavaScript?

A: We only generate ES5 style code now, but it can be added upon request.

Q: Will codegen support TypeScript?

A: The ES5 code that we produce is valid TypeScript. We don’t have methods in the generated snippets, so there isn’t much room for types anyways.

Q: Does Playwright support JavaScript?

A: Yes, the very first version of Playwright was for JavaScript and TypeScript. Learn more at https://playwright.dev/docs/intro.

Q: Though it supports multiple languages, which is the recommended language for Playwright?

A: The one you are comfortable with. Our Node.js version was the first one, so if you don’t care – pick the JavaScript one.

Q: We have various frameworks like Cypress, TestCafe, Jest/Puppeteer, Playwright and many more. How should we decide which framework to pick because at the end of the day all are trying to achieve speed with respect to selenium?

A: I’d take a look at Playwright’s API (picking just one class from this API: https://playwright.dev/docs/api/class-page), the set of browsers we support and the test coverage we have for it. [Editor’s Note: You can also check out a great comparison of these tools here.]

Q: Any support for Dart in the future?

A: Please file a request on GitHub. If it gets enough stars, we will add it.

Q: Can you convert code snippets? For example JavaScript to Python and vice versa?

A: No, we can generate code in all languages, but not convert between languages.

Q: My question is related to building an enterprise framework for test automation using Java. How will Java Playwright be developed in future? Will it solve the limitations of Selenium for a JavaScript based application?

A: Playwright is already many times more powerful than WebDriver-based solutions, and Java version is on par with the rest of the Playwright family. Our primary goal is to keep resolving those limitations.

Q: If APIs are implemented in C#/Java would it be ok to use JavaScript for the test side for Playweight? Or would it make sense to run under same solution and implement in C#/Java?

A: That’s completely your call. Many people use JavaScript and Python for programs written in Java/C#. Language is a tool in your hands, you pick the one you feel more comfortable.

Q: What is better with Playwright? Jest or Folio?

A: Jest-playwright is great, but Folio will be absolutely awesome. We are very close to releasing it to public, please stay tuned. Don’t use Folio just yet, but it is coming really soon.

Q: Does Playwright release on a schedule that coincides with third party dependencies such as WebKit, Chrome upgrades, WebDrivers etc?

A: We release monthly.

Comparison Questions

Q: How does this differ from TestCafe?

A: The feature set is vastly different. For example, our page can do the following things: https://playwright.dev/docs/api/class-page

Q: There seems to be a movement to no-code automation. How does playwright stack up against those vendor products?

A: Playwright is a foundation. Its APIs bet on the script-based testing, but in reality it just makes so much possible when it comes to cross-browser testing. I can imagine a lot of third party no-code solutions emerging that would utilize Playwright as a foundation.

Q: How is Playwright different from other tools like Cypress?

A: Playwright supports all rendering engines, it has more capabilities (network, proxies, mobile, emulation, many more) because we hook into the browsers directly. We also have a traditional programming model (instead of chained API).

Design Questions

Q: Do you use an equivalent to Page Object Model? I need to try to avoid hard coded values for objects and other items.

A: Playwright script is a program, so you can treat it as such. As in any program, you don’t hard-code, but use variables, parameters, etc. You can do the same with Playwright.

Q: Is there any specific test automation design pattern you suggest to use with Playwright? Like Page Object Model was most popular with Selenium.

A: Yes the Page Object Model design pattern works well with Playwright, as do alternative design patterns. It’s really up to you. [Editor’s Note: We’ve got a great blog post demonstrating Page Object Model with Playwright]

Q: From an architecture standpoint, can we apply the Page Object Model, or Screenplay?

A: Playwright is just a library, you can build any programs on top of it using any design patterns.

Q: Can you please help me understand how to implement the Page Object Model using JavaScript, and how to generate an executable?

A: For Node.js, you don’t generate an executable, you typically write a program and then run it with ‘node script.js’. But you don’t need to use Node.js if that is not your preferred language, Playwright for Java / Python is just as great.

Support Questions

Q: Where is the best community support source for Playwright?

A: The best place to go is our Slack channel, which you can find here: https://aka.ms/playwright-slack

Q: Is there any forum for Playwright Java which can be referred to raise issues or queries?

A: Head over to the Playwright Slack and pick the -java channel. Or just file an issue to Playwright Java on GitHub.

Q: Who “owns” Playwright? Microsoft? Will it be supported for a long time?

A: Playwright is developed in the Developer Division at Microsoft. That’s the same people that brought Visual Studio, Visual Studio Code, .NET and Azure tools to you. Developers are our mission, our core business. It is not a side library that we did for fun, we are here to stay.

Q: How can we be confident this framework will last? Do you have a long-term support plan?

A: This is a core business for our division, we’d like to see more testing that can run in the cloud. Everything that runs in the cloud is great for Microsoft as well. You’ll actually see more Playwright-related announcements and deliverables coming from us.

Thank You

We wanted to take the chance again to say thank you to Andrew Lushnikov and Pavel Feldman of Microsoft for leading the Playwright webinar that inspired all these questions and providing all these answers. And of course, thanks to everyone who participated in the webinar and asked these thoughtful questions.

If you’re looking to do visual testing alongside Playwright, be sure to check out the Applitools SDK for Playwright here. Happy testing!

The post Your Favorite Questions about Playwright, Answered From The Applitools Team appeared first on Automated Visual Testing | Applitools.

We have heard a lot about Mobile Testing and how it differs from Web-based testing. However, do you know there is more to test on Mobile than just functionality and device-related testing?

Listen to Appium expert, Justin Ison, and Sr. Automation Architect, Anand Bagmar, in this in-depth session, where they shared hands-on real-world examples that are critical to include in your Mobile Testing.

Talking points include:

• Understand when you need to go beyond regular testing approaches – ex: when to use real devices vs emulators/simulators; when to consider on-field testing along with beta/staged releases of your apps
• Be aware of pitfalls – ex: insights from chaos/monkey testing, and ways to build observability into your app
• Learn techniques that reduce various risks along the Dev-Test-Release cycle – ex: accessibility and visual testing

Slide Deck

Full Recording

Additional Resources

• Easy Parallel Visual Testing with Appium — webinar with Appium Expert Jonathan Lipps
• Getting Started with Appium 2.0 Beta [post]
• Visual Testing with Applitools, Appium, and Amazon AWS Device Farm [post]

Mobile Testing Courses — Exclusive ONLY on Test Automation University

• Automated Visual Testing with Appium — course by Appium Expert Jonathan Lipps
• Android Test Automation with Espresso — course by Moataz Nabil
• Mobile Automation with Appium in JavaScript — course by Orane Findley

— HAPPY TESTING —

The post Stop Testing (Only) The Functionality of Your Mobile Apps! [webinar recording] appeared first on Automated Visual Testing | Applitools.

Watch this on-demand webinar to learn how the new GitHub Actions can help you build, test and deploy faster and easier!

Curious about GitHub Actions and how this feature can be used for test automation? So is test automation guru Angie Jones!

GitHub Actions makes it easy to automate all your software workflows, now with CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.

Watch this on-demand webinar, where Angie Jones chats with Brian Douglas — staff developer advocate at GitHub — about this exciting new offering and how it can be utilized for test automation.

Brian also showed a demo illustrating how easy it is to add automated tests to a project and kick them off with GitHub Actions!

Angie’s Slide Deck

https://slides.com/angiejones/github-actions

Full Webinar Recording

Additional GitHub Action Links Provided by Brian

• https://bdougie.live/
• https://github.com/bdougie/live
• https://lab.github.com/githubtraining/devops-with-github-actions
• https://youtube.com/ilikerobot
• https://github.com/bdougie
• https://docs.github.com/en/actions/reference/events-that-trigger-workflows
• https://github.com/mxschmitt/action-tmate
• https://github.com/nektos/act
• https://github.community/c/github-ecosystem/37
• https://github.com/github/roadmap/projects/1

Additional Resources and Reading Materials

• Angie’s Twitter
• Brian’s Twitter

• Top 10 Most Popular Free Test Automation Courses of 2020 — post by Angie Jones
• Selenium vs Cypress: The Rematch — post by Angie Jones
• Where To Learn Test Programming – July 2020 Edition (post)
• Start visually testing your app and site today, with your free Applitools account — Click here

— HAPPY TESTING —

The post Intro to GitHub Actions for Test Automation — with Angie Jones and Brian Douglas [webinar recording] appeared first on Automated Visual Testing | Applitools.

What is refactoring? Do we, as testers, need to perform such a task? Why and when? And especially, how can we do it easily?

Listen to Senior Automation Engineer Corina Pip as she explains how IntelliJ can help us with this refactoring task, by highlighting areas that need improvement, or by automatically improving them.

This is a very hands-on, practical session!

Corina provided plenty of code examples, as she demonstrated — step by step — what needs to be improved and how we can easily make it happen.

Talking points included:

• Why and when do we need to perform refactoring?
• How did we get to this and what can we do to avoid it in the future?
• A LOT of examples of how IntelliJ can help with this activity

Full Webinar Recording

Additional Resources

• Test Automation University exclusive — IntelliJ for Test Automation Engineers — 12-chapter course on the IntelliJ Code Editor by Corina Pip
• Top 10 Most Popular Free Test Automation Courses of 2020 — post by Angie Jones
• Selenium vs Cypress: The Rematch — post by Angie Jones
• Where To Learn Test Programming – July 2020 Edition (post)
• Start visually testing your app and site today, with your free Applitools account — Click here

— HAPPY TESTING —

The post Refactoring Made Easy with IntelliJ — Corina Pip [webinar recording] appeared first on Automated Visual Testing | Applitools.

Test automation is a tool that has been used for decades to assist us in ensuring quality in software development — but like any tool, there are many use cases, and it is up to the individual user to decide how the tool is implemented.

Alejandro Sanchez-Giraldo is a Test Innovation Manager @ Vodafone, and during his 13 years as a Quality Engineer and Advocate, he has seen many implementations of automation. Some created benefits and enabled teams to deliver scalable quality, compared to others that generated overheads and caused teams to be disjointed.

In this talk, Alejandro showcased 3 main approaches he has used to automate — which enabled teams to successfully deliver quality software: Celebrate Failure, Social Automation, and Sustainable Teams.

Alejandro discussed each method, showed examples from his own professional experience, and shared tips and best practices on how to ensure your team gets the most out of its automation tools and processes.

Alejandro’s Slidedeck

Full Webinar Recording

Additional Test Automation Resources

• Angie Jones Presents: “Automated Visual Testing: A Fast Path To Test Automation Success” — ace automated visual testing with this free 9-chapter online course, exclusive on Test Automation University!
• Maintaining Quality Across the Entire Dev-Test-Release Cycle — at Jira/ Atlassian and Pushpay — webinar recording
• Follow Alejandro on LinkedIn.
• Start visually testing your app and site today, with your free Applitools account — Click here

The post “Automate to Collaborate” – with Vodafone’s Test Innovation Manager: Alejandro Sanchez-Giraldo [webinar recording] appeared first on Automated Visual Testing | Applitools.

Two leading developers — from Jira/ Atlassian and Pushpay — shared their insights, tip, tricks, and best practices on how to maintain quality across the Dev-Test-Release cycle, without losing speed or coverage.

Talk 1: Reducing the Risk of Rapid Development and Continuous Delivery — by David Corbett (Director of Engineering @ Pushpay)
In this talk, David showed us what goes on under the hood of Pushpay’s development cycle.
He also talked about the ways in which Pushpay is empowering Dev and Test teams to be more autonomous, and prompting them to use advanced test automation tools & techniques, such as visual validation, in order to gain confidence in deploying many times each day.

Talk 2: Testing Hourglass at Jira Frontend — by Alexey Shpakov (Sr. Developer – Jira Frontend @ Atlassian)
We often hear people talk about the testing pyramid.
In Jira Frontend, we talk about testing hourglass — that means we expect our developers to be responsible for the whole lifecycle of the code — starting from creating tests and finishing with running a 24/7 on-call.
In this talk, Alexey did a deep-dive into the various types of testing they have in Jira Frontend, and discussed the various tools that allow them to deliver Jira to customers in a low-risk manner.

David Corbett — Reduced Risk with rapid Deployment and Continuous Delivery @ Pushpay — slide deck

Alexey Shpakov — Testing Hourglass at Jira Frontend — slide deck

Maintaining Quality at Jira/ Atlassian and Pushpay — Full Webinar Recording

Additional Resources

• Angie Jones Presents: “Automated Visual Testing: A Fast Path To Test Automation Success” — ace automated visual testing with this free 9-chapter online course, exclusive on Test Automation University!
• 2020 Most Popular Frontend Automation Testing Tools (post) — a data-centric view into the most popular tools for automated testing
• Start visually testing your app and site today, with your free Applitools account — Click here

— HAPPY TESTING —

The post Maintaining Quality Across the Entire Dev-Test-Release Cycle — at Jira/ Atlassian and Pushpay [webinar recording] appeared first on Automated Visual Testing | Applitools.

The Selenium WebDriver maintainers have been hard at work on a brand-new version: Selenium 4!

Some of the features are already available in alpha mode — which allows us the opportunity to try it out, provide feedback, and even pitch in with development.

Watch this webinar, where Angie Jones explored some of the most promising new capabilities of Selenium 4.

Angie also took for a test-drive some of its new features such as Relative Locators, Chrome Devtools Protocol (CDP), Window Management, and more.

You know Angie – so you know code samples were provided!

Angie’s Slide Deck

https://slides.com/angiejones/selenium4/fullscreen

Full Webinar Recording

Additional Selenium 4 Resources

• What’s New In Selenium 4? — post by Manoj Kumar
• Selenium 4 – New Features of Selenium WebDriver (Part 1) — post by Shama Ugale
• How to Install Selenium 4 — post by Shama Ugale
• Migrating to Selenium 4: Here’s What Has Changed — post by Shama Ugale

• Angie Jones Presents: “Automated Visual Testing: A Fast Path To Test Automation Success” — ace automated visual testing with this free 9-chapter online course, exclusive on Test Automation University!
• Test Automation University exclusive — Selenium WebDriver with Java — 14-chapter intro course by Angie Jones, to advanced Selenium WebDriver with Java.
• Start visually testing your app and site today, with your free Applitools account — Click here

— HAPPY TESTING —

The post Test Driving Selenium 4 – with Angie Jones [webinar recording] appeared first on Automated Visual Testing | Applitools.

Ace functional and visual component testing with Cypress.io and Applitools — watch this on-demand webinar with two engineering giants: Gleb Bahmutov (VP Engineering @ Cypress.io) and Gil Tayar (Sr. Architect @ Applitools).

Functional and visual testing work very well together for end-to-end frontend tests.

In this hands-on session, Gleb Bahmutov and Gil Tayar showed how to apply the same approach to the individual components — the building blocks of modern web apps.

Component testing ensures a thorough validation of all our components’ variants such as different inputs and states, edge-case network responses, and responsive widths.

Gleb and Gil used Cypress.io to drive the tests and check them functionally, while integrating the Applitools plugin to provide us with the capability to quickly and easily test them visually across all browsers and response widths. Watch this on-demand webinar to see the full power of the functional and visual component tests’ ability to speed up the development cycle.

Slide Deck (PDF)

Slide Deck — Online Slides (here, you can play the videos shown in the session directly from the browser)

https://cypress.slides.com/cypress-io/component-testing-with-cypress-and-applitools

Full On-demand Recording

Additional Materials & Resources Worth Checking Out:

• Check out Gil Tayar’s Introduction to Cypress.io Course — exclusive on Test Automation University: the leading free, online educational platform for all things Quality Engineering, with courses with world-leading instructors — Enroll and start improving your skills today.
• Gleb Bahmutov — VP of Engineering @ Cypress.io — check our Gleb’s blog, GitHub, Twitter, and conference presentations on Slide.com
• Cross-browser Testing With Cypress.io — post by Bilal Haidar
• 2020 Most Popular Front End Automation Testing Tools — post by industry leader Angie Jones
• Creating a Flawless User Experience, End-to-End, Functional to Visual – Practical Hands-on Session — on-demand webinar with Gleb Bahmutov and Gil Tayar
• Modern Cross Browser Testing with Cypress and Applitools — blog post by Marie Drake, Principal Test Automation Engineer @ News UK, and Cypress.io Ambassador
• Cypress vs Selenium WebDriver: Better, or just different? — post by Gil Tayar
• … And the webinar version of the post above: Cypress vs Selenium WebDriver: Better, Or Just Different? – on-demand webinar by Gil Tayar
• How Easy Is Cross Browser Testing? — post by Dan Iosif, winner of our Hackathon
• Can Automated Cross-Browser Testing Be Ultrafast? — blog post by Marie Drake, Principal Test Automation Engineer @ News UK, and Cypress.io Ambassador
• Cypress and Angular Testing with Applitools — post by Bilal Haidar
• Roadmap to Testing a Design System – on-demand webinar by Marie Drake
• Check out all the Cypress.io related content on the Applitools blog

• Start visually testing today — with your free Applitools account — open yours right here

— HAPPY TESTING —

The post Visual Component Testing with Cypress.io and Applitools [webinar recording] appeared first on Automated Visual Testing | Applitools.

Branch Insurance is disrupting the insurance industry by allowing customers to bundle home and car insurance in as little as 30 seconds.

What exactly is under the hood that helps a company like Branch to disrupt a well-established industry such as insurance?

In this webinar, Joe Emison — Co-founder and CEO of Branch Insurance — shared the technology he relies on. He also discussed in detail his approach to software quality, his requirements from technology partners, his tech-stack, and how AI has helped him on this journey.

Joe was joined by Erik Fogg — Co-Founder & CRO of ProdPerfect — and Daniel Levy — Sr. Director of Product Marketing at Applitools — as they demonstrated how Branch Insurance has achieved effortless, self-updating end-to-end testing — functional and visual — that gives them an unparalleled advantage of deploying faster at lower costs, all while making QA coverage one less thing for Joe’s team to worry about.

Slide Deck

Webinar Recording

Additional Resources

• Angie Jones Presents: “Automated Visual Testing: A Fast Path To Test Automation Success” — ace automated visual testing with this free 9-chapter online course by industry leader Angie Jones, exclusive on Test Automation University!
• Whitepaper — The Impact of Visual AI on Test Automation Report — Empirical data from 288 quality engineers automating tests with Cypress, Selenium, and Webdriver IO
• Ask 288 Of Your Peers About Visual AI [post]
• Visual UI Testing as an Aid to Functional Testing [post by Gil Tayar]
• Evolving Test Strategy to Meet Growing Business Challenges — Fiserv Use Case on PDF Testing [webinar recording]
• The Quality Maturity Rubric: Achieving Elite Quality Status – with Greg Sypolt, VP of Quality Assurance @ EverFi [webinar recording]
• Meeting CI/CD Requirements: Key Factors in Test Automation We Must Consider – webinar w/ Angie Jones and Jessica Deen [recording]
• Start visually testing today — with your free Applitools account — open yours right here
• Click here to learn more about ProdPerfect’s E2E Test Automation & Regression Testing Solution
• Check out Branch Insurance

— HAPPY TESTING —

The post Software Quality in the Service of Innovation in the Insurance Industry [webinar recording] appeared first on Automated Visual Testing | Applitools.