80+ questions and answers about one of the
hottest new testing frameworks around, Playwright!
Playwright is a relatively new open source cross-browser automation framework for end-to-end testing, developed and maintained by Microsoft. It tests across all modern browsers and is designed to be a framework that solves the needs of testing for today’s web apps.
Andrey Lushnikov, Principal Engineer at Microsoft, recently spoke with us in a webinar covering the inspiration for the development of Playwright, numerous features and efficiency tricks, as well as its cross-browser availability and support for multiple languages.
You can check out the full webinar, Playwright: A New Test Automation Framework for the Modern Web, at the link below.
There was a lively Q&A component to the webinar, and there wasn’t time to answer all the questions live. So the team at Microsoft looked at the remaining questions (all 80 of them!) and answered them. We’ve collected them all here for you so that, between the webinar and this extended Q&A, you can get all your Playwright questions answered in one place.
If you have a question that’s not answered here, you can also feel free to drop it on the Playwright Slack. Let us know about it @Applitools and we’ll be sure to add it to this post.
Extended Q&A: Your 80 Questions about Playwright
Click to skip to a section:
- Features Questions
- Integration Questions
- Migration Questions
- Platform Questions
- Availability Questions
- Comparison Questions
- Design Questions
- Support Questions
Features Questions
Q: Is there any built in reporting in Playwright?
A: No, reporting is typically used on the test runner level. You can use existing reporting tools along with the existing test runners with Playwright. Having said that, we are working on our own test runner for Node to address some of the existing e2e testing pain points.
Q: Can Playwright discover UI performance issues, for example, if a drop down list takes some time to load the mouse movement is not being followed on the items in the list, is that something that Playwright can help with discovering?
A: Here’s a quick overview of our performance story:
- On the high level, you can measure the time your calls took. We are working on a playwright trace that would have this information in a serialized and machine-processable form. That’ll happen ~v1.12, in a month.
- For Chromium, you can use tracing to drop this file into devtools and get in-depth performance story: https://playwright.dev/docs/next/api/class-browser#browserstarttracingpage-options this link is to the new doc, but the method has been there since forever.
- We expose resource timing https://playwright.dev/docs/api/class-request#requesttiming
I think Java will only catch up with chrome tracing API starting with v1.11. JavaScript and Python already have it.
Q: Does it support mocking? Like cy.intercpet? Can we get a coverage report from playwright? And can we wait like cy.wait()? And can we run a webkit test in container as part of pipeline?
A: The answer is yes – you can see more here: https://playwright.dev/docs/network#handle-requests
Q: Does Playwright provide access to windows displayed by the browser itself, not by web pages? (E.g. File Download/Upload windows)
A: Yes! Here’s more about Downloads / Uploads. Playwright exposes APIs to accept / decline / process downloads as well as for uploads. You can handle file upload dialog as well as set files into the input element.
Q: When a page is opened using browser context, is the page rendered in GUI or opened headlessly? In other words – what is the chance that rendered page will differ for the human when compared to test code?
A: There is no difference between headless and headful in terms of rendering.
Q: How does Playwright handle Salesforce iFrames?
A: If this is about new out-of-process iFrames in Chromium-based browsers, yes, we support them!
Q: How long does Playwright wait before giving up on waiting for an element?
A: It waits for a given timeout (can be local or global, 30 sec default). Check out this link for more: https://playwright.dev/docs/actionability
Q: Can Playwright identify elements by ID?
A: Yes, and many more – see more about our selectors here: https://playwright.dev/docs/selectors
Q: Can the tests run in parallel?
A: Yes, test runners like pytest and Jest will parallelize Playwright-based tests.
Q: Is it possible to show the cursor on a video recording?
A: Not at this moment, but please request it via GitHub.
Q: Are these assertions [demonstrated in the webinar] native to Playwright? or are they from Jest library?
A: Examples are shown using expect library (Jest). But that’s really up to you.
Q: How do you implement Data Driven methods? How do you get external data into script and drive multiple iterations with different data sets?
A: That’s more of a test runner question. Different test runners offer different APIs for that.
Q: Does Playwright offer a special functionality to handle canvas?
A: Nothing special for canvas. You can click with position to do free hand input there, but that’s not specific to canvas.
Q: What if I want to click a button that is disabled? What is Playwright’s wait strategy – does the test run into a timeout?
A: It’ll wait until the timeout: https://playwright.dev/docs/actionability.
Q: How do you run a set of test cases on different files?
A: That would be defined by the test runner you use: jest-playwright, cucumber, etc. We are working on a test runner for Playwright ourselves, which we will talk about a little later.
Q: Does an element need to be in the viewport to be active?
A: We will scroll to the element. And make sure it is not occluded by the sticky header.
Q: Do we get verbose logs saying what Playwright was trying to do before any test failure?
A: Yes you do – you can read more about verbose logs in Playwright here.
Q: Can we see what errors look like when a test fails?
A: They will be in the form of exceptions, but those exceptions contain verbose logs of activities.
Q: Does playwright provide some support to easily achieve multithreading?
A: Nothing special for multithreading. The test runner will shard testing processes to achieve concurrency.
Q: On Auto-waiting – how does this work? If, for example, a button gets enabled after a request comes in it needs to check this continuously? So you would need some kind of probing mechanisms that check every x seconds and timing out after y seconds?
A: Something like this. It will be rAF on log time scale probes, but happening on the browser side for perf reasons.
Q: In the demo [shown in the webinar], there is a waitForNavigate method. What exactly is implemented in waitForNavigate?
A: Put simply, waitForNavigation will wait for any navigation to happen.
Q: Why is waitForNavigation needed if Playwright has an auto-wait mechanism?
A: It is there for when you might need it, for example to make it absolutely clear that you are expecting a navigation. Say for example you’re filling a multi-page form and each page has a “next” button that brings you to the next page. If you want to click “next” on the first two pages, you might use waitForNavigation after the first click and before the second one to make sure you don’t click the same first button twice.
Q: What about automation animated elements, for instance pages with WebGL. Does Playwright support that kind of interaction?
A: As far as Playwright is concerned, WebGL is just a canvas element, you can interact with it using the mouse and script it using JavaScript. There is no specific support for WebGL vs regular canvas.
Q: Do you plan to include something for component testing at some point?
A: That’s a very interesting and hot topic. For those interested in this, please file an issue here for us to learn more about your use case.
Q: How easy is it to run parallel test against other tools like Cypress?
A: Test runners will do that seamlessly, you don’t need to do anything.
Q: Do you recommend Playwright for API testing?
A: That depends. We are seeing customers using Playwright for API and load testing when they need real user patterns, but in most of the cases you are better of with the dedicated API testing tools.
Q: Is it mandatory to use geolocation? If so, how do we get a position?
A: No, you can set it to test geo behavior, but you don’t need to.
Q: How does Playwright handle (or bypass) any SSL cert/protocol error?
A: We hook into the browsers and tell them to trust any TLS/certificates you wish. Check out the docs: https://playwright.dev/docs/api/class-browser?_highlight=ignorehttps#browsernewcontextoptions
Q: Does Playwright auto detect page ready?
A: There is no such thing as a page ready in general terms. We encourage you to keep things visual, you say page.click('text=Submit') and you don’t really need to wait for page to be ready, all you need to wait for is an interactable ‘Submit’ button. That makes the life of test writing so much easier.
Q: Can I wait for, intercept and parse HTTP requests (e.g. API calls) happening in the background, when page is loaded or is changed?
A: Absolutely: https://playwright.dev/docs/network#handle-requests
Q: Is multi-user testing (e.g. shared Excel access at the same time) with updating/data integration between two users possible?
A: I think I misunderstood this question during the live session. If two different users that have the same web page open, then absolutely, you can modify page on behalf of one user and see results in the other.
Q: Does Playwright by default work asynchronously? That is, do we not have to use async/await explicitly?
A: That depends on the language. Our Java is sync API, our Python can be both sync and async, our JavaScript is async. We try to make it fit the ecosystems and be idiomatic.
Q: How do you use Playwright Inspector when working in Visual Studio Code?
A: We would expect that you would typically use one or the other, depending on the use case.
Q: What’s an alternative of cy.click({force:true}) in Playwright ? Is it integrated in the Auto Wait feature?
A: It’s page.click({ force: true }). For more, see: https://playwright.dev/docs/api/class-page#pageclickselector-options
Q: Sometimes you need to wait for page to be fully loaded before taking a screenshot. For SPA, waitForNavigation and waitForLoadState do not work well. Is waitForSelector the best choice for this situation?
A: Yes, you nailed it, there is nothing better than a manual page ready criteria based on something visual.
Integration Questions
Q: What about reporting in Playwright? Does it have a CLI to aggregate reports from multiple runs? A reporting dashboard?
A: We don’t (yet!) ship a test runner with Playwright. We assume you will use your favorite runner and utilize its reporter.
Q: How does Playwright work with Continuous Integration?
A: You can run Playwright with your favorite test runner on your favorite CI: https://playwright.dev/docs/ci
Q: Can we integrate unit test libraries such as JUnit and TestNg with Playwright?
A: You can use Playwright with any of these unit testing frameworks! That’s in fact what most of our customers do.
Q: How easy it is to integrate parallel testing solutions like zalenium?
A: I am not familiar with zalenium, but parallel testing is a priority for us. We make sure all test runners can run Playwright in parallel.
Q: Can we integrate Playwright with visual tools?
A: Applitools Visual testing has a SDK for Playwright here. [Editor’s Note: We’ve also got a blog post with a great example of using Playwright with visual testing here]
Q: Will there be an integration with Cucumber?
A: There already is one – you can read more about it here: https://tally-b.medium.com/e2e-testing-with-cucumber-and-playwright-9584d3ef3360
Q: Will there be integration with a test management tool, such as TestRail?
A: That would need to happen on the higher level (test runner level).
Q: Does playwright come with a test explorer for Visual Studio Code?
A: That would need to happen on the test runner level too.
Q: Is there a GitHub Action for Playwright cloud execution?
A: Yes! You can read more about Playwright and GitHub Actions here: https://playwright.dev/docs/ci#github-actions
Q: Are Playwright assertions unique? Does it depend on a separate assertion library like Chai?
A: Playwright does not assert, you use it with your preferred library. You can use Expect or Chai, or anything else.
Migration Questions
Q: In our organization we use Jest-Puppeteer-Cucumber (TypeScript). Is there anything in Playwright that could help ease a possible migration from that tech stack?
A: There is no migration tool / guide, but the APIs for basic operations are very similar. You’ll end up deleting a bunch of waitForSelectors and then it’ll work as is.
Q: In terms of migration, would we need to rework our current Selenium WebDriver-based framework and redo a new one based on Playwright?
A: If you are using modern CI/CD, Playwright will work out of the box, for example using GitHub Actions. But if you have on-premise Selenium grid, you would need your current framework to be extended to allow pointing Playwright clients to your browser farm. Then you can gradually migrate.
Platform Questions
Q: Does Playwright have any plans for mobile apps? Native and Hybrid?
A: Experimental support for Android is available: https://playwright.dev/docs/api/class-android, both browser and WebViews. Nothing for iOS so far. Please star the issue on GitHub and share your requirements there.
Q: Is there an open source scale-out tool like Selenium Grid that will work with Playwright? If not, are there plans to build one?
A: Our APIs allow third parties to run services providing Selenium Grid-like features with Playwright. We are not commenting on whether we are building one.
Q: How does Playwright talk to the browser? With an HTTP request?
A: Playwright uses browser remote debugging protocols (e.g. CDP for Chromium) over the process pipe.
Q: Does Playwright support the Safari browser?
A: Yes, Safari is based on WebKit, one of the three rendering engines we support. Learn more at https://playwright.dev/docs/browsers.
Q: Are there any browser version restrictions for Playwright?
A: Playwright supports the latest versions of the browsers: their stable, beta channels for Chromium and Firefox-based browsers. Our WebKit corresponds to the Safari Technology Preview version. Learn more at https://playwright.dev/docs/browsers.
Q: How does Playwright manage multiple testing over multiple browser versions ? Ie, chrome 88, 87, etc.
A: Google Chrome and Microsoft Edge are trying hard to not allow downloads of the stale browsers and force-update their users. We keep Playwright in sync with the latest, but you can always fetch an older version of Playwright that works with say 86.
Q: Why is the Chromium browser launched in incognito mode?
A: For browser context isolation, we are using an internal concept called off-the-record-profile. It is rendered as incognito, because it is, well, off the record 🙂
Q: Is there any target to have a complete framework that covers web, mobile and desktop testing? And what about cloud testing?
A: We are only focusing on web everywhere at this point. Playwright does also have experimental Android and experimental Electron.
Availability Questions
Q: Is Playwright an open source or license-based library?
A: It’s open source.
Q: Does Playwright support Node.js and JavaScript right now?
A: Playwright supports JavaScript, TypeScript, Java, Python, Python asyncio, and C# in alpha. Learn more at https://playwright.dev/docs/languages
Q: Can you tell me more about the Playwright C# implementation?
A: You can find more here: https://github.com/microsoft/playwright-sharp. Again, it’s in alpha state as of now.
Q: Is there anyway to get the codegen to write ES6 JavaScript?
A: We only generate ES5 style code now, but it can be added upon request.
Q: Will codegen support TypeScript?
A: The ES5 code that we produce is valid TypeScript. We don’t have methods in the generated snippets, so there isn’t much room for types anyways.
Q: Does Playwright support JavaScript?
A: Yes, the very first version of Playwright was for JavaScript and TypeScript. Learn more at https://playwright.dev/docs/intro.
Q: Though it supports multiple languages, which is the recommended language for Playwright?
A: The one you are comfortable with. Our Node.js version was the first one, so if you don’t care – pick the JavaScript one.
Q: We have various frameworks like Cypress, TestCafe, Jest/Puppeteer, Playwright and many more. How should we decide which framework to pick because at the end of the day all are trying to achieve speed with respect to selenium?
A: I’d take a look at Playwright’s API (picking just one class from this API: https://playwright.dev/docs/api/class-page), the set of browsers we support and the test coverage we have for it. [Editor’s Note: You can also check out a great comparison of these tools here.]
Q: Any support for Dart in the future?
A: Please file a request on GitHub. If it gets enough stars, we will add it.
Q: Can you convert code snippets? For example JavaScript to Python and vice versa?
A: No, we can generate code in all languages, but not convert between languages.
Q: My question is related to building an enterprise framework for test automation using Java. How will Java Playwright be developed in future? Will it solve the limitations of Selenium for a JavaScript based application?
A: Playwright is already many times more powerful than WebDriver-based solutions, and Java version is on par with the rest of the Playwright family. Our primary goal is to keep resolving those limitations.
Q: If APIs are implemented in C#/Java would it be ok to use JavaScript for the test side for Playweight? Or would it make sense to run under same solution and implement in C#/Java?
A: That’s completely your call. Many people use JavaScript and Python for programs written in Java/C#. Language is a tool in your hands, you pick the one you feel more comfortable.
Q: What is better with Playwright? Jest or Folio?
A: Jest-playwright is great, but Folio will be absolutely awesome. We are very close to releasing it to public, please stay tuned. Don’t use Folio just yet, but it is coming really soon.
Q: Does Playwright release on a schedule that coincides with third party dependencies such as WebKit, Chrome upgrades, WebDrivers etc?
A: We release monthly.
Comparison Questions
Q: How does this differ from TestCafe?
A: The feature set is vastly different. For example, our page can do the following things: https://playwright.dev/docs/api/class-page
Q: There seems to be a movement to no-code automation. How does playwright stack up against those vendor products?
A: Playwright is a foundation. Its APIs bet on the script-based testing, but in reality it just makes so much possible when it comes to cross-browser testing. I can imagine a lot of third party no-code solutions emerging that would utilize Playwright as a foundation.
Q: How is Playwright different from other tools like Cypress?
A: Playwright supports all rendering engines, it has more capabilities (network, proxies, mobile, emulation, many more) because we hook into the browsers directly. We also have a traditional programming model (instead of chained API).
Design Questions
Q: Do you use an equivalent to Page Object Model? I need to try to avoid hard coded values for objects and other items.
A: Playwright script is a program, so you can treat it as such. As in any program, you don’t hard-code, but use variables, parameters, etc. You can do the same with Playwright.
Q: Is there any specific test automation design pattern you suggest to use with Playwright? Like Page Object Model was most popular with Selenium.
A: Yes the Page Object Model design pattern works well with Playwright, as do alternative design patterns. It’s really up to you. [Editor’s Note: We’ve got a great blog post demonstrating Page Object Model with Playwright]
Q: From an architecture standpoint, can we apply the Page Object Model, or Screenplay?
A: Playwright is just a library, you can build any programs on top of it using any design patterns.
Q: Can you please help me understand how to implement the Page Object Model using JavaScript, and how to generate an executable?
A: For Node.js, you don’t generate an executable, you typically write a program and then run it with ‘node script.js’. But you don’t need to use Node.js if that is not your preferred language, Playwright for Java / Python is just as great.
Support Questions
Q: Where is the best community support source for Playwright?
A: The best place to go is our Slack channel, which you can find here: https://aka.ms/playwright-slack
Q: Is there any forum for Playwright Java which can be referred to raise issues or queries?
A: Head over to the Playwright Slack and pick the -java channel. Or just file an issue to Playwright Java on GitHub.
Q: Who “owns” Playwright? Microsoft? Will it be supported for a long time?
A: Playwright is developed in the Developer Division at Microsoft. That’s the same people that brought Visual Studio, Visual Studio Code, .NET and Azure tools to you. Developers are our mission, our core business. It is not a side library that we did for fun, we are here to stay.
Q: How can we be confident this framework will last? Do you have a long-term support plan?
A: This is a core business for our division, we’d like to see more testing that can run in the cloud. Everything that runs in the cloud is great for Microsoft as well. You’ll actually see more Playwright-related announcements and deliverables coming from us.
Thank You
We wanted to take the chance again to say thank you to Andrew Lushnikov and Pavel Feldman of Microsoft for leading the Playwright webinar that inspired all these questions and providing all these answers. And of course, thanks to everyone who participated in the webinar and asked these thoughtful questions.
If you’re looking to do visual testing alongside Playwright, be sure to check out the Applitools SDK for Playwright here. Happy testing!
