Let the Engineers Speak! Part 4: Changing Frameworks

Andy Knight (moderator):

So I’m going to ask another popcorn question here. And this one again comes with an audience poll. So if you’re in the audience, hop over to the polls and take a look here. Have you ever considered switching your test tool or framework? And if so, why? If so, why not?

Carter Capocaccia (Cypress):

Yeah, we considered it and did it. You know, I think you have to acknowledge it at a certain point that maybe what you’re doing isn’t working or the tool doesn’t fit your application or your architecture. And it’s a really hard decision to make, right? Because typically, by the time you get to that point, you’ve invested tens, hundreds, if not thousands of hours into a framework or an architecture. And making that switch is not an easy task.

Andy Knight (moderator):

What did you switch from and to?

Carter Capocaccia (Cypress):

Yeah, so [Selenium] WebDriver to Cypress. And you can already kind of see that architecturally they’re very, very different. But yeah, it was not an easy task, not an easy undertaking, but I think we had to admit to ourselves that WebDriver did not fit our needs. It didn’t operate in the way we wanted it to operate. At least for us, it was set up in a way that did not match our development cycles, did not match our UI devs’ patterns and processes.

So when we talk about, you know, kind of making a team of test engineers have additional help, well, one way to do that is [to align] your test automation practices with your UI dev practices. And so if we can have a JavaScript test framework that runs in a web app, that acts like a web app, that feels like a web app, well, we’ve now just added an entire team of UI developers that we’ve kind of made them testers, but we haven’t told them that yet. That’s kind of how that works. You adopt these people by just aligning with their tooling and all of a sudden they realize like, hey, I can do this too.

Andy Knight (moderator):

Mm-hmm. So I want to ask another question. Since you’re saying you used Selenium WebDriver before, in what language did you automate your test? Was that in JavaScript, just like first test, or was that in Java or Python or something else?

Carter Capocaccia (Cypress):

Yep, so it’s JavaScript. But when we look [at the] architecture of the app, right? So the architecture of WebDriver is very different than Cypress. [With] Cypress, you write a JavaScript function just like you write any other function. And most of the time, it’s going to operate just like you expect it to. I think that’s where, instead of them having to learn a WebDriver framework and what the execution order of things was and how the configurations were, Cypress is just pretty apparent in how it functions. When you look at like, well, how is the Cypress test executed? It uses Mocha under the hood as the test runner. So that’s just functions that are just kind of inside of describe and context and it blocks. So I think anybody that doesn’t know anything about testing but is a JavaScript developer can hop right into Cypress, look at it, and say, yeah, I think I know what’s going on here. And probably within the first 5 to 10 minutes, write an assertion or write a selector.

Andy Knight (moderator):

Awesome. That’s really cool. And so was that something that you and your team or org did recently? Was this like a couple years ago that you made this transition?

Carter Capocaccia (Cypress):

You know, I think I’d be lying to you if I told you the transition ever ends, right? I think it starts and you get to a point where you’ve scaled and you’ve gotten to where you’ve written a ton of tests on it, and then that’s great. But of course, it’s kind of the bell curve of testing. Like it starts really, really slow because you’re moving off the old framework, then you start to scale really quickly. You peak, and then at that point, you have these kind of outliers on the other side of the bell that are just running legacy applications or they just don’t want to move off of it. Or for whatever reason, they can’t move off of it. Let’s say they’re running like Node 10 still. It’s like okay, well, maybe we don’t want to invest the time there to move them off of it.

Yeah. So it’s a long journey to do a task like this. So it’s never done. By the time we’re finished, you know, there’s gonna be a new testing tool or Cypress will have a new version or something else will be new in there. You know, pipelines will [be through] some AI network at this point. Who knows? But I’m sure we’ll have to start the next task.

Andy Knight (moderator):

Sure. So how about others in our group of panelist speakers here? Have y’all ever transitioned from one framework or tool to another? Or in your current project, did you establish it and you were just super happy with what you chose? And tell us the reasons why.

Tally Barak (Playwright):

Yes. So, yeah, we also switched from WebdriverIO as I said at the beginning. We switched to Playwright, obviously. It was not that painful, to be honest, maybe because we kept the Cucumber side. So we had [fewer] scenarios than what we have today. So it was less to move. I think it went over a period [of] a few months. Unlike you, Carter, we actually moved completely away from WebdriverIO. There is no WebdriverIO, everything is with Playwright. But again, probably a smaller organization, so we were able to do that. And are we happy? I mean, [we are] thankful every single day for this switch.

Gleb Bahmutov (Cypress):

I can comment. At Mercari, we did switch from WebDriver in Python and mabl. So both were because the maintenance of tests and debugging of failed tests was [a] very hard problem, and everyone agreed that there is a problem. The transition was, again, faster than we expected, just because the previous tests did not work very well. So at some point we were like, well, we have 50 failing daily tests, right? It’s not like we can keep them. It’s not a sunken cost. It’s a cost of not working. So we have to recreate the test. But it was not that bad. So for us, the cost of debugging and maintaining the previous tool just became overwhelming, and we decided, let’s just do something else.

Jose Morales (WebdriverIO):

In my experience, we were using Selenium with JavaScript as a previous framework, and then we moved to WebdriverIO with JavaScript and Applitools. Because in Domino’s, we have different kind of products. We have another product that is called Next Generation of the stores. We’re using a reactive application there. This is also a website application. And we’re evaluating other tools for automation such as UiPath. And one thing that I like about UiPath is [it’s] a low-coding solution. That means you don’t need to really be a very proficient developer in order to create the scenarios or automate with UiPath, because it only [requires] drags and drops [to] create the scenarios. And it’s very easy to automate. And in the top of that, with UiPath, it’s a very enterprise solution.

We have different kinds of components like an orchestrator, we have a test manager, we have integration with Jenkins. We have robots that execute our test cases. So it’s an enterprise solution that we implemented for that particular project. And we have the same results as WebdriverIO with Applitools. We’re happy with the two solutions – the two frameworks – and we’re considering to remove WebdriverIO and Applitools and use UiPath for the main website for Domino’s. So I think in the future, the low-coding solution for automation is a way to go in the future. So definitely UiPath is a good candidate for us for our automation in the website.

Andy Knight (moderator):

How about you, Steve? I know you’re right now on a Selenium-based test project. Have you and your team thought about either moving to something else, or do you feel pretty good about the whole Selenium-ness?

Steve Hernandez (Selenium):

I think clearly there appears to be major speed-ups that you get from the JavaScript-based solutions. And I know you’ve evangelized a little bit about Playwright and then I’m hearing good things from others. I mean, we have a mature solution. We have 98% pass rates. You know, some of the limitation is more what our application can handle when we blast it with tests. I don’t know if your developers after each local build are tests running locally using some of these JavaScripts solutions. But I think one of the things that is appealing is the speed. And with Boa Constrictor 3, the roadmap is to – well, it has been modularized. So you can bring your own Rusty API client or potentially swap in Selenium and swap in something like a module for Playwright. So we would get that speed up but keep the same API that we’re using. So we’re toying with it, but if we can pull that off in Boa Constrictor, that would take care of that for us.

Andy Knight (moderator):

Yeah, man, definitely. So, I mean, it sounds like a big trend from a lot of folks has been, you know, step away from things like Selenium WebDriver to [a] more modern framework. You know, because Selenium WebDriver is pretty much just a low-level browser automator, which is still great. It’s still fine. It’s still awesome. But I mean, one of the things I personally love about Playwright – as well as things like Cypress – is that it gives you that nice feature richness around it to not only help you run your tests, but also help develop your tests.