Let the Engineers Speak! Part 2: The Biggest Challenges

Carter Capocaccia (Cypress):

So I think one of the biggest challenges that I personally faced was that a lot of the times – I think initially when you’re starting out with automation – the tech problems that you face are where all the focus gets put on. And people lose sight that there’s an entire component to process around this. Meaning that, well, once you’ve written, you know, your hundred tests, you’ve got your architecture stable, you’ve figured out how to seed your database and correct your APIs into all the stuff that you need to do to make your tests run. Well, what do you do with those results? How does it actually impact your app? And I think a lot of people lose sight of that.

Of course, with the advent of CI/CD pipelines, it becomes a little bit easier because now you can basically prevent merges or prevent releases based on the results of these tests. But I think that gets forgotten until we’re way down the line. So, you know, you’ve written all your tests and you come back to your dev teams and you say, “Hey, I’m gonna start failing your pipelines now, and you’re not gonna be able to merge, you’re not gonna be able to to deploy.” And then you immediately have resentment against this process, right? So I think maybe the marketing of why you’re doing what you’re doing, how it’s gonna impact the process, and what is the positive result that comes out of this is equally as important as tackling those technical hurdles.

Tally Barak (Playwright):

Yeah I think I will go somehow along those lines. Besides some technical difficulties that I already shared about the shadow DOM and trivial actions like dragging, again, some of the things that we encountered. The biggest challenge was creating a culture of testing in the organization, making sure that people take care of the test. Because of some flakiness, we didn’t want to prevent merges. I mean, [for] a long time, we built more and more into that, but at the beginning it was like, “People, this is your helper. This is something to help you write better code.” And it didn’t come very easily, I would say.

Gleb Bahmutov (Cypress):

Can I comment on this slide [of the] poll? I see selectors as one of the challenges and it’s [a] pretty big one, apparently, for a lot of people. I just want to say what we’ve done at Mercari US. So, sometimes [when] you’re writing an end-to-end test, you come to a part of a page and there [are] some dynamic classes, no good selector specific for testing. So what we do in our test [is] we select however we can – by text, by position, you name it. But we do add a to-do comment. So in our test repo, we have lots of to-do [comments] like “needs better test selector”.

And we have [an] automated script that actually counts it and puts it in a rhythm. So we know how many to-dos for selectors we have total. And one of our objectives as a team is to drive a number down. So we [are] adding selectors and it’s kind of a good small task to do, but having good selectors is a key. So don’t just kind of assume someone else will add it for you. Have it as one of your automated priorities.

Andy Knight (moderator):

That’s really cool to actually have it kind of scan through and kind of pinpoint. So it’s almost like you’re self-linting. Wow, that’s awesome. If I may ask another question off of this topic of the challenges specifically around selectors. What other challenges have y’all faced with selectors or innovative solutions to try to get on top of them? Like, does anybody use smart selectors or visual selectors or anything at all?

Carter Capocaccia (Cypress):

Can I share my really staunch opinion on this? I personally don’t allow merge requests that have XPath selectors in them. I disallow it. It is a forbidden practice at our organization. We actually start with – we have a selector hierarchy that we try and follow. And I believe it’s Testing Library. And know it’s a really generic name for everybody in this room, but genuinely, it’s called Testing Library. And they have a selector pattern basically as a hierarchical representation based. I think it says, you know, accessibility selectors, and then at the very bottom, it even goes down to classes and then IDs. But it starts with accessibility selectors, and then content and then, you know, some other pieces that you should go through. And nowhere in that list is XPath.

I know that a lot of folks feel like XPaths are necessary, but I think that you’ve gotta find a way around them. I’m sorry if that hurts you there, Pandy. But yeah, in my opinion, I think XPaths are sometimes used as a crutch. And in today’s world where you have components that are conditionally rendered or potentially even, like for situations that I’m working in, it depends on what part of the country you’re in or what part of the world you’re in. Your page may render it completely differently than another. So therefore I cannot rely on a DOM structure to not be shifting at a moment’s notice, or there may be a CMS tied into this so they can add a component to the page. It’s gonna completely change the DOM structure. So we have to do better than that and, you know, tests more and more nowadays are being kept right alongside the application code.

And I think that, speaking back to what I was saying about kind of changing the process around what you do. Well, if I’m a test engineer, if I really can’t get to a DOM element any other way, why can’t I go into that component and add a data test ID attribute? I should have that ability as a test engineer, just like a UI dev should have the ability to come into my test and modify that. So that’s I think a way you can kind of shift culture when you even think about something as simple as DOM selectors. But yeah, the long-winded answer to say, I personally think that if you want to have maintainable selectors, avoid XPaths.

Jose Morales (WebdriverIO):

I would like to share my opinion in two ways. One is the challenges that we have in Domino’s, and another one is the selectors.

The challenge is the maintenance of the test cases. And basically what we are following is a good process. And with a good process, I mean we have a team that is reviewing the executions and we do analysis about the results. We figured out if we need to fix the test cases, we need to change the test cases because the feature is now different – or what is the root cause of the problem, right? We are following good practices in terms of development, like code reviews, pull requests. We’re doing peer programming. If some automation engineer is facing some kind of challenging problem/blocker, we have working sessions they requested for those working sessions. And we work together, right, as a team to figure out what’s the problem, what [are] the possible solutions that we can do. And it’s very active.

And commenting about the selectors, we encourage to use the ID. And in Domino’s we have a custom ID is the UUID that all the web developers add that particular identifier for us to use. So there is a close communication between the developers and the automation engineers to work together as a team, how we can help to [better test] our product. That’s another thing that we normally do. And respect as much as possible the page-object design pattern. That is what we are doing in Domino’s.

Tally Barak (Playwright):

The slide that was showing actually three items that were shining there, which is flakiness, maintainability, and selectors. And I think they’re all related to some extent. Because if your selectors are too accurate, like XPath with all the DOM structure, then you have a maintainability problem and you might have flakiness. And I have to say, if you are talking about the tools, this is probably the one thing that Playwright has solved in just a way that makes writing tests really easy. I was hearing the other people here and they went like, “Oh, we already forgot about this problem after working with two and a half years.” We used to work with XPaths because that was the only thing that could pierce the shadow DOM.

Now we are working with plain selectors. We’re keeping them to the minimum and mostly the user-facing selectors. So things like text usually. Maybe something like a header component and then a text. And we will know how to address the very exact selector. And that reduces the maintenance because this is the last part that is likely to change. And because of the way Playwright is working with the auto waiting, it’ll just wait, we don’t need to put some time out. It’ll know to wait until a certain time out for the component to show up.

You can write very small and very concise selectors. And the one other thing is that they have the strict mode, which means that if it finds two elements that correspond to the same selector, it’ll break the test. And one of the problems that we’ve seen previously is that it would go with the first element. So it might click a different button or something else, and the test will continue and it’ll only fade like three steps or five steps ahead. And this is really hard to understand what is the problem. Playwright here will just tell you. Of course, you can change if you want it or not. It’s an option. But if you do that, you get a lot less flakiness in your test because the first time it’ll encounter the element, it’ll tell you, okay, “I have two here. You’re probably on the wrong path. Go ahead and fix your test.” So all these three things, the maintenance, the flakiness, the selectors is one of the, I think, the greatest advantages of Playwright in these terms.

Andy Knight (moderator):

Awesome. So I know we’ve spent a lot of time talking about selectors and DOM and modeling the page. What design patterns do y’all use for or within your test projects? I know, I think it was Jose mentioned like they use page objects and they stick to it very rigidly – or maybe rigidly is not the best word, but they stick with page objects. Are y’all using page objects or are y’all using Screenplay? I know Cypress has those action or command methods, I forget. What are y’all using to help make your interactions using those selectors better?

Steve Hernandez (Selenium):

Screenplay.

Gleb Bahmutov (Cypress):

We don’t try to make the test super dry, right? Eventually, if we see a duplication of code, we will write a custom command. For example, we have custom commands for selecting by data test id. So it’s really easy by data test ID or ARIA label, we use custom commands. We almost never write page objects. We prefer little reusable functions, right? We need to create a user. Here’s a function, creates a user for you.

There is one advantage, though, because I like writing Cypress plugins, we can abstract something that is really specific and just move it to [a] third party public open source plugin, but we’ll reuse in our project. So we’ve been doing that a lot. I just looked, we have more than 20 Cypress plugins with our test report, but 17 out of them is what I’ve written. So, we take that abstraction and we make it and just have it by itself so it becomes more powerful and reusable. But small functions and custom commands for us.

Tally Barak (Playwright):

We are working with Cucumber. So this is basically our structure. Each step has the automation behind the scene, and then obviously you can reuse because the step is reusable in any of your tests. And we do try to build them sometimes a bit more, you know, a sophisticated kind of code, so you can reuse them in different places.

Jose Morales (WebdriverIO):

[About] using other design patterns – another thing that really helped us. And I saw a very interesting question about how do you integrate a backend with a JavaScript frontend? And that’s a very interesting question because for us, we try to consume APIs as well. So for example, in Domino’s, we have an application that is an interface for configuration for our stores. So every manager in our stores can configure certain parameters for the store, right? Like the store phone number. So they can go to a specific website and then change the store’s number, for example. So if you think about the testing perspective, you can go to the UI and change a particular store number and then hit an endpoint to refresh the backend for that particular store configuration.

And then you go to another backend to hit the store profile and then get the store profile with a request and read the JSON back, right? So you don’t need to interact with the UI all the time, but you can also integrate with the backend as much as possible. So that’s a very clean strategy, because it helps you to test faster and test better and reduce the flakiness.

And another thing that I recommend is to use the plugins, right? So explore the plugins that you can get in the industry. For example, in the backend and if you are interested in measuring the performance, you can use Lighthouse with WebdriverIO. So you can – with the standards in the industry – measure what is the performance for your web application, right? So we don’t need to reinvent the wheel. And that’s my recommendation.

Andy Knight (moderator):

Awesome. Awesome. So I know, Steve, right off the bat, you had mentioned the Screenplay pattern. I wanted to come back to that. What have you found has been good about using Screenplay over something like raw calls or page objects or something else?

Steve Hernandez (Selenium):

Personally, I haven’t used a lot of raw calls. I’ve had the luxury of using Boa Constrictor. It kind of wraps the Selenium stuff in a warm hug, and it handles a lot of the timing issues that you might run into. Boa Constrictor has a lot of built-in – I hear you guys talk about plug-ins – we have a lot of built-in interactions, they’re called. So they can be questions that you’ll ask of an API endpoint. They can be questions that you’ll ask of the page in the form of, “I’m gonna scrape this column here that I use over and over to compare to something in the future”.

So we’ll have our Gherkin steps and essentially backing each of those step definitions are these different reusable components or questions or interactions. That works very well with the level of parallel parallelism we’re doing. You know, if you’re firing off 50 threads at a time or 80 or more, we need stuff to be fast, not step on each other’s toes. So we write the interactions in a way where they’re very atomic and they’re not going to affect something else that’s happening on another thread. As far as page objects, I haven’t used those a whole lot.