Taking the next steps

Earlier this month, Applitools hosted a webinar, Let the Engineers Speak: Selectors, where testing experts discussed one of the most common pain points that pretty much anyone who’s ever done web UI testing has felt. The first article in our two-part series defined our terms and challenges of locating web elements using selectors, as well as recapped Christian’s WebDriverIO selectors tutorial and WebdriverIO Q&A. Be sure to read that article first to help set context.

Introducing our experts

I’m Pandy Knight, the Automation Panda. I moderated our two testing experts in our discussion of selectors:

• Christian Bromann, WebdriverIO
• Filip Hric, Cypress

Locating web elements with Cypress

Filip walked us through selectors in Cypress, starting with the basics and using Trello as an example app.

Note: All the following text in this section is based on Filip’s part of the webinar. Filip’s repository is available on GitHub.

When talking about selectors in web applications, we are trying to target an HTML element. Cypress has two basic commands that can help with selecting these elements: cy.get and cy.contains. In Filip’s example demo, he has VS Code on the left side of his screen and Cypress running in graphic user interface mode (open mode) on the right side.

The example test uses different kinds of selector strategies, with each approach using either the get command or the contains command.

Locating elements by class, ID, or attribute

The first first approach calls cy.get(‘h2’) using the H2 tag. In the Cypress window, if you hover over the get h2 command, it will highlight the selector that has been selected.

If you’re struggling to find the right selector in Cypress, there’s this really nice tool that basically works like the inspect element tool, but you don’t have to open the dev tools if you don’t want to.

In our other three approaches with the get command, we are using the class, the ID, and the attribute, respectively:

  cy.get('h2')
  cy.get('.board') // class
  cy.get('#board-1') // id
  cy.get('[data-cy=board-item]') // attribute

The syntax for the get commands is basically just CSS selectors. If you are selecting an element with a class of board, you need to prefix the class with a dot. You can even write more complex selectors, like adding an h2 tag inside the element that has the class board like cy.get(‘.board > h2’). The options are endless. If you have ever worked with CSS, you know that you can pretty much target any element you like.

Locating elements by text

Another strategy that we have in Cypress is selecting elements by text. This approach may not always work, but in cases like a login, sign up, sent, or okay button, these usually need to have specific text. To select an element using text, we use the contains command. The contains command will only select one element, and it’s actually going to look for elements within some context.

The example uses two different texts to search: cy.contains(‘My Shopping’) and cy.contains(‘My’). On the Trello board page, ‘My Shopping’ appears once and ‘My’ appears twice. The contains call using ‘My’ will return with the first element that has the text ‘My’ on the page, which is ‘My Board’ in the example. So that’s something to watch out for. If you want to be more specific with the kind of element you want to select, you can actually combine these two approaches, selecting a CSS element and specifying the text which you want to find. For example, cy.contains(‘.board’, ‘My’) would return the correct element.

  cy.contains('My Shopping') // text
  cy.contains('My') // find the first one
  cy.contains('.board', 'My') // specify element to find

Locating elements using XPath

There are other selector strategies like using XPath. Cypress has an official plugin for XPath. If you install that, you will get the ability to select elements using XPath. You can use XPaths, but they may be harder to read. For example, cy.xpath(‘(//div[contains(@class, “board”)])[1]’) does the same thing as cy.get(‘board’).eq(0).

// Filter an element by index
cy.xpath('(//div(contains(@class, "board") ]) [1]')

// Select an element containing a specific child element
cy.xpath('//div [contains(@class, "list")] [.//div[contains(@class, "card")]]')

// Select an element by text
cy.xpath('//*[text()[contains(., "My Boards")]]')

// Select an element after a specific element
cy.xpath('//div[contains(@class, "card")][preceding::div[contains(., "milk")]]')

//Filter an element by index
cy.get('.board').eq(0)

// Select an element containing a specific child element
cy.get(".card").parents('.list')

// Select an element by text
cy.contains('My Boards')

// Select an element after a specific element
cy.contains('.card', 'milk').next('.card')

Filip’s recommendation is that you don’t really need XPaths. XPaths can be really powerful in traversing the DOM structure and selecting different elements, but there are other options in Cypress.

Traversing elements in Cypress

For our example, we have a Trello app and two lists with item cards. The first list has two cards, and the second has one card. We want to select each of the cards from a list. We can find the last card by doing a pair of commands cy.get(‘[data-cy=card]’).last(). First, we’re using the get command to target the cards, which will return all three cards. When you hover over your get command, you’ll see that all three cards are selected.

When you use the last command, it’s going to filter to the last card, on which you can then do some action like click or make an assertion.

You can also traverse up or down using Cypress. The next example cy.contains(‘[data-cy=card]’, ‘Soap’).parents(‘[data-cy-list]’) tries to target a parent element using text to select the card and then looks for a parent element using a CSS selector. This example is going to select our whole list.

Alternatively, if you want to traverse between the next element or a previous element, that can be done easily with cy.contains(‘[data-cy=card]’, Milk’).next() or cy.contains(‘[data-cy=card]’, Milk’).prev().

it.only('Find an element on page', () => {

  cy.visit('/board/1')

  // find last card
  cy.get('[data-cy=card]')
    .last()

  // find parent element
  cy.contains('[data-cy=card]', 'Soap')
    .parents('[data-cy=list]')

  // find next element
  cy.contains('[data-cy=card]', 'Milk')
    .next()

  // find next element
  cy.contains('[data-cy=card]', 'Bread')
    .prev()

});

You may sometimes deal with tricky situations like elements loading in at different times. The DOM is going to be flaky, because DOM is pretty much always flaky, right? Things get loaded, things get re-rendered, and so on. There was a Cypress 12 update that makes sure that when we are selecting an element and we have an assertion about that element, if the assertion does not pass, we are going to re-query our DOM. So in the background, these should command this assertion and our querying is interconnected.

it('Dealing with flaky situations', () => {

  cardsLoadRandomly(10000)

  cy.visit('/board/1')

  cy.get('[data-cy=card]')
    .last()
    .should('contain.text', 'Soap')

});

Locating elements in a shadow DOM

Dealing with shadows DOM can be tricky. By default, Cypress is not going to look for shadow DOM elements, only DOM elements. If we want to include the shadow DOM elements, we have a few options in Cypress.

it('closes side panel that is in shadow DOM', () => {

  cy.visit('https://lit.dev/playground/#sample=docs%2Fwhat-is-lit&view-mode=preview')

  cy.get('litdev-drawer', { includeShadowDom: true })
    .find('#openCloseButton', { includeShadowDom: true })
    .click()

});

If we don’t have too many of these shadow DOM elements on the page, we can use either of two commands that do essentially the same thing:

• cy.get(‘litdev-drawer’, { includeShadowDom: true }). find(‘#openCloseButton’, { includeShadowDom: true })
• cy.get(‘litdev-drawer’).shadow().find(‘#openCloseButton’)

Alternatively, if we have a lot of shadow elements in our application, we can use the approach of changing the option in the config file includeShadowDom from false to true. By default it is set to false, but if you set it to true and save your configuration, Cypress will look for shadow DOM elements automatically.

Locating elements within an iframe

Cypress does not have an iframe command. Whenever we traverse, we interact with this timeline that we have in Cypress to see the state of our application as it was during the execution of that command. In order to support the iframes, Cypress would have to do the snapshot of the iframe as well. Essentially, if you want to access an iframe using Cypress, you write a Cypress promise that will resolve the contents of the iframe.

You can add a custom command to your code base to retry the iframe. So if the iframe takes a little bit of time to appear, it’ll resolve when it appears or when the timeout eventually times out. Another way of dealing with that is installing a plugin.

it('dealing with iframes', () => {

  cy.visit('https://kitchen.applitools.com/ingredients/iframe')

  cy.iframe('#the-kitchen-table')
    .find('section')
    .should('contain.text', 'The Kitchen')

});

Cypress selector recommendations

Cypress works best when you have your test code along with the source code in the same repository. These are the recommendations that Cypress gives in the documentation:

The best recommendation is to create custom IDs for elements. As you create your test, you will create those IDs as well. So if you need a test, create an attribute, and that will do a single job, be available for end-to-end test.

Filip recommends two blog posts about selector strategies:

• Writing Strong Front-end Test Element Locators by Mark Noonan
• How to structure a big project in Cypress by Filip Hric

Adding visual testing

We can go beyond accessibility and functional tests and we can add visual tests into our test suite. We can do this with Applitools. Create a free Applitools account to access your API key, and follow our tutorial for testing web apps in JavaScript using Cypress.

Note: Do not share your API key with others. For this demo, Filip rotated his API key, so it’s no longer valid.

The visual testing will have three basic parts:

1. Open your “eyes”.
2. Check the window.
3. Close your “eyes”.

Applitools Eyes will then validate the snapshot it takes against the current baseline. But sometimes we don’t want to test certain areas of the page like dynamic data. Applitools is really intelligent about that and has options in the Test Manager to add ignore regions, but we can help it with our own ignore regions by using selectors.

it('check home screen', () => {

  cy.eyesOpen({
    appName: 'Trello',
  })

  cy.visit('/')

  cy.get('[data-cy=board-item]')
    .should('be.visible')

  cy.eyesCheckWindow({
    ignore: {
      selector: 'hide-in-applitools'
    }
  })

  cy.eyesClose()

});

In the example, we are showing the ignore region and telling which selector it should ignore. You can create a custom hide-in-applitools class to add to all your elements you want to hide, and Applitools will automatically ignore them.

Cypress selectors Q&A

After Filip shared his Cypress demonstration, we turned it over to our Q&A, where Filip responded to questions from the audience.

Using Cypress commands

Question: I was recently working on XPaths and Cypress. I understand cypress-xpath is deprecated and I was suggested to use @cypress/xpath. Is that the case?
Filip’s response: I don’t know. I know this was the situation for the Cypress grab package, which can grab your test so you can just run a subset of your Cypress test. It was a standalone plugin and then they sort of moved it inside a Cypress repository. So now it’s @cypress/grab. I believe the situation with Xpath might be similar.

Using the React testing library

Question: What are your thoughts about using the React testing library plugin for locating elements in Cypress using their accessibility roles and names – findByRole, findByLabel?
Filip’s response: [To clarify the specific library mentioned] there’s this testing library, which is a huge project and has different subprojects. One is the React testing library and their Cypress testing library, and they basically have those commands inside this (findByRole, findByPlaceholder, etc.). So I think the Cypress testing library is just an implementation of the thing you are mentioning. So what’s my opinion on that? I’m a fan. Like I said, I’m not using it right now, but it does two things at once. You can check your functionality as well as accessibility. So if your test fails, it might be annoying, but it also might mean you need to work on the accessibility of the app. So I recommend it.

Using div.board or .board

Question: Do you have a stance/opinion on div.board versus .board for CSS selectors?
Filip’s response: Not really. As I mentioned, I prefer adding my own custom selectors, so I don’t think I would have this dilemma too often.

Tracking API call execution

Question: How can we find out if an API call has executed when we click on an element?
Filip’s response: In Cypress, it’s really easy. There’s this cy.intercept command in which you can define the URL or the method or any kind of details about the API call. And you need to make sure that you put the cy.intercept command before the click happens. So if the click triggers that API call you can then use cy.wait and basically refer through alias to that API call. I suggest you take a look into the intercept command in Cypress docs.

Working with iframes

Question: Is it possible to select elements from an iframe and work with an iframe like normal?

Filip’s response: Well, I don’t see why not. It is a little bit tricky. Iframe is just another location, so it’ll always have a source pointing to a URL. So, alternatively, if you are going to do a lot of testing within that iframe, maybe you just want to open that and test the page that is being iframed. It would be a good thing to consult with developers to see if there’s a communication between the iframe and the parent frame and if there’s anything specific that needs to be covered. But if you do like a lot of heavy testing, I would maybe suggest to open the URL that the iframe opens and test that.

Learn more

So that covers our expert’s takeaways on locating web elements using Cypress. If you want to watch the demos, you can access the on-demand webinar recording.

If you want to learn more about any of these tools, frameworks like Cypress, WebDriverIO, or specifically web element locator strategies, be sure to check out Test Automation University. All the courses and content are free.
Be sure to register for the upcoming Let the Engineers Speak webinar series installment on Test Maintainability coming in May. Engineers Maaret Pyhäjärvi from Selenium and Ed Manlove from Robot Framework will be discussing test maintenance and test maintainability with a live Q&A.

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Earlier this month, Applitools hosted a webinar, Let the Engineers Speak: Selectors, where testing experts discussed one of the most common pain points that pretty much anyone who’s ever done web UI testing has felt. In this two-part blog series, I will recap what our experts had to say about locating their web elements using their respective testing frameworks: WebDriverIO and Cypress.

Introducing our experts

I’m Pandy Knight, the Automation Panda. I moderated our two testing experts in our discussion of selectors:

• Christian Bromann, WebdriverIO
• Filip Hric, Cypress

In each article, we’re going to compare and contrast the approaches between these two frameworks. This comparison is meant to be collaborative, not competitive. This series is meant to showcase and highlight the awesomeness that we have with modern frameworks and tools available.

This first article in our two-part series will define our terms and challenges, as well as recap Christian’s WebDriverIO selectors tutorial and WebdriverIO Q&A. The upcoming second article will recap Filip’s Cypress selectors tutorial and Cypress Q&A.

Defining our terms

Let’s define the words that will be used in this series to frame the discussion and so that we have the same understanding as we discuss selectors, locators, and elements:

• Selectors: A selector is a text-based query used for locating elements. Selectors could be things like CSS selectors, XPaths, or even things like an ID or a class name. In modern frameworks, you can even use things like text selectors to query by the text inside of a button.
• Locators: A locator is an object in the tool or framework you use that uses the selector for finding the element on the active page.
• Elements: The element itself is the entity on the page that exists. An element could be a button, it could be a label, an input field, or a text area. Pretty much anything in HTML that you put on a page is an element.

In one sentence, locators use selectors to find elements. We may sometimes use the terms “selector” and “locator” interchangeably, especially in more modern frameworks.

Challenges of locating web elements

We’re going to talk about selectors, locators, and the pain of trying to get elements the right way. In the simple cases, there may be a button on a page with an ID, and you can just use the ID to get it. But what happens if that element is stuck in a shadow DOM? What if it’s in an iframe? We have to do special things to be able to find these elements.

Locating web elements with WebdriverIO

Christian Bromann demonstrated effective web element locator strategies with WebDriverIO. In this series, we used Trello as the example web app to compare the frameworks. For the purposes of the demo, the app is wrapped over a component test, allowing Christian to work with web on the console to easier show you how to find certain elements on the page.

Note: All the following text in this section is based on Christian’s part of the webinar. Christian’s repository is available on GitHub.

Creating a board

First, we want to start creating a new board. We want to write a test to inspect the input element that the end user sees. From the dev tools, the input element can be found.

There are various ways to query for that input element:

• We could fetch the element with WebdriverIO using $ or $$ commands, depending if you want to fetch one element, the first element on the page, or all elements. Using an input as a selector is not advised, as there could be more inputs and then the wrong input could be fetched.
• We could also use a CSS class called px-2. Using CSS classes is also not advised, as oftentimes these classes are meant to style an element, not necessarily locate it.
• We could use a property, such as the name of the input using the data attribute. This is the suggested way to locate a web element in WebdriverIO – using the accessibility properties of an element.

WebdriverIO has a more simplified way to use accessibility properties. This approach is a much better selector, because it’s actually what the user is seeing and interacting with. To make things easier, Chrome helps with finding the accessibility selector for every element under the accessibility tab.

Note: WebdriverIO doesn’t use the accessibility API of the browser and instead uses a complex XPath to compute the accessibility name.

After you’ve located the input element, press enter using the keys API.

  it('can create an initial board', async () => {
    await $('aria/Name of your first board').setValue('Let the Engineers Speak')
    await browser.keys(Key.Enter)
    await expect(browser).toHaveUrlContaining('/board/1')
    
    await browser.eyesCheck('Empty board')
  })

Creating a list

Now that our board is created, we want to start a list. First, we inspect the “Add list” web element for the accessibility name, and then use that to click the element.

Set the title of the list, and then press enter using the keys API.

  it('can add a list on the board', async () => {
    await $('aria/Enter list title...').setValue('Talking Points')
    await $('aria/Add list').click()

    /**
     * Select element by JS function as it is much more perfomant this way
     * (1 roundtrip vs nth roundtrips)
     */
    await expect($(() => (
      [...document.querySelectorAll('input')]
        .find((input) => input.value === 'Talking Points')
    ))).toBePresent()
  })

Adding list items

To add to the list we created, the button is a different element that’s not accessible, as the accessibility name is empty. Another way to approach this is to use a WebdriverIO function that allows me to add a locator search applicator strategy.

The example applicator strategy basically queries all this on the page to find all divs that have no children and that text of a selector that you provide, and now you should be able to query that custom element. After getting the custom elements located, you can assert that three cards have been created as expected. You can inject your JavaScript script to be run in the browser if you don’t have accessibility selectors to work with.

  it('can add a card to the list', async () => {
    await $('aria/Add another card').click()
    await $('aria/Enter a title for this card...').addValue('Selectors')
    await browser.keys(Key.Enter)
    await expect($$('div[data-cy="card"]')).toBeElementsArrayOfSize(1)
    await $('aria/Enter a title for this card...').addValue('Shadow DOM')
    await browser.keys(Key.Enter)
    await $('aria/Enter a title for this card...').addValue('Visual Testing')
    await browser.keys(Key.Enter)
    await expect($$('div[data-cy="card"]')).toBeElementsArrayOfSize(3)

    await browser.eyesCheck('board with items')
  })

Starring a board

Next, we’ll “star” our board by traversing the DOM using WebdriverIO commands. We need to first locate the element that has the name of the board title using the attribute selector and going to the parent element.

From that parent element, to get to the star button, we chain the next command and call the next element. Now we can click the star and see it change to an enabled state. So with WebdriverIO, you can chain all these element queries and then add your action at the end. WebdriverIO uses a proxy in the background to transform everything and execute the promises after each other so that they’re available. One last thing you can also do is query elements by finding links with certain text.

Summarizing Christian’s suggestions for using selectors in WebdriverIO, always try to use the accessibility name of an element. You have the dev tools that give you the name of it. If you don’t have the accessibility name available, improve the accessibility of your application if possible. And if that’s not an option, there are other tricks like finding an element using JavaScript through propertis that the DOM has.

  it('can star the board', async () => {
    const startBtn = $('aria/Let the Engineers Speak').parentElement().nextElement()
    await startBtn.click()
    await expect(startBtn).toHaveStyle({ color: 'rgba(253,224,71,1)' })
  })

Accessing a shadow root

Using a basic web HTML timer component as an example, Christian discussed shadow roots. The example has multiple elements and a nested shadow root. Trying to access a button within the shadow root results in an error that you don’t have access from the main page to the shadow root. WebdriverIO has two things to deal with this challenge. The first method is the deep shadow root selector. This allows you to access all of the shadow root elements or filter by defined attributes of the elements in the shadow root.

A different way to access elements in the shadow is using the browser shadow command, which basically allows you to switch to the search within the shadow root.

    describe('using deep shadow selector (>>>)', () => {
        beforeEach(async () => {
            await browser.url('https://lit.dev/playground/#sample=docs%2Fwhat-is-lit&view-mode=preview')

            const iframe = await $('>>> iframe[title="Project preview"]')
            await browser.waitUntil(async () => (
                (await iframe.getAttribute('src')) !== ''))
            
            await browser.switchToFrame(iframe)
            await browser.waitUntil(async () => (await $('my-timer').getText()) !== '')
        })

        it('should check the timer components to work', async () => {
            for (const customElem of await $$('my-timer')) {
                const originalValue = await customElem.getText()
                await customElem.$('>>> footer').$('span:first-child').click()
                await sleep()
                await customElem.$('>>> footer').$('span:first-child').click()
                await expect(customElem).not.toHaveTextContaining(originalValue)

                await customElem.$('>>> footer').$('span:last-child').click()
                await expect(customElem).toHaveTextContaining(originalValue)
            }
        })
    })

Integrating with Applitools

Lastly, Christian shared using Applitools with WebdriverIO. The Applitools Eyes SDK for WebdriverIO is imported to take snapshots of the app as the test suite runs and upload them to the Applitools Eyes server.

if (process.argv.find((arg) => arg.includes('applitools.e2e.ts'))) {
    config.services?.push([EyesService, {
        viewportSize: {width: 1200, height: 800},
        batch: {name: 'WebdriverIO Test'},
        useVisualGrid: true,
        browsersInfo: [
            {width: 1200, height: 800, name: 'chrome'},
            {width: 1200, height: 800, name: 'firefox'}
        ]
    }])

With this imported, our tests can be simplified to remove some of the functional assertions, because Applitools does this for you. From the Applitools Eyes Test Manager, you can see the tests have been compared against Firefox and Chrome at the same time, even though only one test was run.

WebdriverIO selectors Q&A

After Christian shared his WebdriverIO demonstration, we turned it over to our Q&A, where Christian responded to questions from the audience.

Using Chrome DevTools

Audience question: Is there documentation on how to run WebdriverIO in DevTools like this to be able to use the browser and $ and $$ commands? This would be very helpful for us for day-to-day test implementation.
Christian’s response: [The demo] is actually not using DevTools at all. It uses ChromeDriver in the background, which is automatically spun up by the test runner using the ChromeDriver service. So there’s no DevTools involved. You can also use the DevTools protocol to automate the browser. The functionality is the same. WebdriverIO executes the same XPath. There’s a compliant DevTools implementation to the WebdriverIO protocol so that the WebdriverIO APIs works on both. But you really don’t need DevTools to use all these accessibility selectors to test your application.

Integrating with Chrome Console

Question: How is WebdriverIO integrated with Chrome Console to type await browser.getHtml() etc.?
Christian’s response: With component tests, it’s similar to what other frameworks do. You actually run a website that is generated by WebdriverIO. WebdriverIO injects a couple of JavaScript scripts and it loads web within that page as well. And then it basically sends commands back to the Node.js world where it’s then executed by ChromeDriver and then the responses are being sent back to the browser. So basically WebdriverIO as a framework is being injected into the browser to give you all the access into the APIs. The actual commands are, however, run by Chrome Driver.

Testing in other local languages

Question: If we’re also testing in other languages (like English, Spanish, French, etc.), wouldn’t using the accessibility text fail? Would the ID not be faster in finding the element as well?
Christian’s response: If you have a website that has multiple languages and you have a system to inject those or to maintain the languages, you can use this tool to fetch the accessibility name of that particular language you test the website in. Otherwise, you can say I only tested one language, because I would assume it would not be different in other languages. Or you create a library or a JSON file that contains the same selectors for different languages but the same accessibility name for different languages. And then you import that JSON to your test and just reference the label to have the right language. So there are ways to go around it. It would make it more difficult in different languages, obviously, but still from the way to maintain end-to-end tests and maintain tests in general, I would always recommend accessibility names and accessibility labels.

Using div.board or .board

Question: Do you have a stance/opinion on div.board versus .board for CSS selectors?
Christian’s response: I would always prefer the first one, just more specific. You know, anyone could add another board class name to an input element or what not. And so being very specific is usually the best way to go in my recommendation.

Tracking API call execution

Question: How can we find out if an API call has executed when we click on an element?
Christian’s response: WebdriverIO has mocking capabilities, so you can look into when a certain URL pattern has been called by the browser. Unfortunately, that’s fairly based on DevTools because WebdriverIO, the first protocol, doesn’t support it. We are working with the browser vendors and the new WebdriverIO binary protocol to get mocking of URLs and stubbing and all that stuff to land in the new browsers. And so it’ll be available across not only Chrome, but also Firefox, Safari, and so on.

Working with iframes

Question: Is it possible to select elements from an iframe and work with an iframe like normal?

Christian’s response: An iframe is almost similar to a shadow DOM – just that an iframe has much more browser context than just a shadow DOM. WebdriverIO has an issue currently. To implement the deep shadow selector for iframes as well, you could do three characters and then name any CSS path and it would make finding an iframe very easy. But you can always switch to an iframe and then query it. So it’s a little bit more difficult with iframes, but it’s doable.

Learn more

So that covers our expert’s takeaways on locating web elements using WebdriverIO. In the next article in our two-part series, I’ll recap Filip’s tutorial using Cypress, as well as the Cypress Q&A with the audience.

If you want to learn more about any of these tools, frameworks like Cypress, WebDriverIO, or specifically web element locator strategies, be sure to check out Test Automation University. All the courses and content are free. You can also learn more about visual testing with Applitools in the Visual Testing learning path.
Be sure to register for the upcoming Let the Engineers Speak webinar series installment on Test Maintainability coming in May. Engineers Maaret Pyhäjärvi from Selenium and Ed Manlove from Robot Framework will be discussing test maintenance and test maintainability with a live Q&A.

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Today, I’m excited to announce that Applitools is launching an Ambassadors program! At Applitools, we want to assemble a group of industry leaders who love Applitools and want to help us shape its future. We’re looking for developers, testers, designers, educators, managers, and directors with expertise in building excellent apps and influence in tech communities.

Who is an “Applitools Ambassador”?

An Applitools Ambassador is essentially a “super-fan.” They just love using the Applitools platform and products for building excellent testing pipelines. Below are some traits of of our Ambassadors:

• An ambassador has hands-on experience with Applitools, either as a practitioner themselves or as someone who has led initiatives using Applitools.
• An ambassador has expertise in software design, development, and testing. They know what works and what doesn’t, and they carry the respect of their peers.
• An ambassador either has or aspires to build an influential presence in their tech communities. They write blogs, speak at events, and advocate for recommended practices – especially involving Applitools.

What do Ambassadors do?

Ambassadors do lots of things! Here are some examples:

• Writing articles for the Applitools blog
• Attending and even speaking at industry events, including Applitools events
• Providing feedback on Applitools’ products and services
• Sharing their software experiences with others

What benefits do Ambassadors receive?

Ambassadors receive quite a few benefits, including:

• Personal Brand – Receive invitations to boost personal brands, such as speaking at events and writing articles for the Applitools blog.
• News – Be the first to hear the latest product updates directly from Applitools, and get hands on with new features as soon as they drop.
• Feedback – Share thoughts directly with R&D to help shape Applitools’ future.
• Networking – Meet other like-minded folks in the community through opportunities for collaboration, including members of the Applitools team.
• Swag – Receive special gifts reserved exclusively for Applitools Ambassadors as a way to stand out at community events.

Who are the Ambassadors?

Applitools has invited several excellent individuals to become part of our Ambassador program launch. We will announce them one-by-one in the coming days. Be sure to keep an eye on the Applitools Ambassadors page as well as Twitter and LinkedIn to see who’s joining!

How can I become an Ambassador?

If you think you’ve got what it takes to become an Applitools Ambassador, then please apply! We’d love to hear from you. In the meantime, keep an eye open as we announce our first set of Ambassadors.

Test Automation University (TAU) is the premier platform for learning about software testing and automation. Powered by Applitools, TAU offers dozens of courses on all testing topics imaginable from the world’s best instructors for FREE.

In my previous article, I shared a breakdown of TAU by the numbers. I covered all the stats and metrics about TAU today. Now, in this article, I’d like to share the future by publicly announcing all the new content we have scheduled in our development pipeline!

New courses

Test Automation Design Patterns by Sarah Watkins

Up first is Test Automation Design Patterns by my good friend Sarah Watkins. Anyone who’s ever worked on a test project knows that good test cases don’t just happen – they’re built intentionally. In this upcoming course, Sarah teaches everything she knows about designing and delivering test automation projects at scale, from formulating tests to follow Arrange-Act-Assert, to using dependency injection for handling test inputs and other objects, to deciding between page objects or the Screenplay Pattern. This will be an advanced course to really help you become the ultimate SDET, or “Software Development Engineer in Test.”

GitHub Actions for Testing by Matthias Zax

Next up is GitHub Actions for Testing by Matthias Zax. Automation means much more than merely scripting test case procedures. It includes automating repetitive segments of your development process, like running tests and reporting results. GitHub Actions are a marvelous, lightweight way to add helpful workflows to manage your source code repositories. In this course, Matthias teaches how to create GitHub Actions specifically for testing and quality concerns. I know GitHub Actions have transformed how I run my own GitHub projects, and I’m sure they’ll make a big difference for you, too!

Behavior-Driven Development with SpecFlow by Bas Dijkstra

Next is a course for a topic very near and dear to my own heart: Behavior-Driven Development with SpecFlow by Bas Dijkstra. Behavior-Driven Development, or “BDD” for short, is a set of pragmatic practices that help you develop high-quality, valuable behaviors. Not only will Bas walk through the phases of Discovery, Formulation, and Automation, but he will show you how to use SpecFlow – one of the best BDD test frameworks around – to automate Gherkin scenarios. He will also teach how to integrate SpecFlow with popular libraries like Selenium WebDriver and RestAssured.NET.

I hope you’re excited for those three upcoming courses. Course development takes time, so we don’t have a hard publishing date set, but be on the lookout for them this year. I know Sarah, Matthias, and Bas are all hard at work recording their chapters.

New learning paths

That’s not all our new content. We have a lot more to announce today, so let’s keep going.

Visual Testing Learning Path

We have a new learning path to announce: Visual Testing! Applitools’ very own Matt Jasaitis is developing a three-part series on visual testing. The example code uses Selenium WebDriver with Java, but the concepts he teaches are universal. The first two courses in this learning path are already available. The third course, with the tentative title Expert Visual Testing Practices, will be available by April.

Leadership Learning Path

Historically, Test Automation University has focused almost exclusively on testing and automation skills. As we grow in our career as professionals, we often find we need to develop other kinds of skills as well. That’s why I’m excited to announce that TAU will be developing a new learning path for leadership!

Managing Test Teams by Jenny Bramble

The first course in our new leadership track will be Managing Test Teams by Jenny Bramble. At some point in our careers, many of us face the question, “Should I become a manager?” Becoming a good manager requires a different skill set than becoming a good tester or a good engineer. It comes with a new set of challenges that you might not expect. In this course, Jenny will share everything she’s learned about leading teams of testers, holding crucial conversations, and deciding if a career path in management is right for you. Jenny is also one of our speakers later today, so be sure to join her session.

Building Up Quality Leaders by Laveena Ramchandani

The second course will be Building Up Quality Leaders by Laveena Ramchandani. While Jenny’s course will focus primarily on being a good manager, Laveena’s course will focus on attributes of leadership that apply to any role within software quality. Laveena will cover leadership skills like servanthood, charisma, decisiveness, and gratitude, and how they all apply to the testing world.

Creating Effective Test Strategies by Erin Crise

The third course will be Creating Effective Test Strategies by Erin Crise. The key to leading successful testing initiatives is a solid, well-grounded, well-balanced strategy. The strategy must balance automation with exploration. It must include a variety of areas such as visuals, accessibility, and performance. It must be communicated effectively across teams. And it must be agile enough to adjust to changes. With her wealth of experience from several software projects, Erin will teach how to plan and execute effective test strategies in this course.

Just like for the other upcoming courses I just announced, these leadership courses aren’t ready yet. Expect them to drop later this year. We are also planning a few more courses in this leadership path, so stay tuned for more details!

Refreshing old courses

Now, we said earlier how TAU is over four years old. In the world of tech, that’s a long time. I hate to say it, but some of our courses are becoming outdated. Tools and frameworks evolve – as they should. It’s time to refresh many of our courses.

Last year, we published our first refresh courses: Introduction to Robot Framework by Paul Merrill and UI Automation with WebdriverIO v7 by Julia Pottinger. Both refresh courses were well received by our community of students.

After auditing our whole course catalog, I decided that it’s time to refresh some more courses!

Cypress courses by Filip Hric

Our intro and advanced Cypress courses are among our most popular. In fact, Introduction to Cypress is currently ranked #8 for monthly completions. This is no surprise given Cypress’s incredible popularity with front end developers. However, our mainstay courses were published before many of Cypress’s current features, like component testing and the new directory structure introduced with Cypress 10.

Filip Hric will take up the challenge to refresh both the intro and advanced Cypress courses. He will redevelop them to flow together cohesively as parts 1 and 2. We will also create a new Cypress learning path for them once we publish them.

Playwright courses by Renata Andrade

As a Playwright Ambassador, I can’t let Cypress have all the fun. Playwright is so hot right now. The community is booming, the downloads are skyrocketing, and even within Applitools and TAU, we see its adoption growing significantly. That’s why we are not only refreshing our introduction-to-Playwright course, but we are also going to publish an all-new Advanced Playwright course!

Our instructor Renata Andrade, like Filip, will develop these courses to be parts 1 and 2. She will also develop them in TypeScript. And you’d better believe we’ll create a Playwright learning path for them as well.

Mobile Automation with Appium in Java by Moataz Nabil

Let’s not forget mobile test automation as well. Moataz Nabil is going to split his course, Mobile Automation with Appium in Java, into two parts. At six and a half hours, this course is currently the longest on TAU. By splitting it into two parts, we hope to make it easier for students to complete.

Web Element Locator Strategies by Andrew Knight

And finally, there’s one more course we plan to redevelop: my very first course, Web Element Locator Strategies. In my refresh, I intend to develop new subchapters to show how to write locators in different frameworks like Playwright and Cypress, not just in Selenium Java. I’ll include new types of locators as well.

Retired courses

We also needed to retire a few of our courses that are out of date:

1. Automated Visual Testing with Java by Angie Jones
2. AI for Element Selection by Jason Arbon
3. From Scripting to Framework with Selenium and C# by Carlos Kidman

In case you missed it, we removed these courses from TAU at the beginning of March. I want to thank these instructors for making these courses.

A full curriculum

I know I’ve talked a lot about learning paths today. One of the most requested new features we receive is certificates for completing learning paths. I’m sorry to say that we are not going to add certificates for learning paths. Learning paths are somewhat dynamic. They can change over time as we publish new courses. So, giving certificates for completing them doesn’t make sense.

Instead, we are toying with a stronger idea: a curriculum. A learning path concentrates on one narrow testing topic, like leadership or visual testing. I want to create a comprehensive program of courses covering a wide breadth of testing topics, and to recognize completion of that program with something like a diploma. We have a rich catalog of courses for building a strong curriculum. Right now, this is still in the idea stage. I’m hoping to bring it into reality later this year.

One more thing…

TAU is four years old, and while I hate to admit it, the TAU platform itself is starting to show its age. With so many courses available, it’s becoming hard to find the ones you want. On the backend, it actually takes quite a bit of manual labor to publish a new course, too.

It’s time to refresh the TAU platform. It’s time for TAU version 2.

Over the coming months, Matt Jasaitis and I will redevelop the whole TAU web app. Yes, that’s the same Matt Jasaitis who is creating the Visual Testing learning path. Our goal is to build a sustainable learning platform to serve Test Automation University for years to come. We will also build a better support system into the platform. Unfortunately, this project is so new that I don’t even have mockups to show you yet, but at least I can announce that TAUv2 is coming.

So, what do y’all think about all this new content coming to TAU? We’d love to hear from you. Let us know through TAU Slack.

Test Automation University (TAU) is the premier platform for learning about software testing and automation. Powered by Applitools, TAU offers dozens of courses on all testing topics imaginable from the world’s best instructors for FREE.

In this article, I’d like to share a breakdown of Test Automation University by the numbers: all the stats and metrics that we gather for insights into the program. I also shared this information in my keynote address for Test Automation University’s 2023 conference, What’s New With TAU? Let’s dive in!

Students, courses, and achievements

Believe it or not, it’s been four years since Applitools launched TAU! Four years is a long time. That’s as long as a typical college undergraduate degree.

I remember feeling thrilled when Applitools first launched TAU. Before then, there wasn’t any cohesive, community-oriented education platform for testing and automation. Most schools barely included testing topics in their programs. While the Web had a wealth of information, it was difficult to know which resources were both comprehensive and trustworthy. Test Automation University provided a central platform that could be trusted with literally the best and the brightest instructors in the world.

Now, I can’t imagine a world without TAU. At least once a week, someone I’ve never met before slides into my DMs and asks me, “Andy, how can I start learning about testing and automation?” I literally just tell them, “Have you heard of Test Automation University?” Boom. Done. I don’t have to check my bookshelf for titles or do a quick search for blog articles. I just say go to TAU. They’re always grateful.

And when they do go to TAU, they join our ever-growing community of students! Just last month, we crossed 150 thousand registered “TAU-dents”! That’s a huge size for any community.

Every time we’ve hit a milestone, we’ve celebrated. When we hit 50 thousand students in June 2020, we threw our first TAU Party. Then, when we hit 100 thousand students, we hosted TAU: The Homecoming, our first virtual conference. That’s when I first joined Applitools. Now, at 150 thousand students, we’re hosting our second virtual conference.

Our growth is also accelerating. Historically, we have grown by 50 thousand students about every year and a half. This most recent milestone happened a little sooner than expected.

Every week, about 800 to 1000 new students sign up for TAU. That’s sustained, consistent growth. It’s awesome to see all the new students joining every week.

Over 4 years, we have published a total of 73 courses. That’s a lot of content! A single course has at least 4 chapters with at least 30 minutes of video lessons. Most courses average about 8 chapters over an hour of content, and our longest courses stretch to 6 hours! We try to publish a new course about once a month. And the best part is that every one of our courses is completely, totally, absolutely FREE!

Our courses would be nothing without the excellent instructors who teach them. To date, we have 48 different instructors who have produced courses for TAU. These folks are the real superstars. They are bona fide software testing champions. Let me tell you, it is no small effort to develop a TAU course.

When Applitools first launched TAU back in 2019, I remember thinking to myself as a younger panda, “Man, that’s awesome. I wish I could be part of that. Maybe someday I can develop a TAU course and be like them.” A few months later, Angie Jones slid into my DMs and asked me if I’d want to develop my first course: Web Element Locator Strategies. In all honesty, TAU is one of the main reasons I work at Applitools today!

Let’s look at achievements next. In total, all students have completed 162 thousand courses! That’s 162 thousand certificates and 162 thousand badges, averaging slightly more than 1 course completion per student.

The credit total is even more impressive. Our students have earned about 116 million course credits. That’s just under 800 credits per student, which correlates to about one course’s worth of credits. 116 million credits earned is a mind-blowing number, and that total is only going to increase with time.

Course completion analysis

Those are just some raw numbers on TAU. I also did some deeper analysis on our courses to learn what topics and formats are most valuable to our community. Let’s take a look at that together.

When evaluating courses, the main metric I measure is course completion – meaning, how many students completed the course and earned a certificate. Anyone can start a course, but it takes dedication to complete the course. Completion signals the highest level of engagement with TAU: a student must dedicate hours of study and coding to earn a course badge.

My goal for any course we publish – and how I determine if a course is “successful” in most basic terms – is to consistently hit 30 completions per month, or 1 completion per day. Very popular courses hit 2 completions per day, and our most popular courses hit at least 3 completions per day.

Our top 10 courses are all in that upper echelon of completion rate. Our most popular course by far is Angie Jones’ Setting a Foundation for Successful Test Automation, followed by my course, Web Element Locator Strategies.

All three of our programming courses – Java, JavaScript, and Python – appear in the top 10. Rounding out the top 10 are courses on Selenium WebDriver, Cypress, API testing, and IntelliJ IDEA. If you want to see more details about the most popular TAU courses, check out the article I wrote about it a few months ago on the Applitools blog. Note that some small shifts have happened since I wrote that article, but the information is still mostly accurate.

When partitioning the full catalog of courses by the programming languages they use, we can see that Java and JavaScript dominate the landscape. This should be no surprise, since those two languages are by far the most popular languages for test automation. Applitools product data also backs this up. TAU offers a good number of Python and C# courses that are reasonably popular. I personally developed 3 of those 7 Python courses. TAU also offers a handful of courses that use Ruby and other languages. Unfortunately, however, we have discovered that those courses, especially for Ruby, have very low completion rates. Again, I surmise that this reflects broader industry trends.

I also analyzed the completion rate for a course versus its length. What I found is probably not surprising: shorter courses have higher completion rates, while longer courses have lower completion rates. The average completion rate for all courses is 57%. The breaking point appears to be at about two hours, with a sweet spot between 60-90 minutes. Since we want to encourage course completions, we are going to encourage instructors to produce courses in the hour-to-hour-and-a-half range moving forward.

So, what’s next?

These numbers describe TAU as it is today. In my next article, I’ll share all the new plans we have for TAU, including upcoming courses!

In our previous article, we learned what autonomous testing is all about: autonomous testing is when a tool can learn an app’s behaviors and automatically execute tests against them. It then provides results to humans who can determine what’s good and what’s bad. Fully autonomous testing solutions are not yet available today, but we can get part of the way there with readily available tools. Let’s learn how to build our own semi-autonomous solution using Playwright and Applitools Eyes!

The solution sketch

Let’s say you have a website with multiple pages. It would be really nice if a test suite could visit each page and make sure it looks okay: no missing buttons, no overlapping text, and no other kinds of visual bugs. The tests wouldn’t be very sophisticated, but they’d quickly catch a lot of problems. They’d be like smoke tests.

There’s a straightforward way to do this. Most websites have a sitemap file that lists the links for all the pages. We could use a browser automation tool like Selenium, Cypress, or Playwright to visit each of those pages, and we could use a tool like Applitools Eyes to capture visual snapshots of each page. Every time we run the suite, it would automatically discover new pages and avoid removed pages. Existing pages would be checked for visual differences. We wouldn’t need to explicitly code what to check – the snapshots would implicitly check everything on the pages! With the Applitools Ultrafast Grid, we could even test these pages against different browsers, devices, and viewport sizes.

This kind of test suite is technically “autonomous” because we, as human testers, don’t need to explicitly write the tests. The sitemap provides the pages, and visual testing covers the assertions.

The website to test

Last year at Applitools, we replaced our old tutorial website with a new website that uses Docusaurus, a very popular documentation framework based on React. We also rewrote several of the guides for our most popular SDKs. Presently, the site has about 60 pages of varying length. Since many of the pages host similar content, we use components to avoid duplication in text and in code. However, that means any change could inadvertently break multiple pages.
Our tutorial site is currently hosted at https://applitools.com/tutorials/:

The tutorial site also has a sitemap file at https://applitools.com/tutorials/sitemap.xml:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9" xmlns:news="http://www.google.com/schemas/sitemap-news/0.9" xmlns:xhtml="http://www.w3.org/1999/xhtml" xmlns:image="http://www.google.com/schemas/sitemap-image/1.1" xmlns:video="http://www.google.com/schemas/sitemap-video/1.1">
    <url>
        <loc>https://applitools.com/tutorials/</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>

    <url>
        <loc>https://applitools.com/tutorials/guides/advanced-tricks/troubleshooting-issues</loc>
        <changefreq>monthly</changefreq>
        <priority>0.5</priority>
    </url>
...

It would be very helpful to test this tutorial site with the semi-autonomous testing tool we just sketched out.

Writing the autonomous testing code

To follow along with this article, you can find the GitHub repository for the project at https://github.com/AutomationPanda/auto-website-testing. In the package.JSON file, you’ll find all the packages needed for this project:

• Playwright
• Applitools Eyes SDK for Playwright
• ts-node

Since I’ll be developing my code in TypeScript instead of raw JavaScript, the ts-node package will allow me to run TypeScript files directly.

{
  "name": "auto-website-testing",
  "version": "1.0.0",
  "description": "A semi-autonomous testing project that visually tests all the pages in a website's sitemap",
  "main": "index.js",
  "scripts": {},
  "repository": {
     "type": "git",
     "url": "git+https://github.com/AutomationPanda/auto-website-testing.git"
   },
   "keywords": [],
   "author": "",
   "license": "ISC",
   "bugs": {
      "url": "https://github.com/AutomationPanda/auto-website-testing/issues"
   },
   "homepage": "https://github.com/AutomationPanda/auto-website-testing#readme",
   "devDependencies": {
     "@applitools/eyes-playwright": "^1.13.0",
     "@playwright/test": "^1.29.1",
     "ts-node": "^10.9.1"
   }
}

The main file in our project is autonomous.ts. This isn’t a typical Playwright test that has described blocks and test functions, but rather autonomous.ts is just a plain old script. The first thing we need to do is read in some environment variables. For test inputs, we’ll need the base URL of the target website. We’ll need the site name of the website for logging purposes and reporting purposes, and we’ll also need a level of test concurrency for the Applitools Ultrafast Grid, which will handle our visual snapshots. If you’re on a free Applitools account, you’ll be limited to one, but I’ll be using a bit more in this example.

import { chromium } from '@playwright/test';
import { BatchInfo, Configuration, VisualGridRunner, BrowserType, Eyes, Target } from '@applitools/eyes-playwright';

(async () => {

     // Read environment variables
     const BASE_URL = process.env.BASE_URL;
     const SITE_NAME = process.env.SITE_NAME;
     const TEST_CONCURRENCY = Number(process.env.TEST_CONCURRENCY) || 1;

Once we read those in, we want to validate those environment variables to make sure their values are given and they’re good. Then what we’ll do is we will figure out what the sitemap URL is. Basically, the sitemap URL will be the base URL plus this standard XML file name.

 // Validate environment variables
 if (!BASE_URL) {
    throw new Error('ERROR: BASE_URL environment variable is not defined');
 }
 if (!SITE_NAME) {
    throw new Error('ERROR: SITE_NAME environment variable is not defined');
 }

 // Parse the base and sitemap URLs
 const baseUrl = BASE_URL.replace(/\/+$/, '');
 const sitemapUrl = baseUrl + '/sitemap.xml';

Next, we will set up Applitools to be able to do visual testing. These are all fairly standard Applitools SDK objects. If you take one of our Applitools SDK tutorials, you’ll see things just like this. Basically, we’ll need a visual grid runner to connect to the Ultrafast Grid for rendering our snapshots. We’ll create a batch which will have the name of our site so that we can see reporting, and we’ll have a configuration object to specify things like the batch, the browsers, and all these other things we want.

 // Create Applitools objects

 let runner = new VisualGridRunner({ testConcurrency: TEST_CONCURRENCY });
 let batch = new BatchInfo({name: SITE_NAME});
 let config = new Configuration();
 let widthAndHeight = {width: 1600, height: 1200};
 let snapshotPromises: Promise<any>[] = [];

With the configuration, we’re going to set the batch and we’re going to add one browser to test. I want to test Chrome with this particular viewport size. If we wanted to, we could also test other browsers in the Ultrafast Grid such as Firefox, Safari, and Edge Chromium. Even if you don’t have those browsers installed on your local machine, it’s all going to be done in the Applitools cloud. For this example, we’ll use one browser.

 // Set Applitools configuration
 config.setBatch(batch);
 config.addBrowser(1600, 1200, BrowserType.CHROME);
 // config.addBrowser(1600, 1200, BrowserType.FIREFOX);
 // config.addBrowser(1600, 1200, BrowserType.SAFARI);
 // config.addBrowser(1600, 1200, BrowserType.EDGE_CHROMIUM);

Now comes the fun part of getting that sitemap file. What we’ll need to do is launch a browser through Playwright, we’ll just use Chromium. Then we’ll need to create a new browser context from that browser and we’ll give it a standard width and height viewport. Then we’ll get a page object from that context, because with Playwright, all interactions happen through a page object.

 // Set up a browser
 const browser = await chromium.launch();

 // Set up a sitemap context and page
 const sitemapContext = await browser.newContext({viewport: widthAndHeight});
 const sitemapPage = await sitemapContext.newPage();

Once we’ve got that page, now we can visit the sitemap page and we can try to find all of the page links inside that sitemap file. Even though it’s XML, Playwright can still parse it just like it’s a regular webpage. Once we’ve got that list of page links, then we’re going to close that session and so this Playwright session will be done.

 // Get the sitemap
 await sitemapPage.goto(sitemapUrl);
 const pageLinks = await sitemapPage.locator("loc").allTextContents();
 sitemapContext.close();

Now that we have the list of all pages from the sitemap, we can visit each one and capture a snapshot. To do that, we’re going to iterate over that list with a for loop for each page we visit. We’re going to make a promise so that we can capture those snapshots asynchronously. In the background, for each page, we are going to create a new browser context and then from that context, create a new page object, and then start an Applitools Eyes session. This is what enables us to capture those visual snapshots.

 // Capture a snapshot for each page
 for (const link of pageLinks) {
     snapshotPromises.push((async () => {

       // Open a new page
       const linkContext = await browser.newContext({viewport: widthAndHeight});
       const linkPage = await linkContext.newPage();

       // Open Eyes
       const eyes = new Eyes(runner, config);
       await eyes.open(
           linkPage,
           SITE_NAME,
           link.replace(baseUrl, ''),
           widthAndHeight
     );

Taking the snapshot is pretty basic. We just visit the page and we say eyes.check, and we take a picture of the whole window. Once we do that, we can close our session so that Applitools knows that’s the only snapshot we’re taking. Firing these off in different promises means that we can run them asynchronously in the background, letting Applitools Eyes crunch through all of the visual validations, and that way we’re not waiting for them one at a time. They’ll just all go. Once we fired off all of the visual snapshots, we can wait for those promises to join at the end.

       // Take the snapshot
       await linkPage.goto(link);
       await eyes.check(link, Target.window().fully());
       console.log(`Checked ${link}`);

       // Close Eyes
       await eyes.close(false);
       console.log(`Closed ${link}`);

    })());
 }

And finally, once that’s complete, we can close the browser and be done with testing.

 // Close all Eyes
 console.log('Waiting for all snapshots to complete...');
 await Promise.all(snapshotPromises);

 // Close the browser
 await browser.close();
 console.log('Complete!');

})();

That’s all there is to our autonomous testing script. It’s pretty concise – only about 80 lines long. It doesn’t take a whole lot of logic to write autonomous tests.

Running the tests

Let’s run the script to test it out. The website I’m going to use in this example is the Applitools tutorial site, which has about 60 pages. You can use any site you want as long as you set your environment variables. If you want to run this, you will need an Applitools account, which you can register for free with your email or GitHub account. Once you register an account, you’ll take your Applitools API key and set that as an environment variable.

We need to run our tests in the terminal. If this is your first time, you’ll need to run npm install to install the npm packages as well as npx playwright install to install the Playwright browsers. Once you’ve run those installers and set your environment variables, you just need to run npx ts-node autonomous.ts to run the script.

The script fetches the sitemap file, parsing out all of those links, and it’s firing off promises to start capturing visual snapshots for each one that’s happening asynchronously. The messages saying “Checked” link means those have now been initiated. What we’ll start to see is as they complete one by one in the Applitools Ultrafast Grid, we’ll see how the session becomes closed. Closed images code, JavaScript, closed mobile browser, that means one by one, the visual testing has been completed.

While the test is running, we can see results in the Applitools Eyes dashboard as a batch.

If I look at all the tests they’re popping in, we can see some of them are still running while others have passed.

If I want to see what the visual comparisons look like, I can compare them side by side. If there are no visual differences, the test for this page will pass.

You can see on the page it captures everything, the title bar, the sidebar table of contents, main body and footer. It will even scroll all the way to the bottom to make sure it gets the full page worth of contents.

No matter how long the page is, everything is being compared visually. It’s pretty cool to see just how many tests our autonomous testing solution can uncover. As I said before, the tutorial site right now has about 60 pages, which means that’s 60 tests that I didn’t have to explicitly write. Playwright plus Applitools Eyes took care of it all for me.

Improving the tool

This example of a semi-autonomous testing tool is rather basic. There are plenty of ways we could improve it:

• Decoupling our tests: We could write separate scripts for fetching sitemap links and taking the visual snapshots. That would let us provide links to visit by ways other than a sitemap file. We could also add an intermittent step to filter links from a sitemap file.
• Expanding test coverage: We could add settings to test different browsers, devices, and viewports. This would provide autonomous cross-browser and cross-device testing coverage for multiple screen sizes.
• Target specific regions: We could exclude parts of pages like navigation bars and sidebars. This could allow us to target specific portions of a page in our tests while ignoring content that is dynamic or that we may want to test separately.

Even with the basics before any of these suggestions are implemented, this tool still provides a lot of value for a little bit of work. You can clone it from the GitHub repository and try it yourself. Let us know @Applitools how you use it!
Read more about Applitools Eyes.

Applitools is working on a fully autonomous testing solution that will be available soon. Reach out to our team to see a demo and learn more!

Autonomous testing – where automated tools figure out what to test for us – is going to be the next great wave in software quality. In this article, we’ll dive deeply into what autonomous testing is, how it will work, and what our workflow as a tester will look like with autonomous tools. Although we don’t have truly autonomous testing tools available yet, like self-driving cars, they’re coming soon. Let’s get ready for them!

What is autonomous testing?

So, what exactly is autonomous testing? It’s the next evolutionary step in efficiency.

A brief step back in time

Let’s take a step back in time. Historically, all testing was done manually. Humans needed to poke and prod software products themselves to check if they worked properly. Teams of testers would fill repositories with test case procedures and then grind through them en masse during release cycles. Testing moved at the speed of humans: it got done when it got done.

Then, as an industry, we began to automate our tests. Many of the tests we ran were rote and repetitive, so we decided to make machines run them for us. We scripted our tests in languages like Java, JavaScript, and Python. We developed a plethora of tools to interact with products, like Selenium for web browsers and Postman for APIs. Eventually, we executed tests as part of Continuous Integration systems so that we could get fast feedback perpetually.

With automation, things were great… mostly. We still needed to take time to develop the tests, but we could run them whenever we wanted. We could also run them in parallel to speed up turnaround time. Suites that took days to complete manually could be completed in hours, if not minutes, with automation.

Unfortunately, test automation is hard. Test automation is full-blown software development. Testers needed to become developers to do it right. Flakiness became a huge pain point. Sometimes, tests missed obvious problems that humans would easily catch, like missing buttons or poor styling. Many teams tried to automate their tests and simply failed because the bar was too high.

What we want is the speed and helpfulness of automation without the challenges in developing it. It would be great if a tool could look at an app, figure out its behaviors, and just go test it. That’s essentially what autonomous testing will do.

Ironically, traditional test automation isn’t fully automated. It still needs human testers to write the steps. Autonomous testing tools will truly be automated because the tool will figure out the test steps for us.

The car analogy

Cars are a great analogy for understanding the differences between manual, automated, and autonomous testing.
Manual testing is like a car with a manual transmission. As the driver, you need to mash the clutch and shift through gears to make the car move. You essentially become one with the vehicle.

Many classic cars, like this vintage Volkswagen Beetle, relied on manual transmissions. Beetle gear shifters were four-on-the-floor with a push-down lockout for reverse.

Automated testing is like a car with an automatic transmission. The car still needs to shift gears, but the transmission does it automatically for you based on how fast the car is going. As the driver, you still need to pay attention to the road, but driving is easier because you have one less concern. You could even put the car on cruise control!
Autonomous testing is like a self-driving car. Now, all you need to do is plug in the destination and let the car take you there. It’s a big step forward, and it enables you, now as a passenger, to focus on other things. Like with self-driving cars, we haven’t quite achieved full autonomous testing yet. It’s still a technology we hope to build in the very near future.

How will it work?

This probably goes without saying, but autonomous testing tools will need to leverage artificial intelligence and machine learning in order to learn an app’s context well enough to test it. For example, if we want to test a mobile app, then at the bare minimum, a tool needs to learn how phones work. It needs to know how to recognize different kinds of elements on a screen. It needs to know that buttons require tapping while input fields require text. Those kinds of things are universal for all mobile apps. At a higher level, it needs to figure out workflows for the app, like how humans would use it. Certain things like search bars and shopping carts may be the same in different apps, but domain specific workflows will be unique. Each business has its own special sauce.

This means that autonomous testing tools won’t be ready to use “out of the box.” Their learning models will come with general training on how apps typically work, but then they’ll need to do more training and learning on the apps they are targeted to test. For example, if I want my tool to test the Uber app, then the tool should already know how to check fields and navigate maps, but it will need to spend time learning how ridesharing works. Autonomous tools will, in a sense, need to learn how to learn. And there are three ways this kind of learning could happen.

Random trial and error

The first way is random trial and error. This is machine learning’s brute force approach. The tool could literally just hammer the app, trying to find all possible paths – whether or not they make sense. This approach would take the most time and yield the poorest quality of results, but it could get the job done. It’s like a Roomba, bonking around your bedroom until it vacuums the whole carpet.

Coaching from a user

The second way for the tool to learn an app is coaching from a user. Instead of attempting to be completely self-reliant, a tool could watch a user do a few basic workflows. Then, it could use what it learned to extend those workflows and find new behaviors. Another way this could work would be for the tool to provide a recommendation system. The tool could try to find behaviors worth testing, suggest those behaviors to the human tester, and the human tester could accept or reject those suggestions. The tool could then learn from that feedback: accepted tests signal good directions, while rejected tests could be avoided in the future.
Essentially, the tool would become a centaur: humans and AI working together and learning from each other. The human lends expertise to the AI to guide its learning, while the AI provides suggestions that go deeper than the human can see at a glance. Both become stronger through symbiosis.

Learning from observability data

A third way for an autonomous testing tool to learn app context acts like a centaur on steroids: learning from observability data. Observability refers to all the data gathered about an app’s real-time operations. It includes aspects like logging, system performance, and events. Essentially, if done right, observability can capture all the behaviors that users exercise in the app. An autonomous testing tool could learn all those behaviors very quickly by plunging the data – probably much faster than watching users one at a time.

What will a workflow look like?

So, let’s say we have an autonomous testing tool that has successfully learned our app. How do we, as developers and testers, use this tool as part of our jobs? What would our day-to-day workflows look like? How would things be different? Here’s what I predict.

Setting baseline behaviors

When a team adopts an autonomous testing tool for their app, the tool will go through that learning phase for the app. It will spend some time playing with the app and learning from users to figure out how it works. Then, it can report these behaviors to the team as suggestions for testing, and the team can pick which of those tests to keep and which to skip. That set then becomes a rough “baseline” for test coverage. The tool will then set itself to run those tests as appropriate, such as part of CI. If it knows what steps to take to exercise the target behaviors, then it can put together scripts for those interactions. Under the hood, it could use tools like Selenium or Appium.

Detecting changes in behaviors

Meanwhile, developers keep on developing. Whenever developers make a code change, the tool can do a few things. First, it can run the automated tests it has. Second, it can go exploring for new behaviors. If it finds any differences, it can report them to the humans, who can decide if they are good or bad. For example, if a developer intentionally added a new page, then the change is probably good, and the team would want the testing tool to figure out its behaviors and add them to the suite. However, if one of the new behaviors yields an ugly error, then that change is probably bad, and the team could flag that as a failure. The tool could then automatically add a test checking against that error, and the developers could start working on a fix.

Defining new roles

In this sense, the autonomous testing tool automates the test automation. It fundamentally changes how we think of test automation. With traditional automation, humans own the responsibility of figuring out interactions, while machines own the responsibility of making verifications. Humans need to develop the tests and code them. The machines then grind out PASS or FAIL. With autonomous testing, those roles switch. The machines figure out interactions by learning app behaviors and exercising them, while humans review those results to determine if they were desirable (PASS) or undesirable (FAIL). Automation is no longer a collection of rote procedures but a sophisticated change detection tool.

What can we do today?

Although full-blown autonomous testing is not yet possible, we can achieve semi-autonomous testing today with readily-available testing tools and a little bit of ingenuity. In the next article, we’ll learn how to build a test project using Playwright and Applitools Eyes that autonomously performs visual and some functional assertions on every page in a website’s sitemap file using Visual AI.

Applitools is working on a fully autonomous testing solution that will be available soon. Reach out to our team to see a demo and learn more!

That’s right, Test Automation University (TAU) is holding a conference! It’ll be a two-day virtual event from March 8–9, 2023 hosted by Applitools. We will sharpen our software quality skills as we go all-in for the testing community. This article gives you nine reasons why you should attend this awesome event.

#1: Find out what’s new with TAU

TAU is alive and kicking, and we’ve got some incredible new developments in the works. In my opening keynote address, I’ll dish out all the details for upcoming courses, community engagements, and learning paths. There might even be a few surprises!

#2: Sharpen your skills with tutorials

Our first day will be stacked with six two-and-a-half-hour tutorials so you can really hone your automation skills. Want to stack up Cypress and Playwright? Filip Hric and I have you covered. Need to drop down to the API layer? Julia Pottinger’s got the session for you. Unit testing? Tariq King will help you think inside the box. Wanna crank it up with load and performance testing? Leandro will show you how to use k6. How about some modern Selenium with visual testing? Anand Bagmar’s our man. It’s the perfect time to sharpen your skills.

#3: Hear inspiring talks from our instructors

There’s no doubt that our TAU instructors are among the best in software testing. We invited many of them to share their knowledge and their wisdom on our second day. Our program is designed to inspire you to step up in your careers.

#4: Test your testing skills

Want to challenge your testing skills together with your other TAU classmates? Well, Carlos Kidman will be offering a live “testing test,” where everyone gets to submit answers in real time. The student with the highest score will win a pretty cool prize, so be sure to study up!

#5: Algorave at the halftime show

Have you ever listened to algorithmically-generated music that is coded live in front of you? It’s an experience! Dan Gorelick will kick out the algorave jams during our halftime show. (Glow sticks not included.)

#6: Cram even more content with lightning talks

Lightning talks bring bite-sized ideas to the front stage in rapid succession. They’re a welcome change of pace from full-length talks. We’ll have a full set of lightning talks from a few of our instructors along with a joint Q&A where you can ask them anything!

#7: Compete in extracurricular games

Every university needs some good extracurricular activities, right? We’ll have a mid-day game where you, as members of the audience, will get to participate! We’re keeping details under wraps for now so that nobody gets an unfair advantage.

#8: Win awards like a homecoming champion

What would a celebration be without superlative awards? We will recognize some of our best instructors and students in our closing ceremony. You might also win some prizes based on your participation in the conference!

#9: Get your conference tickets for FREE!

Applitools is hosting the 2023 TAU Conference as a free community event for everyone to attend. Just like our TAU courses, you don’t need to pay a dime to participate. You don’t even need to be a TAU student! Come join us for the biggest block party in the testing community this year. Register here to reserve your spot.

This article is based on our recent webinar, How to Enhance UI/UX Testing by Leveraging AI, led by Chris Rolls from TTC and Andrew Knight from Applitools. Editing by Marii Boyken.

Last week, I hosted a webinar with Chris Rolls from TTC. In the webinar, Chris and I talked about the current state of software testing, where it’s going, and how visual AI and automation will impact the future of software testing. In this article, I’ll be recapping the insights shared from the webinar.

The current state of testing

Software testing is often seen in businesses as a necessary evil at the end of the software development lifecycle to find issues before they reach production. Chris and I strongly agree that software quality is crucial to modern businesses to help to achieve modern businesses’ goals and needs to be thought of throughout the software development lifecycle.

Digital transformation and continuous delivery

The largest and most relevant companies today have embraced digital transformation and technology to run their businesses and meet their customers’ needs. To keep up with digital transformation, you need modern software development practices like DevOps and continuous delivery. DevOps requires continuous testing to be successful, but we’re seeing that reliance on manual testing is the biggest challenge that organizations face when adopting DevOps.

Software testing and quality assurance

The software world is changing, so we need to change how we deliver technology. That requires modern software development approaches, which requires modern software testing and software quality approaches. Thankfully, testing and quality are far more top of mind now than in previous years.

From the audience poll results from our webinar, we see that continuous delivery is here to stay. When asked how often do you deploy changes to production, over 40% of people stated that they deploy either daily or multiple times per day. This wasn’t the case 10 to 20 years ago for most organizations unless technology was the product. Now, daily deployments are pretty common for most organizations.

Low rates of test automation

However, we’re seeing that getting high test automation coverage is still a huge challenge. In the survey, 55% of the respondents automate less than half of their testing.

The numbers may have a bit of sample bias because Applitools users actually automate on average 1.7 times higher than other respondents. The responses align with anecdotal experience that a lot of organizations are still in lower test automation coverage around 20% to 50%.

Testing complexity and the amount of testing needed are going up increasingly, and this shows in our survey as well. More than 50% of the respondents test two languages or more, three browsers or more, three devices or more, and two applications or more.

With two thirds of the respondents saying that one of the hardest parts of testing UIs is that they are constantly changing, traditional automation tools can’t handle testing at that speed scale and complexity.

Using AI in testing

We know that there’s a lot of excitement around AI tools, and the survey shows that.

When asked What parts of the testing process are you supporting or planning on supporting with AI, test offering, test prioritization, test execution, test management, and visual regression we all mentioned. The top two answers among respondents were test execution and visual regression.

It’s important to remember that continuous testing is about more than just test automation.

While test automation is key to the process, you still need to incorporate other software testing and software quality practices like load testing, security, user experience, and accessibility.

The future of testing

What we’re trying to achieve in the future of testing is to support modern software development practices. The best way we’re seeing to do this is to have software testing and software quality more tightly integrated into the process. Let’s talk about what a modern software approach looks like.

Testing sooner and more frequently

To get tighter integration of quality into the process, testing can’t just be an activity at the end of the development lifecycle. Testing has to happen continuously for teams to be able to provide fast feedback across disciplines and ensure a quality product. When this is done, we see increased speed not just of testing, but of overall software development and deployment. We also see reduced costs and increased quality.  More defects are found before production, and we see quicker responses when finding defects in production.

Increasing how much testing is automated

Traditional testing approaches tend to be done mostly manually. Increasing test coverage doesn’t mean manual testing goes away.

Automating your test cases frees up time to do more exploratory testing. Exploratory testing should be assisted by different tools, so AI has a good role to play here. Tools like ChatGPT are useful to brainstorm things like what to test next. Obviously we want to increase test automation coverage at all levels, including unit, API, and UI. Intelligent automated UI tests provide us more information than functional tests alone.

Knowing when to use AI in testing

What does the future of testing with AI look like? It’s a combination of people, processes, and technology. Software testers need to be thinking about what skills we need to have to support these new ways of testing and delivering quality software.

We need to uncover if a use case is better served by AI and machine learning than an algorithmic solution. To do this, we need to ask the following questions:

• What is the pain point that we are trying to solve?
• Are there good algorithmic or heuristic solutions available to address the pain point?
• Are there enterprise ready AI and machine learning solutions that help?
  • If there are solutions, what happens if the AI doesn’t work?
  • If there aren’t solutions yet, how do we position ourselves to be ready for them?

“It’s quite trendy to talk about artificial intelligence, but the reason why we’re partnered with Applitools is that they apply real machine learning and artificial intelligence to a problem that is not well solved by other types of solutions on the market.”

Chris Rolls, CEO, Americas, TTC

Integrating visual AI into test automation

Let’s talk about how we can integrate AI into our testing to get some of those advantages of increased speed and coverage discussed earlier.

What is visual AI?

I like to explain visual AI visually. Do you remember those spot-the-difference pictures we had in our activity books from when we were kids?

As humans, we could sit around and play with this to find the differences manually. But what we want is to be able to find these differences immediately. And that is what visual AI has the power to do.

Even when the images are a little bit skewed or off by a couple pixels here or there, visual AI can pinpoint any differences between one view and another.

Now you might be thinking, Andy, that’s cute, but how’s this technology gonna help me in the real world? Is it just gonna solve little activity book problems? Well, think about all the apps that you would develop – whether web, mobile, desktop, whatever you have – and all the possible ways that you could have visual bugs in your apps.

Here we’ve got three different views from mobile apps. One for Chipotle, one for a bank, and another one for a healthcare provider.

Visual AI test automation

You can see that visual bugs are pervasive and they come in all different shapes and sizes. Sometimes the formatting is off, sometimes a particular word or phrase or title is just nulled out. What’s really pesky is that sometimes you might have overlapping text.

Traditional automation struggles to find these issues because traditional automation usually hinges on text content purely or on particular attributes of elements on a page. So as long as something appears and it’s enactable, most traditional scripts will pass. Even though we as humans visually can inspect and see when something is completely broken and unuseable.

This is where visual AI can help us, because what we can do is we can take snapshots of our app over time and use visual AI to detect when we have visual regressions. Because if, let’s say, one day the title went from being your bank’s name to null, it’ll pick it up right away in your continuous testing.

Visual AI in action

In the webinar, I gave a live demo of automated visual testing using Applitools Eyes. In case you missed the demo, you can check it out here:

The future will be autonomous

So all this really cool stuff is powered by visual AI, a real world application of AI looking at images and being able to find things in them like they were humanized. Now you may think this is really cool, but what’s even cooler is this is just the beginning of what we can do with the power of AI and machine learning in the testing and automation space.

What we’re going to be seeing in the next couple years is a new thing called autonomous testing where not only are we automating our tests, but we’re automating the process of developing and maintaining our tests. The tests are kind of almost writing themselves in a sense. And visual AI is going to be a key part of that, because if testing is interaction and verification, what we want to make autonomous is both interaction and verification. And visual AI has already made verification autonomous. We’re halfway there, folks.

Be sure to check out our upcoming events page for new webinars coming soon!Learn how Applitools Eyes uses AI to catch visual differences between releases while reducing false positives. Happy testing!

Read more

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

In this final part of our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series, we will cover the audience Q&A, sharing the most popular questions from the audience and the answers our experts gave. Be sure to read our previous post.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

• Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
• Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
• Tally Barak (Playwright) – Software Architect at YOOBIC
• Steve Hernandez (Selenium) – Software Engineer in Test at Q2
• Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

ICYMI: Cypress ambassador Filip Hric shared What’s New in Cypress 12, which includes an update on the feature just discussed.

Conclusion

This article concludes our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series. We got to hear from engineers at Mercari, YOOBIC, Hilton, Q2, and Domino’s about how their teams build their test automation projects and why they made their framework decisions. Our panelists also shared insights into advantages and disadvantages they’ve encountered in their test automation frameworks. If you missed any previous part of the series, be sure to check them out:

• Let the Engineers Speak! Part 1: Real-World Projects
• Let the Engineers Speak! Part 2: The Biggest Challenges
• Let the Engineers Speak! Part 3: Favorite Test Integrations
• Let the Engineers Speak! Part 4: Changing Frameworks

In part 4 of our Cypress, Playwright, Selenium, or WebdriverIO? Let The Engineers Speak recap series, we will recap the stories our panelists have about changing their test frameworks – or not! Be sure to read our previous post where our panelists talked about their favorite test integrations.

The experts

I’m Andrew Knight – the Automation Panda – and I moderated this panel. Here were the panelists and the frameworks they represented:

• Gleb Bahmutov (Cypress) – Senior Director of Engineering at Mercari US
• Carter Capocaccia (Cypress) – Senior Engineering Manager – Quality Automation at Hilton
• Tally Barak (Playwright) – Software Architect at YOOBIC
• Steve Hernandez (Selenium) – Software Engineer in Test at Q2
• Jose Morales (WebdriverIO) – Automation Architect at Domino’s

The discussion

Audience Q&A

So we had a fantastic conversation amongst each of our panel speakers about the considerations they had as their test projects grew and changed. We found that most of our panelists have had to make changes to their test frameworks used as their projects’ needs changed. In the next article, we’ll cover the most popular questions raised by our audience during the webinar.