The latest edition of the Future of Testing events, held on November 7, 2023, was nothing short of inspiring and thought-provoking! Focused on AI in Automation, attendees learned how to leverage AI in software testing with top industry leaders like Angie Jones, Tariq King, Simon Stewart, and many more. All of the sessions are available now on-demand, and below, we take a look back at these groundbreaking sessions to give you a sneak peek of what to expect before you watch.

Opening Remarks

Joe Colantonio from TestGuild and Dave Piacente from Applitools set the stage for a thought-provoking discussion on reimagining test automation with AI. As technology continues to evolve at a rapid pace, it’s important for software testing professionals to adapt and embrace new tools and techniques. Joe and Dave encouraged attendees to explore the potential of AI in test automation and how it can enhance their current processes. They also touch upon the challenges faced by traditional test automation methods and how AI-powered solutions can help overcome them.

Dave shared one of our latest updates – the integration of Applitools Eyes with Preflight! Learn more about Preflight.

Keynote—Reimagining Test Automation with AI by Anand Bagmar

In this opening session, Anand Bagmar explored how to reimagine your test automation strategies with AI at each stage of the test automation life cycle, including a live demo showcasing the power of AI in test automation with Applitools.

Anand first introduced the transition from Waterfall to Agile software delivery practices, and while we can’t imagine going back to a Waterfall way of working, he addressed the challenges Agile brings to the software testing life cycle. Each iteration brings more room for error across analysis, maintenance, and validation of tests. This is why testers should turn toward AI-powered test automation, with the help of tools like Applitools, to help ease the pain of Agile testing.

The session is aimed at helping testers understand the importance of leveraging AI technology for successful test automation, as well as empowering them to become more effective in their roles. Watch now.

From Technical Debt to Technical Capital by Denali Lumma

In this session, Denali Lumma from Modular dived into the concept of technical debt and proposed a new perspective on how we view it – technical capital. She walked attendees through key mathematical concepts that help calculate technical capital, as well as examples comparing Pytorch vs. TensorFlow, MySQL vs.Postgres, Frameworks vs. Code Editors, and more.

Attendees gained insights into calculating technical capital and how it can impact the valuation of a company. Watch now.

Automating Quality: A Vision Beyond AI for Testing by Tariq King

Tariq King of EPAM Systems took attendees on a journey through the evolution of software testing and how it has been impacted by generative AI. He shared his vision for the future of automated quality, one that looks beyond just AI to also prioritize creativity and experimentation. Tariq emphasized the need for quality and not just using AI to “go faster.” The more quality you have, the more productive you will be.

Tariq also dove into the ethical implications of using AI for testing and how it can be used for good or evil. Watch the full session.

Leveraging ChatGPT with Cypress for API Testing: Hands-On Techniques by Anna Patterson

In this session, Anna Patterson of EVERFI explored practical techniques and provided hands-on examples of how to harness the combined power of Cypress and ChatGPT to create robust API tests for your applications.

Anna guided us through writing descriptive and clear test prompts using HTML status codes, with a pet store website as an example. She showed in real-time how meaningful prompts in ChatGPT can help you create a solid API test suite, while also considering the security requirements of your company. Watch now.

PANEL—Testing in the AI Era: Opportunities, Hurdles, and the Evolving Role of Engineers

Joe Colantonio, Test Guild • Janna Loeffler, mParticle • Dave Piacente, Applitools • Stephen Williams, Accenture

As the use of AI in software development continues to grow, it is important for engineers and testers to stay ahead of the curve. In this panel discussion led by Joe Colantonio from Test Guild, Janna Loeffler from mParticle, Dave Piacente from Applitools, and Stephen Williams from Accenture came together to discuss the current state of AI implementation and its impact on testing.

They talked about how AI is still in its early stages of adoption and why there may always be some level of distrust in AI technology. The panel emphasized the importance of first understanding why you might implement AI in your testing strategy so that you can determine what the technology will help to solve vs. jumping in right away. Many more incredible takes and insights were shared in this interactive session! Watch now.

The Fear Factor with Richard Bradshaw

The Friendly Tester, Richard Bradshaw, addressed the common fears about AI and automation in testing. Attendees heard Richard’s open and honest discussion on the challenges and concerns surrounding AI and automation in testing. Ultimately, he calmed many fears around AI and gave attendees a better understanding of how they can begin to use it in their organization and to their own advantage. Watch now.

Tests Too Slow? Rethink CI! by Simon Stewart

Simon Stewart from the Selenium Project discussed the latest updates on how to speed up your testing process and improve the reliability of your CI runs. He shared insights into the challenges and tradeoffs involved in this process, as well as what is to come with Selenium and Bazel.
Attendees learned how to rethink their CI approach and use these tools to get faster feedback and more reliable testing results. Watch now.

Revolutionizing Testing: Empowering Manual Testers with AI-Driven Automation by Dmitry Vinnik

Dmitry Vinnik explored how AI-driven automation is revolutionizing the testing process for manual testers. He showed how Applitools’ Visual AI and Preflight help streamline test maintenance and reduce the need for coding.

Dmitry shared the importance of test maintenance, no code solutions for AI testing, and a first-hand look at Applitools Preflight. Watch this session to better understand how AI is transforming testing and empowering manual testers to become more effective in their roles. Watch the full session.

Keynote—Where Is My Flying Car?! Test Automation in the Space Age by Angie Jones

In her closing keynote, Angie Jones of Block took us on a trip into the future to see how science fiction has influenced the technology we have today. The Jetsons predicted many futuristic inventions such as robots, holograms, 3D printing, smart devices, and drones. We will explore these predictions and see how far we have come regarding automation and technology in the testing space.

As technology continues to evolve, it is important for testers to stay updated and adapt their strategies accordingly. Angie dove into the exciting world of tech innovation and imagined the future for test automation in the space age. Watch now.

Visit the full Future of Testing: AI in Automation on-demand archive to watch now and learn actionable steps to implement AI in your software testing strategy, key considerations before you start, other ideas around ethics and philosophical considerations, the importance of quality and security, and much more.

The post Future of Testing: AI in Automation Recap appeared first on Automated Visual Testing | Applitools.

Scaling your test automation initiatives can be daunting. In a recent webinar, Test Automation at Scale: Lessons from Top Performing Distributed Teams, panelists from Accenture, Bayer, and Eversana shared their insights for overcoming common challenges. Here are their top recommendations.

Establish clear processes for collaboration.
Daily standups, sprint planning, and retrospectives are essential for enabling communication across distributed teams. “The only way that you can build a quality product that actually satisfies the business requirements is [through] that environment where you’ve got the different teams coming together,” said Ariola Qeleposhi, Test Automation Lead at Accenture.

Choose tools that meet current and future needs.
Consider how tools will integrate and the skills required to use them. While a “one-size-fits-all” approach may seem appealing, it may not suit every team’s needs. Think beyond individual products to the overall solution, advised Anand Bagmar, Senior Solution Architect at Applitools. Each product team should have a test pyramid, and tests should run at multiple levels to get real value from your automation.

Start small and build a proof of concept.
Demonstrate how automation reduces manual effort and finds defects faster to gain leadership buy-in. “Proof of concepts will really help to provide a form of evidence in a way to say that, okay, this is our product, this is how we automate or can potentially automate, and what we actually save from that,” said Qeleposhi.

Consider a “quality strategy” not just a “test strategy.”
Involve all roles like business, product, dev, test, and DevOps. “When you think about it as quality, then the role does not matter,” said Bagmar.

Leverage AI and automation as “seatbelts,” not silver bullets.
They enhance human judgment rather than replace it. “Automation is a lot, at least in this instance, it’s like a seatbelt. You don’t think you’ll need it, but when you need it you better have it,” said Kyle Penniston, Senior Software Developer at Bayer.

Build, buy, and reuse.
Don’t reinvent the wheel. Use open-source tools and existing frameworks. “There will be great resources that you can use. Open-source resources, for example, frameworks that might be there that you can use to quickly get started and build on top of that,” said Bagmar.

Provide learning resources for new team members.
For example, Applitools offers Test Automation University with resources for developing automation skills.

Measure and track metrics to ensure value.
Look at reduced manual testing, faster defect finding, test coverage, and other KPIs. “You need to get some metrics really, and then you need to use that from an automation side of things,” said Qeleposhi.

The key to building a solid foundation for scaling test automation is taking an iterative, collaborative approach focused on delivering value and enhancing quality. With the right strategies and tools in place, teams can overcome common challenges and achieve automation success. Watch the full recording.

The post Driving Successful Test Automation at Scale: Key Insights appeared first on Automated Visual Testing | Applitools.

As a test automation engineer, finding the right tools and frameworks is crucial to building a successful test automation strategy. Playwright is an end-to-end testing framework that provides a robust set of features to create fast, reliable, and maintainable tests.

In a recent webinar, Playwright Ambassador and TAU instructor Renata Andrade shared several use cases and best practices for using the framework. Here are some of the most valuable takeaways for test automation engineers:

Use Playwright’s built-in locators for resilient tests.
Playwright recommends using attributes like “text”, “aria-label”, “alt”, and “placeholder” to find elements. These locators are less prone to breakage, leading to more robust tests.

Speed up test creation with the code generator.
The Playwright code generator can automatically generate test code for you. This is useful when you’re first creating tests to quickly get started. You can then tweak and build on the generated code.

Debug tests and view runs with UI mode and the trace viewer.
Playwright’s UI mode and VS Code extension provide visibility into your test runs. You can step through tests, pick locators, view failures, and optimize your tests. The trace viewer gives you a detailed trace of all steps in a test run, which is invaluable for troubleshooting.

Add visual testing with Applitools Eyes.
For complete validation, combine Playwright with Applitools for visual and UI testing. Applitools Eyes catches unintended changes in UI that can be missed by traditional test automation.

Handle dynamic elements with the right locators.
Use a combination of attributes like “text”, “aria-label”, “alt”, “placeholder”, CSS, and XPath to locate dynamic elements that frequently change. This enables you to test dynamic web pages.

Set cookies to test personalization.
You can set cookies in Playwright to handle scenarios like A/B testing where the web page or flow differs based on cookies. This is important for testing personalization on websites.

Playwright provides a robust set of features to build, run, debug, and maintain end-to-end web tests. By leveraging the use cases and best practices shared in the webinar, you can power up your test automation and build a successful testing strategy using Playwright. Watch the full recording and see the session materials.

The post Power Up Your Test Automation with Playwright appeared first on Automated Visual Testing | Applitools.

Test automation is crucial for any software engineering team to ensure high-quality releases and a smooth software development lifecycle. However, test automation efforts can often be tedious, time-consuming, and require specialized skills. New AI tools are emerging that can help accelerate test automation, handle flaky tests, increase test coverage, and improve productivity.

In a recent webinar, Rizel Scarlett and Anand Bagmar discussed how to leverage AI-powered tools like GitHub Copilot and Applitools to boost your test automation strategy.

GitHub Copilot can generate automated tests.

By providing code suggestions based on comments and prompts, Copilot can help quickly write test cases and accelerate test automation development. For example, a comment like “validate phone number” can generate a full regular expression in seconds. Copilot also excels at writing unit tests, which many teams struggle to incorporate efficiently.

Applitools Execution Cloud provides self-healing test capabilities.

The Execution Cloud allows you to run tests in the cloud or on your local machine. With self-healing functionality, tests can continue running successfully even when there are changes to web elements or locators. This helps reduce flaky tests and maintenance time. Although skeptical about self-healing at first, the experts found Applitools to handle updates properly without clicking incorrect elements.

Together, tools like Copilot and Applitools can transform your test automation.

Copilot generates the initial test cases and Applitools provides a self-healing cloud environment to run them. This combination leads to improved productivity, reduced flaky tests, and increased coverage.

Applitools Eyes and Execution Cloud offer innovative AI solutions for automated visual testing. By leveraging new technologies like these, teams can achieve test automation at scale and ship high-quality software with confidence. To see these AI tools in action and learn how they can benefit your team, watch the full webinar recording.

The post AI-Powered Test Automation: How GitHub Copilot and Applitools Can Help appeared first on Automated Visual Testing | Applitools.

The post Resources to Help You Stay on Top of Testing Trends and Techniques appeared first on Automated Visual Testing | Applitools.

The post How to Tell Your Manager Why You Should Attend TAU Conference 2023 appeared first on Automated Visual Testing | Applitools.

Automating different types of tests

To illustrate our different types of test automation, we’ll be using the example project Cypress Heroes. In this full-stack TypeScript app, users can take the following actions:

• Log in with an email and password
• Like heroes, which increments the hero’s number of fans
• Hire heroes, which increments the hero’s number of saves
• Manage hero profile information like name, superpowers, and price

ICYMI: Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see Ely Lucas from Cypress demo the example project.

End-to-end testing

Cypress is traditionally known for end-to-end testing. You automate user interactions for specific scenarios from start to finish in the browser, and then run functional assertions to check the state of elements at each step. End-to-end tests are hidden in an actual web server and hit the site just like a user would.

Measurable stats for code coverage of your end-to-end testing can act as a health metric for your website or app. Adding coverage reports to your automation pipeline as commits can help ensure you’re testing all parts of your code.

Component testing

If you’re using a component-based framework like React or Angular or a design system like Storybook, you can also do component testing to test UI components. In this example, we have a button component with a few tests that pass, the hero card test, and a test for the login form. These components are being mounted in isolation outside of your typical web server.

Think of component tests as “UI unit” tests. While they don’t give end-to-end coverage, they’re quick and easy to run.

API testing

For your back end, you’ll need to automate API tests. The example project is using a community-built plugin called cypress-plugin-api. This plugin provides an interface inside the Cypress app to test APIs. It’s really cool and it’s super fun, and it allows you to write tests that you would have to do manually in a tool like Postman.

Fun fact: Cypress Ambassador Filip Hric developed the cypress-plugin-api. Check out Filip’s Test Automation University courses.

The API tests in our example are in the separate server project. We can use the command npx cypress open, and then we can run those tests in Chrome. We can see all of our results that we’re getting the response of the status codes. We can view a post request, the headers that were sent, the headers that were returned, and other stuff that you normally get from a tool like Postman.

And it’s just baked into the app, which is really nice. Cypress is basically a web app that tests a web app. And so, you could extend Cypress as an app with things like this to help you do your testing and to have it all seamlessly integrated.

Running a pipeline with GitHub Actions

The example project uses GitHub Actions to set up the test automation pipeline. When working on smaller projects, it’s easy to have CI interactions baked into your repository, all in one place.

Configuring your GitHub Action

With GitHub Actions, you declare everything you need in a YAML file in the .github/workflows folder. Your actions become part of your repository and are covered by version control. If you make any changes, you can review them easily with a simple line-by-line diff. GitHub Actions make it easy to automate processes alongside other interactions you make with your repository. For example, if you open a pull request, you can have it automatically kick off your tests and do linting. You can even perform static code analysis before merging changes.

Some environment variables are set at the top of the YAML file. The API URL is what the client app uses to communicate to the API. The example app is hooked up to send test results to the Cypress Cloud. Those results can then be used for analytics, diagnostics, reporting, and optimizing our test workflows. The Cypress cloud also requires a GitHub token, so it can do things like correctly identify what pull request is being merged.

For those new to GitHub Actions: You can define environment variables per step in a job, but declaring them at the top helps you update them painlessly.

Running each of our tests

To keep things simple, there is only one job right now in this GitHub Action. First, it checks out the code straight from GitHub. Next, it builds the project using the Cypress GitHub Action. The Cypress GitHub Action does a few things for you like building your application or npm installing or yarn installing the dependencies.

Building first means that subsequent jobs don’t have to build the app again. We’ve set run test to false, which is a parameter to the Cypress GitHub Action, because we don’t want to run the tests here. We’ll be running the tests separately below.

We have our component tests in our GitHub Action. We tell it to install false, since we installed it up above. And then we run our custom test command, which opens Cypress in run mode and initiates component testing. This record tells the GitHub Action to send the results to Cypress Cloud.

And then we have to start the client and server. For both end-to-end tests and API tests, the application must be up and running component tests. For the end-to-end test and API tests, the example app is hitting live servers.

This run command will start both the React app and the Node server, and then it will run the end-to-end test. We’re telling it again to not install the dependency, since it was already installed. Then, we’re running the command to start the end-to-end testing. The wait command will wait to make sure that both the client URL and the API URL are both up and running before it will start the test. If the test starts before both URLs get up and running, you’ll have some tests fail.

Another thing that the Cypress GitHub Action does is that you have the option to wait for these services to be live before the testing starts. By default, the npm run test commands are going to use the Chromium browser built into Electron. If you want to test on other browsers, you must make sure those browsers are installed on the runner. Cypress provides Docker images that you can add to your configuration to download the different browsers. However, downloading additional browsers increases the file size and makes the runs take longer.

Make sure that the Cypress binary itself is downloaded and installed. It’s going to run headlessly. This is because the command set up in these scripts is run mode, which is headless, whereas open mode is with the UI.

And then it will run the API test, which is very similar to end-to-end tests, except that since we’re not hitting the actual client app – only hitting the API app – we’re only waiting to make sure that the API URL is up and running.

If you write test cases per the local database for end-to-end testing before pushing to GitHub Actions, someone else running those test cases on their system could potentially fail. In whatever kind of test automation you develop, you’ll need to handle test data properly to avoid collisions. There are many different strategies you can follow. For more information on this and solving sample data dependency, watch my talk Managing the Test Data Nightmare.

How long do the test suites take?

When running your tests with Cypress and GitHub Actions, the results are uploaded to Cypress Cloud. You can go into Cypress Cloud and actually watch replays of all these tests that happened. The entire pipeline run in the example was 3 minutes and 50 seconds for all three test suites.

The individual test suites we ran took the following times:

• Component tests: 49 seconds
• End-to-end tests: 1 minute and 18 seconds
• API tests: 13 seconds

Improving test coverage with visual assertions

Since all the Cypress tests are run inside of the browser window, you can visually see them and inspect to make sure that they’re looking correctly. But this type of review is a manual step. If someone accidentally makes a change to the stylesheet, the site could no longer be running properly, but if we run the tests, they’ll pass.

We can use Applitools Eyes to fix this issue.

Visual testing is meant to automate the things that traditional automation is not so good at. For example, as long as particular IDs on your page are in the DOM somewhere, your traditional automation scripts with something like Cypress are still going to find and interact with the elements. Applitools Eyes uses visual AI to look at an app and be able to detect these kinds of visual differences that traditional assertions struggle to capture. Let’s add some visual snapshots to these end-to-end tests.

Adding Applitools to your project

First, you’ll need an Applitools account. You can register a free Applitools account with your GitHub username or your email, and you’ll be good to go. You’ll need your Applitools API key for when we run tests with Applitools.

Next, we’ll need to install the Applitools SDK using npm install @applitools/eyes-cypress.

It can be a dev dependency or it can be a regular dependency – whichever you prefer. In the example project, we use a dev dependency. In the example project, we’re using the Applitools Eyes SDK for Cypress, we have Applitools SDKs for basically every tool framework you got.
Next, we’ll need to create an Applitools configuration file. Where in Cypress projects, you have your cypress.config.js file, basically we want one that’s called applitools.config.js.

Configuring your Applitools runner

In the Applitools config file, we will specify the configuration for running visual tests. There’s a separation between declaring configuration and actually adding test steps.
One of the settings we want is called batchName, and we’re going to set that to “cy heroes visual tests” to reflect the name of our demo app. The batch name will appear in the Eyes Test Manager (or the Applitools “dashboard”) after we run our visual tests.

Next, we’ll set the browsers. This will be a list, with each item being an entry that specifies a browser configuration, including name, width, and height.

Typically, since Cypress runs inside of an Electron app, it can be challenging to test mobile browsers. However, the Applitools Ultrafast Grid enables us to render our visual snapshots on mobile devices. The settings for mobile devices are going to be a bit different than those for browsers. Instead of having a name, we’re going to have a device name.

Our applitools.config.js file is complete. When we run our tests – either locally or in the GitHub Action – Applitools will render the snapshots it captures on these four browser configurations in the Ultrafast Grid and report results using the batch name. Furthermore, the local platform doesn’t matter. Even if you run this test on Windows, the Ultrafast Grid can still render snapshots on Safari and mobile emulators. A snapshot is just going to be a capture of that full page. Applitools will do the re-rendering with the appropriate size, the appropriate browser configuration, and all that will happen in the cloud. Essentially you can do multi-browser and multi-platform testing with simple declarations.

Now that we have completed the configuration, let’s update the tests to capture visual snapshots, starting with the homepage.

Setting up our test suites

You need to make sure that your tests aren’t interfering with other tests. In these tests, we’re going through and modifying some of the heroes that are in the application. The state of the application changes per test, so to get around that, we’re creating a new hero just for working with our tests and deleting the hero after the tests.

In the example, we’re using Cypress tasks, which is code that actually runs on the Node process part of Cypress. It’s directly communicating with our database to add the hero, delete the hero, and all the other types of setup tasks that we want to do before we actually run our test.

So it’s going to happen for each of the tests, and then we’re visiting the homepage and getting access to the hero.

We get our new hero and then we call cy.deleteHero, which is going to call the database to delete the hero. From the describe block at the start of every test, we get our hero. And then, finally, we have the hero card by its name, and we find the button that has the right selectors, so we can actually select it and click the button.

This test is making sure that you’re logged in before you can like this hero. We’re making sure that the modal popped up, clicking the okay on the modal, and then making sure that modal disappears and does not exist anymore.

Down below we have another suite for when a normal user is logged in. And so we’re using a custom Cypress command to log in with this username and password. You can define these custom commands that are like making your own function, encapsulating a little bit of logic so that it could be reusable.

So what we’re doing to test the login is going to the homepage, running the login process, and verifying the login was actually successful. The cy.session is caching a session for us to restore the session later from cookies. This helps speed up your test so you’re not having to go through the whole flow of actually logging in again.

We have another suite here for when an admin user is logged in, because an admin user can edit users and delete heroes.

 In the example, negative login tests – where you use the wrong username and/or password – are under the component tests.

In the login form component test, when an email and password is invalid, an error should show. The example uses cy.intercept to mock the API request that goes to the cert, which goes to the off endpoint and returns a status code 401, which represents an invalid login.

You can either write a component test or an end-to-end test. In this case, a component test makes it easier to set up the mock data.

Adding a call to Applitools Eyes

With the test suites set up, we’re ready to add some visual snapshots here. We need to call an Eyes session using the Applitools Eyes SDK. The idea is that we open our eyes, and we can take visual snapshots. And then at the end of the test, we will close our eyes to say that we’ve captured all the snapshots for that session or for that test. And at that point, Applitools Eyes will upload the snapshots that are captured to the Applitools Eyes server, do all of the re-rendering of the things of those four browsers in the Ultrafast Grid. Then we can log into the Applitools dashboard and we can see exactly what happened with our testing.
To get the autocomplete for Eyes commands, we need to set up the Applitools Eyes stuff with the Cypress project. We already did npm install on the package, so we’ll need to run npx eyes-setup.

We’ll want to use the command cy.eyesOpen in the homepage describe block under the beforeEach method. We want to pass an app name and test name for logging and reporting purposes. You might also put their Cypress eyes open code in the beforeEach of the test cases, so the call doesn’t need to be duplicated.

Then, in the afterEach block, you’ll call cy.eyesClose.

In this test, you must log in, make sure that the modal pops up, log in, and then click okay in the modal and make sure the modal disappears, so we’ll need a snapshot when the modal is up and one when the modal goes away. In this case, we’ll capture the whole window.

If we didn’t want to capture everything, we could actually capture a region, like a div or even an individual element. On a small scale, using the region option does not make a measurable difference in execution speed, but it gives you a way to tune the type of snapshot we want.

For capturing the next step, we can basically copy the whole call there and paste it, changing the tag to homepage with the modal dismissed.

These snapshots are very straightforward to write, and something that we could consider is that some of those other assertions you might arguably be able to remove. The visual snapshot is going to capture everything on that window, so if it’s there and visible, we’re going to capture it and track it over time.

You would still need to keep all of your interactions, but you can remove most of your assertions checking visible elements. However, there are certainly things where if you want to check a very specific numeric value, you still want to keep those assertions.

Running the updated tests

All we need to do to run this test is make sure that we have our Applitools API key from our account saved as an environment variable of the Cypress application.

Note: If you happen to steal someone’s API key, it doesn’t really help you. It just means they’ll see your results, and you won’t. API keys should be kept secret and safe.

Using the Applitools Eyes dashboard

So to see the visual testing results, we will need to view them in the Applitools Eyes dashboard.

You can view your test results in a grid to see the UI quickly, or you can view your results in a list to see your configurations quickly.

On the left, you’ve got the batch name that was set. Then on the main part of the body, you’ll see there are actually four tests. We only wrote one test, but each test is run once per browser configuration we specified, providing cross-browser and cross-platform testing without additional steps.

If we open up the snapshots, you can see the two snapshots that we captured. These results are new, because this is the first time we’ve run the test.

We’ve established the snapshot as a baseline image, meaning anything in the future will be checked against that.

That’s where that visual aspect of the testing comes in. Your Cypress results will essentially tell you if it was bare bones basic functional, and then Eyes will tell you what it actually looked like. You get richer results together.

Resolving test results in the dashboard

Let’s see what this looks like if we make that visual change.

In the main file, we’ve updated the stylesheet and run the test again. There is no need to do anything in the Applitools Eyes dashboard before re-running the test.

The new test batch is in an unresolved state because Eyes detected a visual difference. In theory, a visual difference could be good or bad. You could be making an intentional change. Visual AI is basically a change detector that makes it obvious to you, the human, to decide what is good or bad. Then anytime Applitools Eyes sees the same kind of passing or failing behavior in the future, it’ll remember.

It’s important to note that the unresolved test results won’t stop your test automation job or your automation flow. Test automation would complete normally. You as the human tester would review visual test results in the Eyes Test Manager (the “dashboard”) afterwards. The pipeline would not wait for you to manually mark visual test results.

Let’s open up one of those snapshots so we can see it full screen.

In the upper left, below the View menu in the ribbon, there’s a dropdown to show both so that you can see the baseline and test side by side.

In the example, we had removed the stylesheet, so we can see very clearly that it’s very different. It’s not always this obvious. In this case, pretty much the whole screen is different. But if it were like a single button that was missing or something shunted a little bit, it would show that a specific area was different. That’s the power of the visual AI check.

Whenever Applitools detects a visual change, you can mark it as “passing” with a thumbs-up. Then that snapshot automatically becomes the new baseline against which future checkpoints are compared. Applitools will go to the background and track similar images. And it will automatically update those appropriately as well.

Note: If you ever want to “reset” snapshots, you can also delete the baselines and run your tests “fresh” as if for the first time. The snapshots they capture will automatically become new baseline images.

Once we’ve resolved all test results, we’ll need to save. And now if we were to rerun our test again, Applitools Eyes would see the new snapshots and pass tests as appropriate. If you have dynamic content or test data, you add region annotations, which will ignore anything in the region box.
It is possible to compare your production and staging environments. You can use our GitHub Integration to manage different branches or versions of your application. We also support different baselines for A/B testing.

Closing thoughts

That’s basically how you would do visual testing with Applitools and Cypress. There are two big points to remember if you want to add visual testing to your own test suites:

• To get these tests running in your pipeline, the only change you’d have to make is to inject the Applitools API key in those environment variables.
• We didn’t really add a fourth suite of tests. Visual testing is more of a technique or an aspect of testing, not necessarily its own category of tests. All you have to do is work in the SDK, capture some snapshots, and you’re good to roll.

We hope this guide has helped you to build out your test automation pipeline to be more reliable and scalable. If you liked the guide, check out our Applitools tutorials for other guides on building your test automation pipeline. Watch the on-demand recording of Future-Proofing Your Automation Pipeline to see the full walkthrough. To keep up-to-date with test automation, you can peruse our latest courses taught by industry-leading testing experts on Test Automation University. Happy testing!

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

The post Future-Proofing Your Test Automation Pipeline appeared first on Automated Visual Testing | Applitools.

Cypress vs. Playwright: The Rematch

One of our most popular series is Let the Code Speak, where we compare testing frameworks in real examples. In our rematch of Let the Code Speak: Cypress vs. Playwright, Andrew Knight and Filip Hric dive deeper to how Cypress and Playwright work in practical projects. Quality Engineer Beth Marshall moderates this battle of testing frameworks while Andy and Filip explore comparisons of their respective testing frameworks in the areas of developer experience, finding selectors, reporting, and more.

Automating Testing in a Component Library

Visual testing components allows teams to find bugs earlier, across a variety of browsers and viewports, by testing reused components in isolation. Software Engineering Manager David Lindley and Senior Software Engineer Ben Hudson joined us last year to detail how Vodafone introduced Applitools into its workflow to automate visual component testing. They also share the challenges and improvements they saw when automating their component testing.

When to Shift Left, Move to Centre, and Go Right in Testing

Quality isn’t limited to the end of the development process, so testing should be kept in mind long before your app is built. Quality Advocate Millan Kaul offers actionable strategies and answers to questions about how to approach testing during different development phases and when you should or shouldn’t automate. Millan also shares real examples of how to do performance and security testing.

You, Me, and Accessibility: Empathy and Human-Centered Design Thinking

Inclusive design makes it easier for your customers with your varying needs and devices are able to use your product. Accessibility Advocate and Crema Test Engineer Erin Hess talks about the principles of accessible design, how empathy empowers teams and end users, and how to make accessibility more approachable to teams that are newer to it. This webinar is helpful all team members, whether you’re a designer, developer, tester, product owner, or customer advocate.

Erin also shared a recap along with the audience poll results in a follow-up blog post.

Future of Testing October 2022

Our October Future of Testing event was full of experts from SenseIT, Studylog, Meta, This Dot, EVERSANA, EVERFI, LAB Group, and our own speakers from Applitools. We covered test automation topics across ROI measurement, accessibility, testing at scale, and more. Andrew Knight, Director of Test Automation University, concludes the event with eight testing convictions inspired by Ukiyo-e Japanese woodblock prints. Check out the full Future of Testing October 2022 event library for all of the sessions.

Skills and Strategies for New Test Managers

Being a good Test Manager is about more than just choosing the right tools for your team. EasyJet Test Manager Laveena Ramchandani shares what she has learned in her experience on how to succeed in QA leadership. Some of Laveena’s strategies include how to create a culture that values feedback and communication. This webinar is great for anyone looking to become a Test Manager or for anyone who has newly started the role.

Ensuring a Reliable Digital Experience This Black Friday

With so much data and so many combinations of state, digital shopping experiences can be challenging to test. Senior Director of Product Marketing Dan Giordano talks about how to test your eCommerce application to prioritize coverage on the most important parts of your application. He also shares some common shopper personas to help you start putting together your own user scenarios. The live demo shows how AI-powered automated visual testing can help retail businesses in the areas of visual regression testing, accessibility testing, and multi-baseline testing for A/B experiments.

Dan gave a recap and went a little deeper into eCommerce testing in a follow-up blog post.

Cypress, Playwright, Selenium, or WebdriverIO? Let the Engineers Speak!

Our popular Let the Code Speak webinar series focused primarily on differences in syntax and features, but it doesn’t really cover how these frameworks hold up in the long term. In our new Let the Engineers Speak webinar, we spoke with a panel of engineers from Mercari US, YOOBIC, Hilton, Q2, and Domino’s about how they use Cypress, Playwright, Selenium, and WebdriverIO in their day-to-day operations. Andrew Knight moderated as our panelists discussed what challenges they faced and if they ever switched from one framework to another. The webinar gives a great view into the factors that go into deciding what tool is right for the project.

Read the Let the Engineers Speak recap blog series

Let the Engineers Speak! Part 5: Audience Q&A

Articles — 01.11.2023

Let the Engineers Speak! Part 4: Changing Frameworks

Articles — 01.05.2023

Let the Engineers Speak! Part 3: Favorite Test Integrations

Articles — 12.28.2022

Let the Engineers Speak! Part 2: The Biggest Challenges

Articles — 12.22.2022

The post Test Automation Video Winter Roundup: September – December 2022 appeared first on Automated Visual Testing | Applitools.

Jenkins is a popular open source automation server. It’s used to implement Continuous Integration (CI) and Continuous Delivery (CD) for any development project.

CI/CD, a key component of a DevOps strategy, allows you to shorten the development lifecycle while maintaining quality by automating tasks like testing. In short, a successful implementation will help you test more often and deploy faster with high quality. 

Jenkins, as a Java-based CI server with strong community support and a huge plugin ecosystem, is a powerful tool for anyone looking to add CI/CD to their project.

History of Jenkins

Jenkins has a long history, stretching back to 2011, and its roots extend back even further to 2004. That was the year that Kohsuke Kawaguchi, a Java developer at Sun Microsystems, built an automation tool to help him answer the question of whether his code would break the build before he committed it. The open source tool he built, called Hudson, automated tests and builds and helped him understand whether his code would work much sooner. 

In 2011, there was an issue between Oracle (which had acquired Sun Microsystems) and the open source project’s community, and the community created a fork named Jenkins. For a while both projects were updated independently, but Jenkins proved to be more popular and Hudson is no longer developed. 

What Does Jenkins Do?

Jenkins helps you automate tasks for CI/CD. Once the Jenkins server is configured, you will be able to execute a series of automated tests and builds so that your code is always up to date and valid. Implementing CI/CD through a tool like Jenkins can greatly simplify the process of ensuring a high level of code quality and successful builds. It’s particularly powerful when large development teams are working on a single project, as traditional methods can result in a lot of conflicting code commits that require complex troubleshooting.

Before we dive further into the functionality of Jenkins, let’s take a moment to define continuous integration/delivery. Then we’ll discuss how Jenkins helps you achieve it.

What is Continuous Integration and Continuous Delivery (CI/CD)?

CI is a process that enables you to integrate code changes from multiple developers working on a single project quickly and repeatedly. When a developer commits code, it can immediately be tested. If the tests pass, the code can be integrated into the build, which can similarly be tested and immediately verified. 

Automated tests are typically used alongside automated builds so that the process can be fast, effective and easily repeatable.

How Does Jenkins Help with Automated Testing?

Using Jenkins to implement a CI/CD process can dramatically improve your ability to test and ship a high quality product. 

One of the chief benefits is the ability to rapidly discover and fix bugs. If newly committed code introduces a defect into the build, not only is it caught immediately, but it is easy to know whose code caused the error. The problematic code can then be isolated, updated and recommitted quickly. 

Getting Started with Jenkins and Jenkins Pipeline

The Jenkins CI server can be run as a standalone application with a built-in Java servlet container, or as a servlet in other Java servlet containers such as Apache Tomcat.

The most popular way to manage CI/CD with Jenkins is through Jenkins Pipeline, which is a suite of plugins that help you integrate continuous delivery pipelines in Jenkins. Essentially, a Pipeline is a linked series of automated steps to be executed by Jenkins. Jenkins Pipelines are implemented in code which can be committed, so that it can be versioned and reviewed along with the rest of your code.

Here’s an example (from the Jenkins docs) of a CD scenario modeled in Jenkins Pipeline:

Pipelines can be written manually or through the newer Blue Ocean GUI. There is also an older “classic UI”, but if you want to use a UI Blue Ocean is recommended.

Extending Jenkins with Plugins

Jenkins is designed to be easily extended via plugins – and over the years a thriving community has created a huge plugin ecosystem. The strength of this community, and subsequently the size of the plugin library, is one of the best things about Jenkins.

You can find a plugin for just about anything you need, as there are already 1500+ plugins contributed by the community (and counting). This is a good place to note that Jenkins is very actively updated and open to community participation.

Many testing tools like Applitools have a plugin that can easily help get your testing and CI/CD working together. Check out the Applitools plugin for Jenkins to see how this works.

Should You Use Jenkins?

Jenkins is a popular open source tool for CI/CD that is free to use. While you may need some server administration skills to configure and monitor Jenkins, there are many advantages to consider. The Jenkins project includes a large plugin ecosystem, the community around it is thriving and it is actively developed. If that sounds appealing to you, then give Jenkins a look for your CI/CD needs.

Learn More

• Continuous Integration with Jenkins [Free Course]
• How to update Jenkins build status in GitHub pull requests [Step-by-Step Tutorial]
• Jenkins Documentation [official]

The post What is Jenkins? How to Use Jenkins for CI/CD and Testing appeared first on Automated Visual Testing | Applitools.

On January 27, 2021, Angie Jones of Applitools hosted Brian Douglas, aka “bdougie”, Staff Developer Advocate at GitHub, for a webinar to help you jump into GitHub Actions. You can watch the entire webinar on YouTube. This blog post goes through the highlights for you.

Introductions

Angie Jones serves as Senior Director of Test Automation University and as Principal Developer Advocate at Applitools. She tweets at @techgirl1908, and her website is https://angiejones.tech.

Brian Douglas serves as the Staff Developer Advocate at GitHub. Insiders know him as the “Beyoncé of GitHub.” He blogs at https://bdougie.live, and tweets as @bdougieYO.

They ran their webinar as a question-and-answer session. Here are some of the key ideas covered.

What Are GitHub Actions?

Angie’s first question asked Brian to jump into GitHub Actions.

Brian explained that GitHub Actions is a feature you can use to automate actions in GitHub. GitHub Actions let you code event-driven automation inside GitHub. You build monitors for events, and when those events occur, they trigger workflows. 

If you’re already storing your code in GitHub, you can use GitHub Actions to automate anything you can access via webhook from GitHub. As a result, you can build and manage all the processes that matter to your code without leaving GitHub. 

Build Test Deploy

Next, Angie asked about Build, Test, Deploy as what she hears about most frequently when she hears about GitHub Actions.

Brian mentioned that the term, GitOps, describes the idea that a push to GitHub drives some kind of activity. A user adding a file should initiate other actions based on that file. External software vendors have built their own hooks to drive things like continuous integration with GitHub. GitHub Actions simplifies these integrations by using native code now built into GitHub.com.

Brian explained how GitHub Actions can launch a workflow. He gave the example that a team has created a JavaScript test in Jest. There’s an existing test using Jest – either npm test, or jest. With GitHub Action workflows, the development team can automate actions based on a starting action.  In this case, the operator can drive GitHub to execute the test based on uploading the JavaScript file.  

Get Back To What You Like To Do

Angie pointed out that this catchphrase, “Get back to what you like to do,” caught her attention. She spends lots of time in meetings and doing other tasks when she’d really just like to be coding. So, she asked Brian, how does that work?

Brian explained that, as teams grow, so much more of the work becomes coordination and orchestration. Leaders have to answer questions like:

• What should happen during a pull request? 
• How do we automate testing? 
• How do we manage our build processes

When engineers have to answer these questions with external products and processes, they stop coding. With GitHub Actions, Brian said, you can code your own workflow controls. You can ensure consistency by coding the actions yourself. And, by using GitHub Actions, you make the processes transparent for everyone on the team.

Do you want a process to call Applitools? That’s easy to set up. 

Brian explained that GitHub hosted a GitHub Actions Hackathon in late 2020. The team coded the controls for the submission process into the hackathon. You can still check it out at githubhackathon.com.

The entire submission process got automated to check for all the proper files being included in a submission. The code recognized completed submissions on the hackathon home page automatically.

Brian then gave the example of work he did on the GitHub Hacktoberfest in October. For the team working on the code, Brian developed a custom workflow that allowed any authenticated individual to sign up to address issues exposed in the Hackathon. Brian’s code latched onto existing authentication code to validate that individuals could participate in the process and assigned their identity to the issue. As the developer, Brain built the workflow for these tasks using GitHub Actions.

What can you automate? Informing your team when a user does a pull request. Send a tweet when the team releases a build. Any webhook in GitHub you can automate with GitHub Actions. For example, you can even automate the nag emails that get sent out when a pull request review does not complete within a specified time. 

Common Actions

Angie then asked about the most common actions that Brian sees users running.

Brian summarized by saying, basically, continuous integration (CI). The most common use is ensuring that tests get run against code as it gets checked in to ensure that test suites get applied. You can have tests run when you push code to a branch, push code to a release branch or do a release, or even when you do a pull request.

While test execution gets run most frequently, there are plenty of tasks that one can automate. Brian did something specific to assign gifts to team members who reviewed pull requests. He also used a cron job to automate a GitHub Action which opened up a global team issue each Sunday US, which happens to be Monday in Australia, and assigned all the team members to this issue. Each member needed to explain what they were working on. This way, the globally-distributed team could stay on top of their work together without a meeting that would occur at an awkward time for at least one group of team members.

Brian talked about people coming up with truly use cases – like someone linking IOT devices to webhooks in existing APIs using GitHub Actions. 

But the cool part of these actions is that most of them are open source and searchable. Anyone can inspect actions and, if they don’t like them, modify them. If a repo includes GitHub Actions, they’re searchable.

On github.dom/bdougie, you can see existing workflows that Brian has already put together.

Jump Into GitHub Actions – What Next?

I shared some of the basic ideas in Brian’s conversation with Angie. If you want to jump into GitHub Actions in more detail, you can check out the full webinar and the slides in Addie Ben Yehuda’s summary blog for the webinar. That blog also includes a number of Brian’s links, several of which I include here as well:

• https://bdougie.live/
• https://github.com/bdougie/live
• https://lab.github.com/githubtraining/devops-with-github-actions
• https://youtube.com/ilikerobot
• https://github.com/bdougie
• https://docs.github.com/en/actions/reference/events-that-trigger-workflows

Enjoy jumping into GitHub Actions!

Featured Photo by Aziz Acharki on Unsplash

The post Get a Jump Into GitHub Actions appeared first on Automated Visual Testing | Applitools.

In February 2020, we were fortunate enough to have Erica Stanley present at the Applitools Days in Atlanta, GA. She spoke about virtual reality, augmented reality, and mixed reality and gave some examples of next-generation mixed reality.

When you create new technology, you think “fail fast”. Why should an R&D team prioritize quality?

In February 2020, we were fortunate enough to have Erica Stanley present at the Applitools Days in Atlanta, GA. Erica is a senior engineering manager on the Emerging Tech, Mixed Reality team at Mozilla. Within the Mixed Reality (MR) team, Erica’s team focuses on MR for desktop & Android and has found a creative way to implement test automation with the help of Applitools, Cypress, OpenCV, and FFMPEG.

Writing this in the midst of the pandemic and looking back to February, so much has changed. Watching her presentation, I could not help but think of the potential such technology might have in a world of social distancing. Imagine being able to shop from home while interacting with the product. Or, you might attend a virtual reality meeting in a shared room with your colleagues. These are just a few examples that Erica talks about in her presentation.

Erica focuses on quality engineering on a team that focuses on innovation. If you think that quality engineering and innovation cannot coexist, Erica explains why they must work together, as well as how she helps her team use quality tools to aid in their innovation process.

Three Realities

Let’s begin with an overview of terminology:

• Virtual Reality (VR) – A fully immersive experience where a user leaves the real-world environment behind to enter a fully digital environment via VR headsets.
• Augmented Reality (AR) – An experience where virtual objects are superimposed onto the real-world environment via smartphones, tablets, heads-up displays, or AR glasses. (Think Pokemon Go game or the IKEA furniture app.)
• Mixed Reality (MR) – A step beyond augmented reality where the virtual objects placed in the real world can be interacted with and respond as if they were real objects.

Four Interfaces

Next, Erica described the different kinds of virtual reality interface terms. If virtual reality provides an extension of human-machine interaction, let’s start at the earliest and move up.

• CLI – Command Line Interface – not necessarily what we think of as virtual reality, but still at the core of every computing system.
• GUI – Graphical User Interface – graphical based, interactive, feedback. Whether using desktop monitors or virtual reality glasses, we still seem to use this interface today.
• NUI – Natural User Interface – you don’t need instructions – it performs naturally. Erica gave the example of being a painter. Her experience of digital painting on the iPad – brush strokes, paint color mixing, etc. – seems natural from her experience as a painter.
• OUI – Organic User Interface – Interface is almost seamless with your reality – the environment learns from you and changes with you.

Examples of Next Generation Mixed Reality

Erica gave some interesting examples of potential real-world applications of mixed reality. These included entertainment, collaboration, and social engagement activities, where MR might provide unique value. Such as:

• Immersive Movies – imagine walking around in a 3D environment and interacting with the movie? Would this mean being in a movie theater or being in the movie itself? What movie would you “drop” into? Or would it be more like a virtual Black Mirror: Bandersnatch – where you interact with the movie to determine the outcome?
• Corporate Meetings – This would bring working from home to the next level – with everyone being “together” in a meeting room. There is nothing like being in-person, but to have a way to share a virtual whiteboard and be in the same virtual room could bring a new level of focus to remote meetings – I am looking forward to this application of MR.
• Civic engagement – how about envisioning planning commission proposals in a real-world walk-around in your neighborhood? Or being able to share visualization of education data results during a school board meeting?

A Focus on Quality: Yes – Even for Research Product Delivery

Some people think of quality as a cost of doing business. Those people think of quality as a way to reduce support and rework costs. If you share that mindset, you probably wonder how quality engineering fits into an R&D lab.

As Erica explains, her team works with partners who are their customers. The partners expect working products just as any customer would. Her team also works with user communities who, in turn, build on top of her team’s code. Each iteration includes new features, enhancements, and bug fixes. Because of all the downstream dependencies, any bugs, misbehaviors, or unexpected behavior changes can cause those downstream teams’ huge issues. To remain a trusted supplier and partner in this R&D world, Erica helped her team prioritize and deliver quality as a core component of the build process.

Quality challenges in Mixed Reality Applications?

Erica and her team knew to value quality for their ecosystem. The question remained: how would her team be able to deliver a quality product? What tools could help her team deploy and test a cutting-edge technology. She realized that she and her team needed to innovate with her solution. At the same time, she needed to import best-practices from across the software development practices.

So, what are some of the challenges that Erica’s team faced in testing mixed reality applications?

• Lack of specialized software for virtual reality testing. Testers must build their own test approaches and test fixturing. So, few people are building AR/VR/MR yet alone running automated tests on such applications. Teams need to figure it out on their own.
• Teams must design their tests to account for both 2D and 3D interactions.
• Teams face an incredibly fragmented device market. Headsets, eyewear, and tactile interfaces have their own specifications and behavior. As in the early days of Android smartphones – everyone complies with and extends beyond the existing specification.

With all these technical challenges, the biggest hurdle Erica and team faced was not technical. Her focus on quality ran headlong into the existing culture of ‘fail fast’. How did she convince research engineers to build in automated testing? It came down to convincing them that quality is a way to deepen and speed up the learning from the research itself.

Erica went on to discuss how she fostered a learning culture including psychological safety, implementation of standards, and deep collaboration between developers and the QA team.

Running Test Automation on Mixed Reality

The team at Mozilla implemented an entire test automation suite for testing their code builds. A large component of that automation involves visual testing, delivered via Applitools. Some of the benefits they received by visual testing AR, VR and MR applications include:

• Avoiding manual testing without the headset. Getting a user into the headset takes time. With visual testing, you can test in the browser.
• Visualization fosters collaboration between Developers and QA team. The team collaborates on what is correct in the baseline images and discuss changes as a team – a great way to speed up research learnings.
• Addressing device fragmentation – you can turn on / off certain snapshots depending on the device under test helping you deal with all the devices in a concise manner.

While the variety of scenarios require a variety of tools, the one consistent was Applitools. Below are the three scenarios that the team at Mozilla has implemented:

• 2D with Applitools & Cypress – to test navigation, essentially, automating the testing of the chrome around the browser.
• 3D on Android with Applitools & OpenCV – This combination helps their team understand where objects are in space. If an object includes another object – you can isolate the object and focus solely on the object under test.
• 360 Video with Applitools & FFMPEG – This enables test frames in 360 video to be isolated visual validated.

What’s Next

Wrapping up, Erica spoke about what happens beyond visual interactions and how this can be tested. One of the challenges up next: How do you automate the test of a controller that is supposed to vibrate with an interaction or test for voice interactions?

AR, VR, and MR comprise a set of fast-evolving user experience technologies. Erica and her team sit among the lead group of quality engineers facing down the challenges that these technologies present.

You can follow Erica on Twitter @ericastanley.

Watch the Full Video

This blog post was previously posted on Jaxenter on July 1, 2020.

The post Visual Testing of AR, VR, and MR in Research Focused Product Delivery appeared first on Automated Visual Testing | Applitools.

What is a design system? Who would use it, and for what benefit?

In May 2020, Applitools had the pleasure of hosting Tyler Krupicka from Intuit for an hour-long webinar discussing design systems and testability.

Tyler works at Intuit, a 9,400 employee company headquartered in Mountain View, California, that specializes in accounting and tax preparation software. At Intuit, Tyler works on the “Player/Design Systems” team, where he focuses on design systems.

Tyler has been working on web development at Intuit since 2015. These days, he works on UI components, tools, and testing. He primarily uses TypeScript, React and Vue. He also focuses on accessibility, and he serves as a “Level 2” Accessibility Champion inside Intuit.As part of Tyler’s job, he ensures that design systems are testable. 

What Is A Design System

Tyler started asking this question. Some people think about large design systems:

• Material Design (Google’s design system)
• Human Interface Guidelines (Apple’s design system)
• Bootstrap (set of UI interface components and CSS that are popular on the web)

People look at these and think, “Is this overkill?” After all, Google uses Material Design for Android. How many people design an entire operating system. So, Tyler dove in to answer his question.

First, Tyler sees design systems as really “Design + Code.” One part of a design system are the “design” – the common components and patterns that get used throughout products. The other part of the design system, Code, involves chunks of functionality interacting with design developers can reuse across a product.  For example, CSS Frameworks, or React, or in some other way that lets teams reuse the same code.

Design

“Design” turns into a consistent voice for your product. Those are things like typography, colors, and spacing, as well as UI components (forms, charts, site navigation), Graphics and Animation standards. These help standardize the visualization – the “look” – of the application. One can imagine an application developed by a large team that lacked a standard design – it would be disorienting to customers who expect consistency from software. A design system lets  you take base pieces, document them well, and reuse them.

Code

“Code” lets you build and document components individually, test them once, and have a way to set expectations on whether or not this application will work.  For instance, you can build a component in the app for one persona and reuse that component multiple times inside the app without worrying that you must retest it. Reuse makes code standard and creates standards for interacting with design. If you ever change design elements, the standard code reduces the effort to validate the new design with existing code. 

Combining design and code into a design system results in increased developer productivity and a consistent voice for the application. And using a design system simplifies the management of an application.

Design Systems at Intuit: TurboTax Example

Next, Tyler dove into the use of design systems for the Intuit TurboTax product. 

Tyler explained how TurboTax works. By providing users with interview-style questions, TurboTax helps millions of people, primarily in the United States and Canada, file their income tax returns. Intuit offers both web and mobile versions of TurboTax.

Because TurboTax uses interview questions to create conditional flows through the product, there are thousands of screens users can encounter based on their unique needs. Screens can range in complexity from a single question to a complete form. 

Building the TurboTax Design System

To create the design, Intuit develops about 50 UI components needed to run the experience. Headers, radio buttons, tiles, multi-select, data entry, action buttons – all become part of the basic UI. 

Intuit also builds a set of about 500 “mocks”, or mock-ups, of what common screens look like for testing. These help show the UI components in common layouts for testing purposes.

By using a design system, Intuit can make changes to the application and ensure consistency for all these screens. 

Tyler gave the example of a decision to deliver a uniform increase in contrast in TurboTax for accessibility. The contrast changes required updates to typography weight, button color, and link color. This work would impact over 90% of all the screens. How could Intuit deliver these changes to the application and ensure they occurred consistently across the application? The team did not have the staff needed to manually check every possible screen. Instead, they made the design changes to the UI components and validated the mocks.

Testing A Design System

Next, Tyler dove into a discussion of testing a design system. 

Approach

Since the design system represents all the functionality and visual representation of the application, Tyler recommends going broad with testing across all the elements, including:

• Functionality
• Visual
• Performance
• Accessibility

From a functional perspective, unit tests serve as key building blocks for code. Integration tests ensure that the code works wiht the back end. Cross-vrowser tests help ensure that the code works on a bunch of different browsers and mobile devices. 

Visual testing validates the impact of UI component changes on the overall UI. If buttons change size or Intuit uses a new font size, does the page continue to render correctly? Do unexpected breaks happen on pages? Do space reductions make pages look too constrained?

Performance testing involves checking the impact of code changes on page loading times.  Specifically, do new packages or package upgrades change the amount of data transferred between the server and the browser – slowing the user experience? 

Finally, Tyler tests accessibility to make sure that any customer who wants to use the product can do so. Whether a user has a screen reader or navigates solely by keyboard, ensure that the user can get around.

Getting Organized

With testing goals articulated, Tyler introduced Storybook as the tool for creating documentation and initial testing. If you use projects like React, Vue and Angular that allow you to build user interfaces in small components, the Storybook project lets you build interactions for those components and document their behavior.  

Effectively, you demonstrate a  use case for your components and show how they all interact together on a page.  Each use case is one story – hence Storybook is a collection of these stories. 

Storybook allows for a range of tests, including accessibility tests. For instance, Tyler uses Storybook to test keyboard navigation and focus. He also tests for things like hover behaviors – where components appear while the cursor hovers over another element, which may be difficult to test with automation. Testing with Storybook lets the testing team run tests early across the range of design system dimensions.

Dog Fooding

To deliver successful products, Tyler’s team must support the construction and use of these UI components. So, they actually write code to try out the components – just to see how they work. Putting together checkboxes or radio buttons, or drop-down lists helps show how the code interacts with components and creates mocks for testing the design system.

Tyler showed the example of a page that looked like static boxes. Each box expanded when clicked to show hidden detail and slid the rest of the page down. His team dog foods this kind of behavior.

All of these activities help create a body of testable code, so that a design change in the design system can be mocked up and validated without impacting production code.

Testing Walkthrough

Tyler then walked us through a simple change that can have a signficant implications – resizing a button. 

Without a design system, you might have to make a button change like this ad-hoc across your application. As a result, you risk changing the page-to-page behavior and look-and-feel. Using a design system lets you make the change globally, and then focus on the implications.

What kinds of implications? A larger button might displace elements on a page. The button may not function as expected with existing elements on a given page. All these changes need to be tested.

Unit Tests

First, Tyler looked at unit tests. Does the button click? Can the button work with an icon associated with it? Will the button work with an transparent background? Does the button work with assigned ARIA (Accessible Rich Internet Applications) attributes? Testing puts these Storybook components through their paces.  

Intuit have a pretty standard way of running through these tests.  Tyler talked about using JEST and WebdriverIO and Selenium to automate this testing across multiple browsers. They have lots of users on multiple web browsers, as well as Mac users with Safari an iOS, so they pay special attention to those target platforms. 

Accessibility Tests

Next, Tyler talked about accessibility testing. His team uses AXE to perform static validation for ARIA standards on application pages, including contrast checking.  Tyler explained how AXE helps perform contrast checking. For instance, if a button is one color, does the background let the button stand out? AXE can do the contrast checking by comparing the two colors. Or, will the text be readable against its background – or is it too small or too thin? Again – AXE can make these calculations.

Tyler also talked about HTML customizations. For example, you want to use a special radio button design that isn’t included in the existing HTML. You build this behavior as a custom implementation – and then you have to validate this behavior across the browsers your customers use. Tyler pointed out that companies make these decisions to help brand their websites for customers. And, these custom behaviors must be validated for functionality and accessible behavior across multiple platforms.  At the same time, since you are replacing functionality built into HTML, keyboard navigation still needs to behave as expected. So, you need to test those behaviors. 

At Intuit, the team tests these behaviors manually – and with tools designed to help. By using popular screenreaders like Jaws, Voiceover, and NVDA, Tyler’s team ensures that these screen readers don’t get thrown off by customizations. Also through manual testing, the team checks to ensure that keyboard navigation results in expected page responses – for example, they make sure that focus states behave as expected. And, they also ensure that the page behaves correctly when a customer has selected “reduced motion”.

Performance Tests

From a design perspective, the design system simplifies app delivery. By using standard libraries and building standard behaviors, the design system creates this great look-and-feel. At the same time, developers need to consider the impact on performance. For instance, a specific package might add great mobile behavior – at the cost of an additional 1-2 seconds of page load time. Users likely won’t value the behavior over slower page load time.  

You can choose a great library that gives you great functionality – but at what cost? How does the size of your Javascript package impact your page downloads? Often, this problem creeps up on you.  You use an early version of the library and everything works well. But, when you update your application, your library update suddenly doubles in size. This happens quite often. New versions of libraries include new features – which add to the library load times. And, you have to rethink the software you’re using.

Tyler talked about how Intuit handles these kinds of challenges. If Intuit uses code from an open source project, they might contribute back to the project to make a library with reduced size. Or, they might use a different open-source package.  

Visual Testing

Each time a new component gets introduced or modified, Intuit runs tests through Applitools Ultrafast Grid. Intuit has been running Grid since its first beta tests. With Ultrafast Grid, Intuit can set checkpoints and take screenshots across a range of browsers and devices. They can set multiple screen sizes (from mobile to Full HD and  higher). Their goal – ensure that responsive pages behave as expected, and understand the impact of design changes on existing pages. Ultrafast grid gives them the capability to easily compare changes across different versions of an application.

Tyler talked about using Ultrafast Grid to test against Chrome, Edge, and Safari. He talked about setting different viewport sizes to check responsive page behavior against expectations. 

Component and Mock Level

Tyler explained how Intuit tests components and mocks. 

In Storybook, Intuit keeps the latest components, as well as page mocks for each component in use. When a change occurs in the design system, Intuit can do two kinds of checks:

1. They can check the components to see if the component changes have violated any specifications
2. They check the mocks, to determine how the component changes affect page rendering.

So, for instance, Intuit can see if a collection of multiple new components violate spacing guidelines or other design requirements on pages. Applitools Ultrafast Grid helps Intuit do this evaluation.

Also using the mocks, Intuit can also see if a change violates accessibility guidelines using AXE.  

Accessing Intuit Tools for Design Systems

Tyler discussed the two tools Intuit uses to ensure success in their design systems.

The first, the Intuit DS-CLI, makes it easy to implement Storybook for your designs. Just like Create-React-App lets you build a web app easily, DS-CLI contains all the code you need to utilize Storybook for component and mock management. This includes preconfigured Storybook, component templates with best practices, great Typescript support, and bundle size tracking.  Tyler mentioned that DS-CLI is free and open source. 

One great part about DS-CLI is size tracking.  Each component can be tracked over time for package size. You can see when your bundles are growing – and why.

With the second, Intuit Proof, you can automate the development of test running in Storybook. Proof is a tapable testrunner for Storybook. Proof helps you automate tests for components and mocks by helping you know which ones have tests. In addition, Proof has built-in hooks for Applitools and Axe. So, instead of fumbling about inside Storybook to see what has been tested and what remains, you know what tests exist, which still need to be written, and helps you track their status. 

Proof is also free and open source.

Conclusion

Tyler’s presentation made it clear that a design system can make your life much easier, but you need to invest in tools and methodology to help ensure consistency – the benefit of a design system. 

You need a component evaluation system. You need mockups of your pages to test. And you need a testing environment.  Intuit does all this inside Storybook.

Next, you need a testing strategy to get components evaluated and integrated into your pages. Invest in unit testing and test runners. Especially if you care about being perceived as open to all users – invest in accessibility tools. And, finally, know that rendering matters – invest in visual evaluation solutions like Applitools.

Finally, you need some pieces of infrastructure to make your design system work. Intuit shares theirs open source.

If you make these investments up front, a design system can serve your development and test strategy to help you easily develop incremental improvements across your application without fear of code changes. 

Read More

• Impact of Visual AI on Test Automation Report
• A/B Testing: Validating Multiple Variations
• Modern Dev Teams Are More Efficient Than Ever — Quality Engineering Should Be Too.
• The Next Generation of Cross Browser Testing is Ultrafast
• Sign up for a free Applitools account
• Request an Applitools Demo

The post Design Systems and Testability appeared first on Automated Visual Testing | Applitools.