Learn about what’s new in iOS 16, including some new updates test engineers should be looking out for.

It’s an exciting time of the year for anyone who uses Apple devices – and that includes QA engineers charged with mobile testing. Apple has just unveiled iOS 16, and as usual it is filled with new features for iOS users to enjoy.

Many of these new features, of course, affect the look and feel and usability of any application running on iOS. If you’re in QA, that means you’ve now got a lot of new testing to do to make sure your application works as perfectly on iOS 16 as it did on previous versions of the operating system.

For example, Apple has just upgraded their iconic “notch” into a “Dynamic Island.” This is significant redesign of a small but highly visual component that your users will see every time they look at their phone. If your app doesn’t function appropriately with this new UI change, your users will notice.

If you’re using Native Mobile Grid for your mobile testing, no need to worry, as Native Mobile Grid already supports automated testing of iOS 16 on Apple devices.

With this in mind, let’s take a look through some of the most exciting new features of iOS 16, with a focus on how they can affect your life as a test engineer.

Customizable Lock Screen

The lockscreen on iOS 16 devices can now be customized far more than before, going beyond changing the background image – you can now alter the appearance of the time as well as add new widgets. Another notable change here is that notifications now pop up from the bottom instead of the top.

As a QA engineer, there are a few things to consider here. First, if your app will have a new lockscreen widget, you certainly need to test it carefully. Performing visual regression testing and getting contrast right will be especially important on an uncertain background.

Even if you don’t develop a widget, it’s worth thinking about (and then verifying) whether the user experience could be affected by your notifications moving from the top of the user’s screen to the bottom. Be sure and take a look at how they will appear when stacked as well to make sure the right information is always visible.

Notch –> Dynamic Island

As we mentioned above, the notch is getting redesigned into a “Dynamic Island.” This new version of the cutout required for the front-facing camera can now present contextual information about the app you’re using. It will expand and contract based on the info it’s displaying, so it’s not a fixed size.

That means your app may now be resizing around the new “Dynamic Island” in ways it never did with the old notch. Similarly, your contextual notifications may not look quite the same either. This is definitely something worth testing to make sure the user experience is still exactly the way you meant it to be.

Other New iOS 16 Features

There are a lot of other new features, of course. Some of these may not have as direct an impact on the UI or functionality of you own applications, but it’s worth being familiar with them all. Here are a few of the other biggest changes – check them carefully against your own app and be sure to test accordingly.

• Send, Edit and Unsend Messages: You can now send, edit and unsend content in the Messages app, and you can now send/unsend (as well as schedule delivery) in the Mail app as well
• Notifications and Live Activities: As mentioned, notifications now come up from the bottom. They can also “update” so that you don’t need to get repeated new notifications from the same app (eg: sports games scores, rideshare ETAs)
• Live Text and Visual Lookup: iOS users can now extract live text from both photos and videos, as well as copy the subject of an image out of its background and paste it elsewhere
• Focus Mode and Focus Filters: Focus mode (to limit distractions) can now be attached to custom lockscreens, and applied not just to an app but within an app (eg: restricting specific tabs in a browser)
• Private Access Tokens: For some apps and websites, Apple will use these tokens to verify that users are human and bypass traditional CAPTCHA checks
• Other improvements: The Fitness app, Health app, Maps app, iCloud, Wallet and more all got various improvements as well. Siri did too (you can now “speak” emojis ?). See the full list of iOS 16 updates.

Make Your Mobile Testing Easier

Mobile testing is a challenge for many organizations. The number of devices, browsers and screens in play make achieving full coverage extremely time-consuming using traditional mobile testing solutions. At Applitools, we’re focused on making software testing easier and more effective – that’s why we pioneered our industry-leading Visual AI. With the new Native Mobile Grid, you can significantly reduce the time you spend testing mobile apps while ensuring full coverage in a native environment.

Learn more about how you can scale your mobile automation testing with Native Mobile Grid, and sign up for access to get started with Native Mobile Grid today.

The post iOS 16 – What’s New for Test Engineers appeared first on Automated Visual Testing | Applitools.

Learn why cross-browser testing is so important and an approach you can take to make cross-browser testing with Selenium much faster.

What is Cross Browser Testing?

Cross-browser testing is a form of functional testing in which an application is tested on multiple browsers (Chrome, Firefox, Edge, Safari, IE, etc.) to validate that functionality performs as expected.

In other words, it is designed to answer the question: Does your app work the way it’s supposed to on every browser your customers use?

Why is Cross Browser Testing Important?

While modern browsers generally conform to key web standards today, important problems remain. Differences in interpretations of web standards, varying support for new CSS or other design features, and rendering discrepancies between the different browsers can all yield a user experience that is different from one browser to the next.

A modern application needs to perform as expected across all major browsers. Not only is this a baseline user expectation these days, but it is critical to delivering a positive user experience and a successful app.

At the same time, the number of screen combinations (between screen sizes, devices and versions) is rising quickly. In recent years the number of screens required to test has exploded, rising to an industry average of 81,480 screens and reaching 681,296 for the top 30% of companies.

Ensuring complete coverage of each screen on every browser is a common challenge. Effective and fast cross-browser testing can help alleviate the bottleneck from all these screens that require testing.

How to Perform Modern Cross Browser Testing in Selenium with Visual Testing

Traditional approaches to cross-browser testing in Selenium have existed for a while, and while they still work, they have not scaled well to handle the challenge of complex modern applications. They can be time-consuming to build, slow to execute and challenging to maintain in the face of apps that change frequently.

Applitools Developer Advocate and Test Automation University Director Andrew Knight (AKA Pandy Knight) recently conducted a hands-on workshop where he explored the history of cross-browser testing, its evolution over time and the pros and cons of different approaches.

Andrew then explores a modern cross-browser testing solution with Selenium and Applitools. He walks you through a live demo (which you can replicate yourself by following his shared Github repo) and explains the benefits and how to get started. He also covers how you can accelerate test automation with integration into CI/CD to achieve Continuous Testing.

Check out the workshop below, and follow along with the Github repo here.

More on Cross Browser Testing in Cypress, Playwright or Storybook

At Applitools we are dedicated to making software testing faster and easier so that testers can be more effective and apps can be visually perfect. That’s why we use our industry-leading Visual AI and built the Applitools Ultrafast Grid, a key component of the Applitools Test Cloud that enables ultrafast cross-browser testing. If you’re looking to do cross-browser testing better but don’t use Selenium, be sure to check out these links too for more info on how we can help:

• Testing Storybook Components in Any Browser – Without Writing Any New Tests!
• Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser
• How to Run Cross Browser Tests with Cypress on All Browsers (and follow-up Q&A)
• Learn More about How to Use Applitools for Cross-Browser Testing

The post Ultrafast Cross Browser Testing with Selenium Java appeared first on Automated Visual Testing | Applitools.

Learn everything you need to know about how to perform UI testing, including why it’s important, a demo of a UI test, and tips and tricks to make UI testing easier.

When users explore web, mobile or desktop applications, the first thing they see is the User Interface (UI). As digital applications become more and more central to the way we all live and work, the way we interact with our digital apps is an increasingly critical part of the user experience.

There are many ways to test an application: Functional testing, regression testing, visual testing, cross-browser testing, cross-device testing and more. Where does UI testing fit into this mix?

UI testing is essential to ensure that the usability and functionality of an application performs as expected. This is critical for delivering the kinds of user experiences that ensure an application’s success. After all, nobody wants to use an app where text is unreadable, or where buttons don’t work. This article will explain the fundamentals of UI testing, why it’s important, and supply a UI testing checklist and examples to help you get started.

What is UI Testing?

UI testing is the process of validating that the visual elements of an application perform as expected. In UI Testing, graphical components such as text, radio buttons, checkboxes, buttons, colors, images and menus are evaluated against a set of specifications to determine if the UI is displaying and functioning correctly.

Why is UI Testing Important?

UI testing is an important way to ensure an application has a reliable UI that always performs as expected. It’s critical for catching visual and even functional bugs that are almost impossible to detect using other kinds of testing.

Modern UI testing, which typically utilizes visual testing, works by validating the visual appearance of an application, but it does much more than make sure things simply look correct. Your application’s functionality can be drastically affected by a visual bug. UI testing is critical for verifying the usability of your UI.

Note: What’s the difference between UI testing and GUI testing? Modern applications are heavily dependent on graphical user interfaces (GUIs). Traditional UI testing can include other forms of user interfaces, including CLIs, or can use DOM-based coded locators to try and verify the UI rather than images. Modern UI testing today frequently involves visual testing.

Let’s take an example of a visual bug that slipped into production from the Southwest Airlines website:

Under a traditional functional testing approach this would pass the test suite. All the elements are present on the page and successfully loaded. But for the user, it’s easy to see the visual bug. 

This does more than deliver a negative user experience that may harm your brand. In this example, the Terms and Conditions are directly overlapping the ‘continue’ button. It’s literally impossible for the user to check out and complete the transaction. That’s a direct hit to conversions and revenue.

With good UI testing in place, bugs like these will be caught before they become visible to the user.

UI Testing Approaches

Manual Testing

Manual UI testing is performed by a human tester, who evaluates the application’s UI against a set of requirements. This means the manual tester must perform a set of tasks to validate that the appearance and functionality of every UI element under test meets expectations. The downsides of manual testing are that it is a time-consuming process and that test coverage is typically low, particularly when it comes to cross-browser or cross-device testing or in CI/CD environments (using Jenkins, etc.). Effectiveness can also vary based on the knowledge of the tester.

Record and Playback Testing

Record and Playback UI testing uses automation software and typically requires limited or no coding skill to implement. The software first records a set of operations executed by a tester, and then saves them as a test that can be replayed as needed and compared to the expected results. Selenium IDE is an example of a record and playback tool, and there is even one built directly into Google Chrome.

Model-Based Testing

Model-based UI testing uses a graphical representation of the states and transitions that an application may undergo in use. This model allows the tester to better understand the system under test. That means tests can be generated and potentially automated more efficiently. In its simplest form, the approach requires the steps below:

1. Build a model representing the system
2. Determine the inputs
3. Understand the expected outputs
4. Execute the tests and compare the results against expectations

Automated UI Testing vs Manual UI Testing

Benefits of Manual UI Testing

Manual testing, as we have seen above, has a few severe limitations. Because the process relies purely on humans performing tasks one at a time, it is a slow process that is difficult to scale effectively. Manual testing does, however, have advantages:

• Manual testing can potentially be done with little to no tooling, and may be sufficient for early application prototypes or very small apps. 
• An experienced manual tester may be able to discover bugs in edge-cases through ad-hoc or exploratory testing, as well as intuitively “feel” the user experience in a way that is difficult to understand with a scripted test.

Benefits of Automated UI Testing

In most cases automation will help testing teams save time by executing pre-determined tests repeatedly. Automation testing frameworks aren’t prone to human errors and can run continuously. They can be parallelized and executed easily at scale. With automated testing, as long as tests are designed correctly they can be run much more frequently with no loss of effectiveness. 

Automation testing frameworks may be able to increase efficiency even further with specialized capabilities for things like cross-browser testing, mobile testing, visual AI and more.

UI Testing Checklist of Test Cases

On the surface, UI testing is simple – just make sure everything “looks” good. Once you poke beneath that surface, testers can quickly find themselves encountering dozens of different types of UI elements that require verification. Here is a quick checklist you can use to make sure you’ve considered all the most common items.

UI Testing Checklist – Common Tests

• Text: Can all text be read? Is the contrast legible? Is anything covered by another element?
• Forms, Fields and Pickers: Are all text fields be visible, and can text be entered and submitted? Do all dropdowns display correctly? Are validation requirements (such as a date in a datepicker) upheld?
• Navigation and Sorting: Whether it’s a site menu, a sortable table or a multi-page form, can the user navigate via the UI? Do all dropdowns display? Can all options be clicked/tapped, and do they have the desired effect? 
• Buttons and Links: Are all buttons and links visible? Are they formatted consistently? Can they be selected, and do they take the user to the intended pages?
• Responsiveness: When you adjust the resolution, do all of the above UI elements continue to behave as intended?

Each of the above must be tested across every page, table, form and menu that your application contains. 

It’s also a good practice to test the UI for specific critical end-to-end user journeys. For example, making sure that it’s possible to journey smoothly from: User clicks Free Trial Signup (Button) > User submits Email Address (Form) > User Logs into Free Trial (Form) > User has trial access (Product)

Challenges of UI Testing

UI testing can be a challenge for many reasons. With the proper tooling and preparation these challenges can be overcome, but it’s important to understand them as you plan your UI testing strategy.

• User Interfaces are complex: As we’ve discussed above, there are numerous distinct elements on each page that must be tested. Embedded forms, iFrames, dropdowns, tables, images, videos and more must all be tested to be sure the UI is working as intended.
• User Interfaces change fast: For many applications the UI is in a near-constant state of flux, as frequent changes to the text, layout or links are implemented. Maintaining full coverage is challenging when this occurs.
• User Interfaces can be slow: Testing the UI of an application can take time, especially compared to smaller and faster tests like unit tests. Depending on the tool you are using, this can make them feel difficult to run as regularly.
• Testing script bottlenecks: Because the UI changes so quickly, not only do testers have to design new test cases, but depending on your tooling, you may have to constantly create new coded test scripts. Testing tools with advanced capabilities, like the Visual AI in Applitools, can mitigate this by requiring far less code to deliver the same coverage.

UI Testing Example

Let’s take an example of an app with a basic use case, such as a login screen.

Even a relatively simple page like this one will have numerous important test cases (TC):

• TC 1: Is the logo at the top appropriate for the screen, and aligned with brand guidelines?
• TC 2: Is the title of the page displaying correctly (font, label, position)?
• TC 3: Is the dividing line displaying correctly? 
• TC 4: Is the Username field properly labeled (font, label, position)?
• TC 5: Is the icon by the Username field displaying correctly?
• TC 6: Is the Username text field accepting text correctly (validation, error messages)?
• TC 7: Is the Password field properly labeled (font, label, position)?
• TC 8: Is the icon by the Password field displaying correctly?
• TC 9: Is the Password text field accepting text correctly (validation, error messages)?
• TC 10: Is the Log In button text displaying correctly (font, label, position)?
• TC 11: Is the Log In button functioning correctly on click (clickable, verify next page)
• TC 12: Is the Remember Me checkbox title displaying correctly (font, label, position)?
• TC 13: Is the Remember Me checkbox functioning correctly on click (clickable, checkbox displays, cookie is set)?

Simply testing each scenario on a single page can be a lengthy process. Then, of course, we encounter one of the challenges listed above – the UI changes quickly, requiring frequent regression testing. 

How to Simplify UI Testing with Automation

Performing this regression testing manually while maintaining the level of test coverage necessary for a strong user experience is possible, but would be a laborious and time-consuming process. One effective strategy to simplify this process is to use automated tools for visual regression testing to verify changes to the UI.

Benefits of Automated Visual Regression Testing for UI Testing

Visual regression testing is a method of ensuring that the visual appearance of the application’s UI is not negatively affected by any changes that are made. While this process can be done manually, modern tools can help you automate your visual testing to verify far more tests far more quickly.

Automated Visual UI Testing Example

Let’s return to our login screen example from earlier. We’ve verified that it works as intended, and now we want to make sure any new changes don’t negatively impact our carefully tested screen. We’ll use automated visual regression testing to make this as easy as possible.

1. As we saw above, our baseline screen looks like this:
   
2. Next, we’ll make a change by adding a row of social buttons. Unfortunately, this will have the effect of inadvertently rendering our login button unusable by pushing it up into the password field:
   
3. We’ll use our automated visual testing tool to evaluate our change against the baseline. In our example, we’ll use a tool that utilizes Visual AI to highlight only the relevant areas of change that a user would notice. The tool would then bring our attention to the new social buttons along with the section around the now unusable button as areas of concern.
   
4. A test engineer will then review the comparison. Any intentional changes that were flagged are marked as accepted changes. On some screens we might expect changes in certain dynamic areas, and these can be flagged for Visual AI to ignore going forward.
   
   We need to address only the remaining areas that are flagged. In our example, every area flagged in red is problematic – we need to shift down the social buttons, and move the button out of the password field. Once we’ve done this, we run test again, and a new baseline is created only when everything passes. The final result is free of visual defects:
   
   

Why Choose Automated Visual Regression Testing with Applitools for UI Testing

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy and includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

The Applitools Ultrafast Test Cloud includes unique features like the Ultrafast Grid, which can run your functional & visual tests once locally and instantly render them across any combination of browsers, devices, and viewports. Our automated maintenance capabilities make use of Visual AI to identify and group similar differences found across your test suite, allowing you to verify multiple checkpoint images at once and to replicate maintenance actions you perform for one step in other relevant steps within a batch.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Read More

• How do Baseline Comparisons Work in Applitools? [Video, 2:11]

• What is Visual Regression Testing [blog]
• Top 9 Tips to Increase UI Automation Speed [blog]
• What is Visual Testing [blog]
• What is Visual AI [blog]
• Overview of visual UI testing [docs]
• The Impact of Visual AI On Test Automation [webinar recap]
• Modern Functional Test Automation Through Visual AI [free course]

The post UI Testing: A Getting Started Guide and Checklist appeared first on Automated Visual Testing | Applitools.

Wondering how to choose between Playwright vs Selenium for your test automation? Read on to see a comparison between the two popular test automation tools.

When it comes to web test automation, Selenium has been the dominant industry tool for several years. However, there are many other automated testing tools on the market. Playwright is a newer tool that has been gaining popularity. How do their features compare, and which one should you choose?

What is Selenium?

Selenium is a long-running open source tool for browser automation. It was originally conceived in 2004 by Jason Huggins, and has been actively developed ever since. Selenium is a widely-used tool with a huge community of users, and the Selenium WebDriver interface even became an official W3C Recommendation in 2018.

The framework is capable of automating and controlling web browsers and interacting with UI elements, and it’s the most popular framework in the industry today. There are several tools in the Selenium suite, including:

• Selenium WebDriver: WebDriver provides a flexible collection of open source APIs that can be used to easily test web applications
• Selenium IDE: This record-and-playback tool enables rapid test development for both engineers and non-technical users
• Selenium Grid: The Grid lets you distribute and run tests in parallel on multiple machines

The impact of Selenium goes even beyond the core framework, as a number of other popular tools, such as Appium and WebDriverIO, have been built directly on top of Selenium’s API.

Selenium is under active development and recently unveiled a major version update to Selenium 4. It supports just about all major browsers and popular programming languages. Thanks to a wide footprint of use and extensive community support, the Selenium open source project continues to be a formidable presence in the browser automation space.

What is Playwright?

Playwright is a relatively new open source tool for browser automation, with its first version released by Microsoft in 2020. It was built by the team behind Puppeteer, which is a headless testing framework for Chrome/Chromium. Playwright goes beyond Puppeteer and provides support for multiple browsers, among other changes.

Playwright is designed for end-to-end automated testing of web apps. It’s cross-platform, cross-browser and cross-language, and includes helpful features like auto-waiting. It is specifically engineered for the modern web and generally runs very quickly, even for complex testing projects.

While far newer than Selenium, Playwright is picking up steam quickly and has a growing following. Due in part to its young age, it supports fewer browsers/languages than Selenium, but by the same token it also includes newer features and capabilities that are more aligned with the modern web. It is actively developed by Microsoft.

Selenium vs Playwright

Selenium and Playwright are both capable web automation tools, and each has its own strengths and weaknesses. Depending on your needs, either one could serve you best. Do you need a wider array of browser/language support? How much does a long track record of support and active development matter to you? Is test execution speed paramount? 

Each tool is open source, cross-language and developer friendly. Both support CI/CD (via Jenkins, Azure Pipelines, etc.), and advanced features like screenshot testing and automated visual testing. However, there are some key architectural and historical differences between the two that explain some of their biggest differences.

Selenium Architecture and History

• Architecture: Selenium uses the WebDriver API to interact between web browsers and browser drivers. It operates by translating test cases into JSON and sending them to the browsers, which then execute the commands and send an HTTP response back.
• History: Selenium has been in continuous operation and development for 18+ years. As a longstanding open source project, it offers broad support for browsers/languages, a wide range of community resources and an ecosystem of support.

Playwright Architecture and History

• Architecture: Playwright uses a WebSocket connection rather than the WebDriver API and HTTP. This stays open for the duration of the test, so everything is sent on one connection. This is one reason why Playwright’s execution speeds tend to be faster.
• History: Playwright is fairly new to the automation scene. It is faster than Selenium and has capabilities that Selenium lacks, but it does not yet have as broad a range of support for browsers/languages or community support. It is open source and backed by Microsoft.

Comparing Playwright vs Selenium Features

It’s important to consider your own needs and pain points when choosing your next test automation framework. The table below will help you compare Playwright vs Selenium.

Should You Use Selenium or Playwright for Test Automation?

Is Selenium better than Playwright? Or is Playwright better than Selenium? Selenium and Playwright both have a number of things going for them – there’s no easy answer here. When choosing between Selenium vs Playwright, it’s important to understand your own requirements and research your options before deciding on a winner.

Selenium vs Playwright: Let the Code Speak

A helpful way to go beyond lists of features and try to get a feel for the practical advantages of each tool is to go straight to the code and compare real-world examples side by side. At Applitools, our goal is to make test automation easier for you – so that’s what we did! 

In the video below, you can see a head to head comparison of Playwright vs Selenium. Angie Jones and Andrew Knight take you through ten rounds of a straight-to-the-code battle, with the live audience deciding the winning framework for each round. Check it out for a unique look at the differences between Playwright and Selenium.

If you like these code battles and want more, we’ve also pitted Playwright vs Cypress and Selenium vs Cypress – check out all our versus battles here.

In fact, our original Playwright vs Cypress battle (recap here) was so popular that we’ve even scheduled our first rematch. Who will win this time? Register for the Playwright vs Cypress Rematch now to join in and vote for the winner yourself!

Learn More about Playwright vs Selenium

Want to learn more about Playwright or Selenium? Keep reading below to dig deeper into the two tools.

• Your Favorite Questions about Playwright, Answered [blog]
• What’s New In Selenium 4? [blog]
• Playwright: A New Test Automation Framework for the Modern Web [webinar w/team at Microsoft]
• Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser [blog]
• Modern Cross Browser Testing with Selenium in Java [webinar]
• Playwright with JavaScript [free course]
• Selenium WebDriver with Java [free course]
• Playing with Playwright [blog]
• Playing with Playwright – Java API and Playwright vs Selenium [blog]
• 16 reasons why to use Selenium IDE in 2021 (and 1 why not) [blog]

The post Playwright vs Selenium: What are the Main Differences and Which is Better? appeared first on Automated Visual Testing | Applitools.

Get all the latest test automation videos you need in one place.

It’s summertime (at least where I am in the US), and this year has been a hot one. Summer is a great season to take a step back, to reflect, and hopefully to relax. The testing world moves so quickly sometimes, and while we’re all doing our jobs it can be hard to find the time to just pause, take a deep breath, and look around you at everything that’s new and growing.

Here at Applitools, we want to help you out with that. While you’ve hopefully been enjoying the nice weather, you may not have had a chance to see every video or event that you might have wanted to, or you may have missed some new developments you’d be interested in. So we’ve rounded up a few of our best test automation videos of the summer so far in one place.

All speakers are brilliant testing experts and we’re excited to share their talks with you – you’ll definitely want to check them all out below.

ICYMI: A few months back we also rounded up our top videos from the first half of 2022.

The State of UI/UX Testing: 2022 Results

Earlier this year, Applitools set out to conduct an industrywide survey on the state of testing in the UI/UX space. We surveyed over 800 testers, developers, designers, and digital thought leaders on the state of testing user interfaces and experiences in modern frontend development. Recently, our own Dan Giordano held a webinar to go over the results in detail. Take a look below – and don’t forget to download your free copy of the report.

Front-End Test Fest 2022

Front-End Test Fest 2022 was an amazing event, featuring leading speakers and testing experts sharing their knowledge on a wide range of topics. If you missed it, a great way to get started is with the thought-provoking opening keynote for the event given by Andrew Knight, AKA the Automation Panda. In this talk, titled The State of the Union for Front End Testing, Andrew explores seven major trends in front end testing to help unlock the best approaches, tools and frameworks you can use.

For more on Front-End Test Fest 2022 and to see all the talks, you can read this dedicated recap post or just head straight to our video library for the event.

Cypress Versus Playwright: Let the Code Speak

There are a lot of opinions out there on the best framework for test automation – why not let the code decide? In the latest installment in our popular versus series, Andrew Knight backs Playwright and goes head to head with Cypress expert Filip Hric. Round for round, Filip and Andy implement small coding challenges in JavaScript, and the live audience voted on the best solution. Who won the battle? You’ll have to watch to find out.

Just kidding, actually – at Applitools we want to make gaining testing knowledge easy, so why would we limit you to just one way of finding the answer? Filip Hric summarizes the code battle (including the final score) in a great recap blog post right here.

Can’t get enough of Cypress vs Playwright? Us either. That’s why we’re hosting a rematch to give these two heavyweights another chance to go head to head. Register today for to be a part of the Cypress vs Playwright Rematch Event on September 8th!

Coded vs. Codeless Testing Tools—And the Space In Between

There are a lot of testing debates out there, and coded vs codeless testing tools is one of the big ones. How can you know which is better, and when to use one or the other? Watch this panel discussion to see leading automation experts discuss the current landscape of coded and codeless tools. Learn what’s trending, common pitfalls with each approach, how a hybrid approach could work, and more.

Your panel for this event includes our own Anand Bagmar and Andrew Knight, along with Mush Honda, Chief Quality Architect and Coty Resenblath, CTO, both from Katalon.

Autonomous Testing, Test Cloud Infrastructure, and Emerging Trends in Software Testing

Looking to get a handle on the where testing is heading in the future? Hear from our Co-Founder and CEO, Gil Sever, as he sits down for a Q&A with QA Financial to discuss the future of testing. Learn about the ways autonomous testing is transforming the market, advancements in the cloud and AI, and the ups and downs of where testing could go in the next few years. Gil also shares insights he’s learned from our latest State of UI/UX Testing survey.

Test Automation Stories from Our Customers

We know that every day you and countless others are innovating in the test automation space, encountering challenges and discovering – or inventing – impressive solutions. Our hope is that hearing how others have solved a similar problem will help you understand that you’re not alone in facing these obstacles, and that their stories will give you a better understanding of your own challenges and spark new ways of thinking.

Automating Manufacturing Quality Control with Visual AI

We all know about web and mobile regression testing, but did you know that Visual AI is solving problems in the manufacturing space as well? Jerome Rieul, Test Automation Architect, explains how a major Swiss luxury brand uses uses Visual AI to detect changes in CAD drawings and surface issues before they hit production lines. A great example of an out-of-the-box application of technology leading to fantastic results.

Simplifying Test Automation with Codeless Tools and Visual AI

Test automation can be hard, and many shops struggle to do it effectively. One way to lower the learning curve is to take advantage of a codeless test automation tool – and that doesn’t mean you have to forego advanced and time-saving capabilities like Visual AI. In this webinar Applitools’ Nikhil Nigam shares how Visual AI can integrate seamlessly with codeless tools like Selenium IDE, Katalon Studio, and Tosca to supercharge verifications and meet industrial-grade needs. (And for more on codeless testing tools, don’t forget to watch our lively panel discussion!)

How EVERFI Moved from No Automation to Continuous Test Generation in 9 Months

Starting up test automation from scratch can be a daunting challenge – but it’s one that countless testing teams across the world have faced before you. In this informative talk, Greg Sypolt, VP of Quality Engineering, and Sneha Viswalingam, Director of Quality Engineering, both from EVERFI, share their journey. Learn about the tools they used, how they approached the project, and the time and productivity savings they achieved.

More to Come!

This is just a selection of our favorite test automation videos that we’ve shared with the community this summer. We’re continuously sharing more too – keep an eye on our upcoming events page to see what we have in store next.

What were your favorite videos? Check out our full video library here, and you can let us know your own favorites @Applitools.

The post Test Automation Video Summer Roundup: May-August 2022 appeared first on Automated Visual Testing | Applitools.

Curious about the latest updates in Storybook.js, including the upcoming Storybook 7? In this post, which will be continuously updated, we sum up the latest Storybook news.

The highly anticipated Storybook 7.0 is currently in alpha, and there is a lot to get excited about. Let’s take a look at everything we know so far.

What’s Coming in Storybook 7

Storybook 7 promises significant changes. In fact, the Storybook team describes it as “a full rework of Storybook’s core with fast build and next-generation interaction testing.” Interaction testing was first included in the most recent Storybook 6.5 release, but we can expect further development there as Storybook 7 develops.

A Storybook 7 Design Preview

Storybook’s developers have just revealed a “sneak peek” at the design and layout in Storybook 7 [update 8/18: these changes are now available in the latest alpha]. Here’s a few of the changes that were highlighted:

• 3.5% more screen space for the Canvas (where components are developed in isolation).
• A “Reload” tool to reload a selected story (component) without refreshing the whole browser.
• Better access for integrators to the same design patterns used to develop Storybook.
• 196 icons that can be used in all projects, each redrawn from scratch. This is 20 more icons than the previous set.
• Form components like Toggle and Slider conform to the new design language.
• Pre-bundling of Storybook to deliver faster start times and avoid dependency issues (dependencies were also audited to reduce bundle size).

Storybook 7 Release Date

We don’t yet know the exact release date for Storybook 7, but we can guess based on their recent development history.

Storybook Version 6 was originally released in August 2020. Since then, there have been 5 major updates, most recently ending in version 6.5 which was released in May 2022. On the journey to the Storybook 6 release, the development team hit the following milestones:

• 47 alphas between January 2020 and April 2020
• 47 betas between April 2020 and July 2020
• 30 release candidates between July 2020 and August 2020

Storybook 7 began its first alpha in June 2022, and as of this writing is on alpha-26. If it follows the same trajectory is Storybook 6, we can estimate that it will enter beta in September, RC in December and official release in January 2023. Of course, time will tell, and we’ll update this post when any concrete information becomes available.

How Visual Testing Helps Component Testing

Component testing is form of software testing that focuses on software components in isolation. Component testing takes each rendered state (or Storybook story) and tests it.

Visual testing of components allows teams to find bugs earlier – and without writing any additional test code. It works across a variety of browsers and viewports at speeds almost as fast as unit testing.

You can learn more about how you can save time by using Applitools and our AI-powered visual testing with Storybook here:

• Visual Testing with Storybook
• Testing Storybook Components in Any Browser – Without Writing Any New Tests!
• How to Add Applitools Eyes to Storybook

Stay Tuned

We’ll be sure to keep this page updated with the latest on what’s new in Storybook 7, so check back often. And of course, we’re working hard to ensure our own Applitools SDKs for Storybook React, Storybook Angular and Storybook Vue are always compatible with the latest Storybook features.

Last Updated: August 26th, 2022

The post What’s New in Storybook 7? appeared first on Automated Visual Testing | Applitools.

User experience is becoming more and more critical to the success of applications every year, making the world of front-end testing a fast-moving and exciting place. Along with our partners at Netlify, we were excited to host our second annual Front-End Test Fest last month. The event featured expert speakers from global brands, lively Q&A sessions as well as games and fun. It was hosted by Angie Jones, Head of Developer Relations at Block, and Cassidy Williams, Head of Developer Experience and Education at Remote.

We’ve rounded up all the videos in one place, so feel free to dive right in, or keep reading for a recap of the event.

Front-End Test Fest 2022

The event began with opening remarks from hosts Angie and Cassidy, before Andrew Knight (AKA the Automation Panda) took the stage to present the opening keynote.

OPENING KEYNOTE: The State of the Union for Front End Testing – Andrew Knight

Web development moves fast, and new techniques are constantly emerging with promises of better performance and simpler maintenance. The JAM stack, server-side rendering frameworks, and APIs for all the things are on the rise. How can we most effectively test our web apps when they follow these new patterns? In this talk, Andrew Knight, Developer Advocate at Applitools and Automation Panda, explores seven major trends in front end testing to help unlock the best approaches, tools and frameworks you can use. Andrew considers the value of new techniques like component testing and visual testing, and touches on the future with a glance at autonomous testing. Check out this exciting “state of the union” on current trends in front end testing!

How to Avoid Common Mistakes When Writing Your Cypress Tests – Filip Hiric

Filip Hric, QA Lead at Slido and Cypress expert, knows the pain of a failed, flaky test. Cypress is a great tool, but it’s only as effective as the tests you design with it. In this talk, Filip shows you a few powerful tools that are built into Cypress that can help you navigate the rough terrain of flaky tests so you can start designing tests that are evergreen and stable. He discusses common mistakes, the proper use of commands and plenty more. Check out his great presentation.

Coffee Break!

One of the nice things about the conference was the “coffee breaks” between the sessions, that gave attendees a chance to “mingle” with some of the speakers. Gleb Bahmutov (Senior Director of Engineering, Mercari US), Tiffany Le-Nguyen, (Sr. Frontend Engineer, Netlify) and Nick Taylor (Staff Software Engineer, Netlify) were available at the morning break.

Shifting Accessibility Testing to the Left – Marie Drake

Why are there still so many websites that are not accessible? Many companies still view accessibility testing as an afterthought or something that will be difficult to implement. In this talk, Marie Drake, Quality Engineering Manager at Zoopla, busts some of these myths around accessibility testing and explains how you can shift it to the left. She also covers the many benefits of improving accessibility, and how to integrate accessibility testing into development and testing workflows. Watch the talk right here.

Expert Panel: Trending Tools and Frameworks – What’s Hype and What’s Not

Cutting through the marketing noise to understand what’s hype and what’s real can be a challenge. In this panel, you’ll hear from leaders Skyler Brungardt, Sr. Manager of Digital Experience & Commerce at Gap, Dan Giordano, Sr. Director of Product Marketing at Applitools, and Tiffany Le-Nguyen, Sr. Frontend Engineer, and Nick Taylor, Staff Software Engineer, both from Netlify.

Take a look at this panel for an exciting conversation based on hands-on experience with the latest tools and frameworks being used at leading companies. The panel was expertly moderated by Joe Colantonio, the founder of TestGuild.

Lunch Break: Making Music with (Live) Code – Dan Gorelick

This was a fun one. For the lunch break, independent developer and artist Dan Gorelick gave a live musical performance using only code. Using the coding framework TidalCycles, Dan briefly introduces us to the technology and how it works before improvising a great live show. Check this one out for a bit of musical theory, conversation around how to combine code with musical ideas, and just for some fun music.

Slice and Dice Your End-to-End Tests – Gleb Bahmutov

In this presentation, Gleb Bahmutov, Senior Director of Engineering at Mercari US, shares some practical tips and tricks for reliable end-to-end testing at serious scale. Using parallelization and a bit of custom tinkering, his CI system has his team running 500+ end-to-end front-end tests – and running them quickly. Want to get developers and other stakeholders quick, high-quality feedback from your E2E tests? Watch this talk.

Increasing Automated Test Flow in a Monorepo – David Lindley, Ben Hudson

David Lindley and Ben Hudson, Software Engineering Manager and Senior Software Engineer at Vodafone UK, respectively, talk about the challenges and benefits involved in one of their global projects – a shared React component library situation in a monorepo. The library has 220+ shared components and developers across the globe contribute to it, so ensuring quality is critical! David and Ben share how they overcame issues that came up as they progressed, including how they used dependency checking to make sure they test only what is needed, speeding up test flow. See it here.

Coffee Break!

Another opportunity to schmooze with the speakers! Andrew Knight (Developer Advocate and Automation Panda, Applitools), Adam Murray (Developer Experience Team Lead, Cypress.io) and Ben Hudson (Sr. Software Engineer, Vodafone UK) were hanging out to chat in this afternoon break.

Getting Started with Component Testing in Cypress – Adam Murray

In the latest Cypress 10 release, the Cypress team has added something new – official component testing (previously, it was an experimental feature). In this hands-on talk, Adam Murray, Developer Experience Team Lead at Cypress, helps you get going with it. He starts at the very beginning, showing you how to install Cypress 10, configure it in a new application, and add your first component tests. Adam goes on to cover more advanced topics like cy.intercept() as well. Take a look.

CLOSING KEYNOTE: Looking Forward – Trends That Will Shape Modern Web Development – Jason Lengstorf

The web ecosystem is changing quickly, and it can feel hard to keep up with all the changes. In this fascinating closing keynote, Jason Lengstorf, VP of Developer Experience at Netlify, delivers some comfort – things are perhaps not as “new and different” as they seem. Jason looks forwards and back at the landscape of the web, discussing not only where we’re going but how it all builds on historical trends. Check it out here.

After the closing keynote, Angie and Cassidy delivered the closing remarks as the event concluded.

Thank You!

We want to extend a huge thanks to everyone involved in making this event such a huge success. The list could go on, but we’ll start with Angie and Cassidy for being rock star hosts, our brilliant speakers for sharing their wisdom and experience, and of course to every attendee for adding your voices and presence. Thank you – we could never have delivered an event this great without all of you.

If you liked these videos, you might also like our videos from our previous virtual events too – all free to watch. Happy testing!

The post Front-End Test Fest 2022 Recap appeared first on Automated Visual Testing | Applitools.

In this guide, you’ll learn what visual regression testing is and why visual regression tests are important. We’ll go through a use case with an example and talk about how to get started and choose the best tool for your needs.

What is Visual Regression Testing?

Visual regression testing is a method of validating that changes made to an application do not negatively affect the visual appearance of the application’s user interface (UI). By verifying that the layout and visual elements align with expectations, the goal of visual regression testing is to ensure the user experience is visually perfect.

Visual regression testing is a kind of regression testing. In regression testing, an application is tested to ensure that a new change to the code doesn’t break existing functionality. Visual regression testing specifically focuses on verifying the appearance and the usability of the UI after a code change.

In other words, visual regression testing (also called just visual testing or UI testing) is focused on validating the appearance of all the visual elements a user interacts with or sees. These visual validations include the location, brightness, contrast and color of buttons, menus, components, text and much more.

Why is Visual Regression Testing Important?

Visual regression tests are important to prevent costly visual bugs from escaping into production. Failure to visually validate can severely compromise the user experience and in many cases lead directly to lost sales. This is because traditional functional testing works by simply validating data input and output. This method of testing catches many bugs, but it can’t discover visual bugs. Without visual testing these bugs are prone to slipping through even on an otherwise well tested application. 

As an example, here is a screenshot of a visual bug in production on the Southwest Airlines website:

This page would pass a typical suite of functional tests because all of the elements are present on the page and have loaded successfully. However, the visual bug is obvious. Not only that, but because the Terms and Conditions are inadvertently overlapping the button, the user literally cannot check out and complete their purchase. Visual regression testing would catch this kind of bug easily before it slipped into production.

Visual testing can also enhance functional testing practices and make them more efficient. Because visual tests can “see” the elements on a page they do not have to rely on individual coded assertions using unique selectors to validate each element. In a traditional functional testing suite, these assertions are often time-consuming to create and maintain as the application changes. Visual testing greatly simplifies that process.

How Do Visual Regression Tests Work?

At its core, visual regression testing works by capturing screenshots of the UI before a change is made and comparing it to a screenshot taken after. Differences are then highlighted for a test engineer to review. In practice, there are several different visual regression testing techniques available.

Types of Visual Regression Testing

• Manual visual testing: Visual regression testing can be done manually and without any tools. Designers and developers take time during every release to scan pages, manually looking for visual defects. While it is slow and extremely cumbersome to do this for an entire application, not to mention prone to human error, manual testing in this way can allow for ad-hoc or exploratory testing of the UI, especially at early stages of development.
• Pixel-by-Pixel comparison: This approach compares the two screenshots and analyzes each at the pixel level, alerting the test engineer of any discrepancies found. Pixel comparison, also called pixel diffs, will be certain to flag all possible issues, but will also include many irrelevant differences that are invisible to the human eye and have no effect on usability (such as rendering, anti-aliasing, or padding/margin differences). These “false-positives” must be painstakingly sifted through manually by the test engineer with every test run.
• DOM-based comparison: A comparison based on the Document Object Model (DOM) analyzes the DOM before and after a state change and flags any differences. This will be effective in drawing attention to any alterations in the code that comprises the DOM, but is not truly a visual comparison. False negatives/positives are frequently produced when the code does not change but the UI does (e.g.: dynamic content, embedded content, etc.) or when the code changes but the UI does not. As a result, test results are often flaky and must be slowly and carefully reviewed to avoid escaped visual bugs.
• Visual AI comparison: This type of visual regression testing leverages Visual AI, which uses computer vision to “see” the UI the same way a human would. A well-trained AI will be able to assist test engineers by only surfacing the kind of differences a human would notice, eliminating the time-consuming “false-positive” issues that plague pixel and DOM comparison tests. It can also include other capabilities, such as the ability to test dynamic content and flag issues only in the areas or regions where changes are not expected.

Automated Visual Testing Use Case and Example

Getting started with automated visual regression testing takes only a few steps. Let’s walk through the typical visual regression testing process and then consider a brief example.

1. Define your test scenarios. What will be captured in the screenshots, and at what point in the test will they be taken? With some automated tools, a basic test can be as simple as a single line of code that will take a screenshot of an entire page at the end of a test.
2. Use an automated testing tool to compare the new screenshots against a baseline image. The baseline is the most recent existing screenshot of the application that has already been approved by a tester.
3. The tool will automatically generate a report highlighting the differences found between the two images. Using pixel diff this will be every pixel difference found, or with Visual AI, you will see a report showing only meaningful differences. 
4. A test engineer reviews the report and determines what is a bug and what is an acceptable or valid change (a false positive). After all bugs are resolved, the baseline is updated with the new screenshot.

Visual Regression Testing Example

Let’s review a quick example of the four steps above with a basic use case, such as a login screen.

1. Define your test scenario: In this case, we’ll capture the entire screen and review for any changes. Our baseline might look like this:
   
2. Next, we’ll make some changes to the code, such as adding a row of social buttons. Unfortunately, doing so has pushed up the login button so that it is unusable. Our new login screen might look like this:
   
3. The tool will then compare the two and generate a report. In our example, we’ll use Visual AI, which will highlight only the relevant areas of change that a user would notice. In this case, that’s the row with the new social buttons, and the area with the now unusable button. The comparison would look like this:
   
4. A test engineer will then review the comparison. If any intentional changes were flagged, these are marked as accepted changes. Similarly, if there are expected changes in dynamic areas, these can be flagged for Visual AI to ignore going forward. Remaining areas flagged are marked as bugs to be addressed. In this case, every area flagged in red is problematic – the social buttons need to be shifted down, and the button needs to come down out of the password field. Once these are addressed, the test is run again, and a new baseline is created only when everything passes. The end result is free of visual defects:
   

How to Choose a Visual Testing Tool

Choosing the best tool for your visual regression tests will depend on your needs, as there are many options available. Here are some questions you should be asking as you consider a new tool:

• Automated or Manual? How frequently do you want to conduct visual tests? For occasional spot checks, manual testing may suffice. If you want to run tests with every change to ensure no visual bugs escape, automated testing will be much more efficient. For automated testing, consider the learning curve of the tool and how easy it is to integrate into your existing CI/CD workflow.
• Is your UI dynamic or static? How often does your user interface change? For completely static pages, simpler tools may serve to spot any visual bugs. Pages with dynamic content that changes regularly may be better served by tools with advanced capabilities like Visual AI.
• How many browsers/devices/platforms? Do you have many browsers, devices or platforms to cover with your tests? A tool may be efficient for a single combination but quite inefficient when attempting to cover a wide range of configurations. If you need to cover a broad range of situations, you need to make sure you pick a tool that can quickly re-render visual snapshots on different configurations, or achieving full coverage can become a time-consuming headache.
• Does your team have time? How much time does your QA team have to spend on UI testing? If they have capacity, sifting through potential false positives from pixel diff tools may not be an issue, or manual testing could be an option. For teams looking to be as efficient as possible, particularly with large or dynamic applications, automated visual testing with Visual AI will save time.
• What is your build/release frequency? Are you running tests infrequently, daily, or even multiple times a day? If testing is quite infrequent, you may be able to absorb some inefficiency in test execution. Organizations running tests regularly, or seeking to increase their test velocity, should place significant value in a tool that can enable their QA team to achieve full coverage by easily executing a large quantity of tests quickly.
• How many bugs are slipping through? For many teams, due to the increasing complexity of web and mobile development, visual bugs that can harm a company’s reputation or even sales escape more than they would like. In this case the value of automated visual testing is clear. However, if your team is catching all bugs or you can live with the level of bugs escaping, you may not need to invest in visual testing, at least for now.

Automated visual testing tools can be paid or open source. Visual testing tools are typically paired with an automated testing tool to automatically handle interactions and take screenshots. Some popular open source automated testing tools compatible with visual testing include Selenium for web testing and Appium for mobile testing.

Why Choose Automated Visual Regression Testing with Applitools

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy and includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

The Applitools Ultrafast Test Cloud includes unique features like the Ultrafast Grid, which can run your functional & visual tests once locally and instantly render them across any combination of browsers, devices, and viewports. Our automated maintenance capabilities make use of Visual AI to identify and group similar differences found across your test suite, allowing you to verify multiple checkpoint images at once and to replicate maintenance actions you perform for one step in other relevant steps within a batch.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Read More

• How do Baseline Comparisons Work in Applitools? [Video, 2:11]

• What is Visual Testing [blog]
• What is Visual AI [blog]
• Visual Testing for Mobile Apps [blog]
• The Impact of Visual AI On Test Automation [webinar recap]
• Modern Functional Test Automation Through Visual AI [free course]

The post What is Visual Regression Testing? appeared first on Automated Visual Testing | Applitools.

We’re approaching the end of May, which means we’re just a handful of weeks the midpoint of 2022 already. If you’re like me, you’re wondering where the year has gone. Maybe it has to do with life in the northeastern US where I live, where we’ve really just had our first week of warm weather. Didn’t winter just end?

As always, the year is flying by, and it can be hard to keep up with all the great videos or events you might have wanted to watch or attend. To help you out, we’ve rounded up some of our most popular test automation videos of 2022 so far. These are all top-notch workshops or webinars with test automation experts sharing their knowledge and their stories – you’ll definitely want to check them out.

Cross Browser Test Automation

Cross-browser testing is a well-known challenge to test automation practitioners. Luckily, Andy Knight, AKA the Automation Panda, is here to walk you through a modern approach to getting it done. Whether you use Cypress, Playwright, or are testing Storybook components, we have something for you.

Modern Cross Browser Testing with Cypress

For more, see this blog post: How to Run Cross Browser Tests with Cypress on All Browsers (plus bonus post specifically covering the live Q&A from this workshop).

Modern Cross Browser Testing in JavaScript Using Playwright

For more, see this blog post: Running Lightning-Fast Cross-Browser Playwright Tests Against any Browser.

Modern Cross Browser Testing for Storybook Components

For more, see this blog post: Testing Storybook Components in Any Browser – Without Writing Any New Tests!

Test Automation with GitHub or Chrome DevTools

GitHub and Chrome DevTools are both incredibly popular with the developer and testing communities – odds are if you’re reading this you use one or both on a regular basis. We recently spoke with developer advocates Rizel Scarlett of GitHub and Jecelyn Yeen of Google as they explained how you can leverage these extremely popular tools to become a better tester and improve your own testing experience. Click through for more info about each video and get watching.

Make Testing Easy with GitHub

For more, see this blog post: Using GitHub Copilot to Automate Tests.

Automating Tests with Chrome DevTools Recorder

For more, see this blog post: Creating Your First Test With Google Chrome DevTools Recorder.

Test Automation Stories from Our Customers

When it comes to implementing and innovating around test automation, you’re never alone, even though it doesn’t always feel that way. Countless others are struggling with the same challenges that you are and coming up with solutions. Sometimes all it takes is hearing how someone else solved a similar problem to spark an idea or gain a better understanding of how to solve your own.

Accelerating Visual Testing

Nina Westenbrink, Software Engineer at a leading European telecom, talks about how the visual time to test the company’s design system was decreased and simplified, offering helpful tips and tricks along the way. Nina also speaks about her career as a woman in testing and how to empower women and overcome biases in software engineering.

Continuously Testing UX for Enterprise Platforms

Govind Ramachandran, Head of Testing and Quality Assurance for Asia Technology Services at Manulife Asia, discusses challenges around UI/UX testing for enterprise-wide digital programs. Check out his blueprint for continuous testing of the customer experience using Figma and Applitools.

This is just a taste of our favorite videos that we’ve shared with the community from 2022. What were yours? You can check out our full video library here, and let us know your own favorites @Applitools.

The post Our Best Test Automation Videos of 2022 (So Far) appeared first on Automated Visual Testing | Applitools.

What is Functional Testing?

Functional testing is a type of software testing where the basic functionalities of an application are tested against a predetermined set of specifications. Using Black Box Testing techniques, functional tests measure whether a given input returns the desired output, regardless of any other details. Results are binary: tests pass or fail.

Why is Functional Testing Important?

Functional testing is important because without it, you may not accurately understand whether your application functions as intended. An application may pass non-functional tests and otherwise perform well, but if it doesn’t deliver the key expected outputs to the end-user, the application cannot be considered working.

What is the Difference between Functional and Non-Functional Testing?

Functional tests verify whether specified functional requirements are met, where non-functional tests can be used to test non-functional things like performance, security, scalability or quality of the application. To put it another way, functional testing is concerned with if key functions are operating, and non-functional tests are more concerned with how the operations take place.

Examples of Functional Testing Types

There are many types of functional tests that you may want to complete as you test your application. 

A few of the most common include:

Unit Testing

Unit testing breaks down the desired outcome into individual units, allowing you to test whether a small number of inputs (sometimes just one) produce the desired output. Unit tests tend to be among the smallest tests to write and execute quickly, as each is designed to cover only a single section of code (a function, method, object, etc.) and verify its functionality.

Smoke Testing

Smoke testing is done to verify that the most critical parts of the application work as intended. It’s a first pass through the testing process, and is not intended to be exhaustive. Smoke tests ensure that the application is operational on a basic level. If it’s not, there’s no need to progress to more detailed testing, and the application can go right back to the development team for review.

Sanity Testing

Sanity testing is in some ways a cousin to smoke testing, as it is also intended to verify basic functionality and potentially avoid detailed testing of broken software. The difference is that sanity tests are done later in the process in order to test whether a new code change has had the desired effect. It is a “sanity check” on a specific change to determine if the new code roughly performs as expected. 

Integration Testing

Integration testing determines whether combinations of individual software modules function properly together. Individual modules may already have passed independent tests, but when they are dependent on other modules to operate successfully, this kind of testing is necessary to ensure that all parts work together as expected.

Regression Testing

Regression testing makes sure that the addition of new code does not break existing functionalities. In other words, did your new code cause the quality of your application to “regress” or go backwards? Regression tests target the changes that were made and ensure the whole application continues to remain stable and function as expected.

User Acceptance Testing (UAT)/Beta Testing

Usability testing involves exposing your application to a limited group of real users in a production environment. The feedback from these live users – who have no prior experience with the application and may discover critical bugs that were unknown to internal teams – is used to make further changes to the application before a full launch.

UI/UX Testing 

UI/UX testing evaluates the graphical user interface of the application. The performance of UI components such as menus, buttons, text fields and more are verified to ensure that the user experience is ideal for the application’s users. UI/UX testing is also known as visual testing and can be manual or automated.

Other classifications of functional testing include black box testing, white box testing, component testing, API testing, system testing and production testing.

How to Perform Functional Testing

The essence of a functional test involves three steps:

• Determine the desired test input values
• Execute the tests
• Evaluate the resulting test output values

Essentially, when you executed a task with input (e.g.: enter an email address into a text field and click submit), did your application generate the expected output (e.g.: user is subscribed and thank you page is displayed)?

We can understand this further with a quick example.

Functional Testing Example

Let’s begin with a straightforward application: a calculator. 

To create a set of functional tests, you would need to:

• Evaluate all the possible inputs – such as numbers and mathematical symbols – and design assertions to test their functionality
• Execute the tests (either automated or manually)
• Ensure that the desired outputs are generated – e.g.: each mathematical function works as intended, the final result is given correctly in all cases, the formula history is displayed accurately, etc.

For more on how to create a functional test, you can see a full guide on how to write an automated functional test for this example.

Functional Testing Techniques 

There are many functional testing techniques you might use to design a test suite for this:

• Boundary value tests evaluate what happens if inputs are received outside of specified limits – such as a user entering a number that was too large (if there is a specified limit) or attempting to enter non-numeric input
• Decision-based tests verify the results after a user decides to take an action, such as clearing the history
• User-based tests evaluate how components work together within an application – if the calculator’s history was stored in the cloud, this kind of test would verify that it did so successfully
• Ad-Hoc tests can be done at the end to try and discover bugs other methods did not uncover by seeking to break the application and check its response

Other common functional testing techniques include equivalence testing, alternate flow testing, positive testing and negative testing.

Automated Functional Testing vs Manual Functional Testing

Manual functional testing requires a developer or test engineer to design, create and execute every test by hand. It is flexible and can be powerful with the right team. However, as software grows in complexity and release windows get shorter, a purely manual testing strategy will face challenges keeping up a large degree of test coverage.

Automated functional testing automates many parts of the testing process, allowing tests to run continuously without human interaction – and with less chance for human error. Tests must still be designed and have their results evaluated by humans, but recent improvements in AI mean that with the right tool an increasing share of the load can be handled autonomously.

How to Use Automated Visual Testing for Functional Tests

One way to automate your functional tests is by using automated visual testing. Automated visual testing uses Visual AI to view software in the same way a human would, and can automatically highlight any unexpected differences with a high degree of accuracy.

Visual testing allows you to test for visual bugs, which are otherwise extremely challenging to uncover with traditional functional testing tools. For example, if an unrelated change caused a “submit” button to be shifted to the far right of a page and it could no longer be clicked by the user, but it was still technically on the page and using the correct identifier, it would pass a traditional functional test. Visual testing would catch this bug and ensure functionality is not broken by a visual regression.

How to Choose an Automated Testing Tool?

Here are a few key considerations to keep in mind when choosing an automated testing tool:

• Ease of Use: Is it something easy for your existing QA team to use, or easy to hire for? Does it require an extensive learning curve or can it be picked up quickly?
• Flexibility: Can it be used across different platforms? Can it easily integrate with your current testing environment, and does it allow you the freedom to change your environment in the future?
• Reusability/AI Assistance: How easy is it to reuse tests, particularly if the UI changes? Is there meaningful AI that can help you test more efficiently, particularly at the scale you need?
• Support: What level of customer support do you require, and how easily can you receive it from the provider of your tool?

Automated testing tools can be paid or open source. Some popular open source tools include Selenium for web testing and Appium for mobile testing.

Why Choose Automated Visual Testing with Applitools

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy and includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Keep Learning

Looking to learn more about Functional Testing? Check out the resources below to find out more.

• Applitools Eyes: Modern Functional Testing [Video, 1:40]

• Modern Functional Test Automation Through Visual AI [Free Course]
• Why Should Software Testers Understand Unit Testing? [Blog]
• The end of Smoke, Sanity and Regression [Blog]
• The Impact of Visual AI On Test Automation [webinar recap]
• Cutting-edge Functional UI Testing Techniques [webinar recap]
  

The post What is Functional Testing? Types and Example (Full Guide) appeared first on Automated Visual Testing | Applitools.

Applitools is proud to announce its positioning in the Major Players category in the IDC MarketScape: Worldwide Cloud Testing 2022 Vendor Assessment — Empowering Business Velocity. Applitools was also positioned in the Major Players category in the IDC MarketScape: Worldwide Mobile Testing and Digital Quality 2022 Vendor Assessment — Enabling Multimodal Dynamism for Digital Innovation.

Written by Melinda-Carol Ballou, Research Director, Agile ALM, Quality & Portfolio Strategies at IDC Research, this IDC study uses the IDC MarketScape model to provide an assessment of 24 vendors for worldwide cloud testing and enterprise automated software quality (ASQ) SaaS solutions.

“Participating vendors needed to have sufficient cloud testing automated software quality capabilities available in key areas of concern (e.g., test infrastructure provisioning and configuration management; deep analytics for analysis of performance optimization, service virtualization, and architectural and other analysis to enable visibility into the health of applications deployed in native and hybrid cloud; readiness for software targeting the cloud; and/or delivery of their ASQ software solution in the cloud with partner integration for other capabilities) for IDC clients.”

Source: IDC

Applitools products considered as part of these vendor assessments include Applitools Eyes and Applitools Ultrafast Test Cloud (the combination of Applitools Eyes, Applitools Ultrafast Grid and Applitools Native Mobile Grid).

How Applitools Enables Automated Software Quality

As software development teams rapidly deliver new products and services to market through more frequent and shorter release cycles, they struggle to fully test the customer experience due to increasing application complexity and an explosion of device/browser combinations. When development teams are confident they can fix functional and visual bugs faster, they can push more high-quality code faster than ever before.

To enable automated software quality consistently, quickly, and at a fraction of the cost, Applitools extends to its customers the power of Applitools Visual AI—the only AI-powered computer vision that accurately mimics the human eye and brain to avoid undetected functional and visual bugs, minimizing false positive bug alerts.

Visual AI is trained on more than 1 billion images and supports analysis across custom regions, with advanced comparison modes/match levels and auto-maintenance of test results. Tests infused with Visual AI (via Applitools Eyes) are created 5.8x faster, run 3.8x more stable, and catch 45% more bugs vs. traditional functional testing. In addition, tests powered by Visual AI can take advantage of the ultrafast speed and stability of Applitools Ultrafast Test Cloud.

The Ultrafast Test Cloud can instantly validate entire application pages and detect issues on even the most complex and dynamic pages. It allows users to write and execute tests once locally with support for more than 50 test frameworks and programming languages including: Cypress, Storybook, Selenium Java, Selenium JavaScript, Selenium C#, Selenium Python, Selenium Ruby, Selenium IDE, Webdriver.IO, TestCafe, and more.

A single functional test run captures the DOM & CSS rules for every browser state, automatically rendering it in parallel across all browsers (Chrome, Firefox, Safari, Edge, and Internet Explorer) and viewports using the Ultrafast Grid. Screenshots are then instantly analyzed by Applitools Eyes to find functional and visual bugs. 

Applitools integrates with all the major CI/CD platforms, including GitHub, GitLab, Bitbucket, Jenkins, Azure DevOps, Travis CI, Circle CI, Semaphore, TeamCity, and Bamboo, as well as with defect- tracking/collaboration systems, including Jira, CA Rally, Microsoft Teams, and Slack. The Applitools Eyes dashboard also enables collaboration between design, product, development, testing, and DevOps teams.

Automated software testing powered by Visual AI can help developers test 18 times faster across the full test cycle including writing, running, analyzing, reporting, and maintaining tests. That’s because Visual AI-powered automated tests that leverage the Ultrafast Test Cloud can run 30-50x faster than traditional solutions, with 99.9999% accuracy.* 

“References contacted by IDC found that Applitools’ solution led to greater efficiency in testing operations. They reported cost savings of up to 95% by making the switch to Applitools in combination with adoption of containers — additional testing tools were not required. Another customer reported that it was able to increase the speed of testing to reduce approval time for moving websites to production from 29 days to 1.5 hours. This is especially significant for a company maintaining 2,400 websites — Applitools helped ensure that sites maintained the same digital experience look and feel across pages as they changed dynamically, supporting up to five browsers and four viewpoints”.

Source: IDC

Applitools Customers, Use Cases and Community 

Applitools is helping more than 400 of the world’s top digital brands release, test, and monitor flawless mobile, web, and native apps in a fully automated way. We help our customers modernize critical test automation use cases — functional and visual regression testing, web and mobile UI/UX testing, cross browser / cross device testing, localization testing, PDF testing, digital accessibility and legal/compliance testing — to transform the way their businesses deliver innovation at the speed of DevOps without jeopardizing brand integrity.

Learn about the impact Applitools is having on these organizations by reading the customer stories in our case study library: https://applitools.com/case-studies/.  

To support the community, Applitools also maintains and manages Test Automation University, a free online learning community with over 100,000 members, that hosts more than 50 courses on a wide range of test automation topics and best practices.

“Being recognized as a major player in the IDC MarketScape for worldwide cloud testing underscores our ability to bring speed and stability to automated software quality with Visual AI and Ultrafast Test Cloud. We’re the world’s fastest-growing software testing company because our customers are using the most advanced, yet simple way to ensure brand integrity across any digital end-user experience. Their success is our success.”

Moshe Milman, COO and co-founder, Applitools

Thank You

It takes a village. Our fantastic team has worked long and hard to achieve this recognition from IDC Research, but this is also the direct result of the feedback and collaboration we’ve had with our customers – we could not have done it without you. Because of our strong community and our valued partnerships, the industry has also recognized Applitools as the:

• Best Testing Service/Tool in the DevOps Dozen² Awards for 2021
• Leading software vendor in testing in the 2021 North American Testing Awards
• Fastest growing software testing company in North America on the 2021 Deloitte Technology Fast 500 list 
• Category leader in automated testing on GetApp and Capterra

Thank you for trusting Applitools to deliver flawless automated testing for you, and we’re excited to head into the future of testing together.

Learn More

For more, schedule a demo and see for yourself how Visual AI is helping industry leaders deliver visually perfect digital experiences.

*Modern Cross Browser Testing Through Visual AI Report https://applitools.com/modern-cross-browser-testing-report/ 

The post Applitools Recognized as ‘Major Player’ in IDC MarketScape: Worldwide Cloud Testing 2022 Vendor Assessment appeared first on Automated Visual Testing | Applitools.

In this guide, we’ll explore Visual Artificial Intelligence (AI) and what it means. Read on to learn what Visual AI is, how it’s being applied today, and why it’s critical across a range of industries – and in particular for software development and testing.

From the moment we open our eyes, humans are highly visual creatures. The visual data we process today increasingly comes in digital form. Whether via a desktop, a laptop, or a smartphone, most people and businesses rely on having an incredible amount of computing power available to them and the ability to display any of millions of applications that are easy to use.

The modern digital world we live in, with so much visual data to process, would not be possible without Artificial Intelligence to help us. Visual AI is the ability for computer vision to see images in the same way a human would. As digital media becomes more and more visual, the power of AI to help us understand and process images at a massive scale has become increasingly critical.

What is AI? Background on Artificial Intelligence and Machine Learning

Artificial Intelligence refers to a computer or machine that can understand its environment and make choices to maximize its chance of achieving a goal. As a concept, AI has been with us for a long time, with our modern understanding informed by stories such as Mary Shelley’s Frankenstein and the science fiction writers of the early 20^th century. Many of the modern mathematical underpinnings of AI were advanced by English mathematician Alan Turing over 70 years ago.

Since Turing’s day, our understanding of AI has improved. However, even more crucially, the computational power available to the world has skyrocketed. AI is able to easily handle tasks today that were once only theoretical, including natural language processing (NLP), optical character recognition (OCR), and computer vision.

What is Visual Artificial Intelligence (Visual AI)?

Visual AI is the application of Artificial Intelligence to what humans see, meaning that it enables a computer to understand what is visible and make choices based on this visual understanding.

In other words, Visual AI lets computers see the world just as a human does, and make decisions and recommendations accordingly. It essentially gives software a pair of eyes and the ability to perceive the world with them.

As an example, seeing “just as a human does” means going beyond simply comparing the digital pixels in two images. This “pixel comparison” kind of analysis frequently uncovers slight “differences” that are in fact invisible – and often of no interest – to a genuine human observer. Visual AI is smart enough to understand how and when what it perceives is relevant for humans, and to make decisions accordingly.

How is Visual AI Used Today?

Visual AI is already in widespread use today, and has the potential to dramatically impact a number of markets and industries. If you’ve ever logged into your phone with Apple’s Face ID, let Google Photos automatically label your pictures, or bought a candy bar at a cashierless store like Amazon Go, you’ve engaged with Visual AI. 

Technologies like self-driving cars, medical image analysis, advanced image editing capabilities (from Photoshop tools to TikTok filters) and visual testing of software to prevent bugs are all enabled by advances in Visual AI.

How Does Visual AI Help?

One of the most powerful use cases for AI today is to complete tasks that would be repetitive or mundane for humans to do. Humans are prone to miss small details when working on repetitive tasks, whereas AI can repeatedly spot even minute changes or issues without loss of accuracy. Any issues found can then either be handled by the AI, or flagged and sent to a human for evaluation if necessary. This has the dual benefit of improving the efficiency of simple tasks and freeing up humans for more complex or creative goals.

Visual AI, then, can help humans with visual inspection of images. While there are many potential applications of Visual AI, the ability to automatically spot changes or issues without human intervention is significant. 

Cameras at Amazon Go can watch a vegetable shelf and understand both the type and the quantity of items taken by a customer. When monitoring a production line for defects, Visual AI can not only spot potential defects but understand whether they are dangerous or trivial. Similarly, Visual AI can observe the user interface of software applications to not only notice when changes are made in a frequently updated application, but also to understand when they will negatively impact the customer experience.

How Does Visual AI Help in Software Development and Testing Today?

Traditional testing methods for software testing often require a lot of manual testing. Even at organizations with sophisticated automated testing practices, validating the complete digital experience – requiring functional testing, visual testing and cross browser testing – has long been difficult to achieve with automation. 

Without an effective way to validate the whole page, Automation Engineers are stuck writing cumbersome locators and complicated assertions for every element under test. Even after that’s done, Quality Engineers and other software testers must spend a lot of time squinting at their screens, trying to ensure that no bugs were introduced in the latest release. This has to be done for every platform, every browser, and sometimes every single device their customers use. 

At the same time, software development is growing more complex. Applications have more pages to evaluate and increasingly faster – even continuous – releases that need testing. This can result in tens or even hundreds of thousands of potential screens to test (see below). Traditional testing, which scales linearly with the resources allocated to it, simply cannot scale to meet this demand. Organizations relying on traditional methods are forced to either slow down releases or reduce their test coverage.

At Applitools, we believe AI can transform the way software is developed and tested today. That’s why we invented Visual AI for software testing. We’ve trained our AI on over a billion images and use numerous machine learning and AI algorithms to deliver 99.9999% accuracy. Using our Visual AI, you can achieve automated testing that scales with you, no matter how many pages or browsers you need to test. 

That means Automation Engineers can quickly take snapshots that Visual AI can analyze rather than writing endless assertions. It means manual testers will only need to evaluate the issues Visual AI presents to them rather than hunt down every edge and corner case. Most importantly, it means organizations can release better quality software far faster than they could without it.

How Visual AI Enables Cross Browser/Cross Device Testing

Additionally, due to the high level of accuracy, and efficient validation of the entire screen, Visual AI opens the door to simplifying and accelerating the challenges of cross browser and cross device testing. Leveraging an approach for ‘rendering’ rather than ‘executing’ across all the device/browser combinations, teams can get test results 18.2x faster using the Applitools Ultrafast Test Cloud than traditional execution grids or device farms.

How Will Visual AI Advance in the Future?

As computing power increases and algorithms are refined, the impact of Artificial Intelligence, and Visual AI in particular, will only continue to grow.

In the world of software testing, we’re excited to use Visual AI to move past simply improving automated testing – we are paving the way towards autonomous testing. For this vision (no pun intended), we have been repeatedly recognized as an industry leader by the industry and our customers.

Keep Reading: More about Visual AI and Visual Testing

What is Visual Testing (blog)

The Path to Autonomous Testing (video)

What is Applitools Visual AI (learn)

Why Visual AI Beats Pixel and DOM Diffs for Web App Testing (article)

How AI Can Help Address Modern Software Testing (blog)

The Impact of Visual AI on Test Automation (report)

How Visual AI Accelerates Release Velocity (blog)

Modern Functional Test Automation Through Visual AI (free course)

Computer Vision defined (Wikipedia)

The post What is Visual AI? appeared first on Automated Visual Testing | Applitools.