Learn everything you need to know about how to perform UI testing, including why it’s important, a demo of a UI test, and tips and tricks to make UI testing easier.

When users explore web, mobile or desktop applications, the first thing they see is the User Interface (UI). As digital applications become more and more central to the way we all live and work, the way we interact with our digital apps is an increasingly critical part of the user experience.

There are many ways to test an application: Functional testing, regression testing, visual testing, cross-browser testing, cross-device testing and more. Where does UI testing fit into this mix?

UI testing is essential to ensure that the usability and functionality of an application performs as expected. This is critical for delivering the kinds of user experiences that ensure an application’s success. After all, nobody wants to use an app where text is unreadable, or where buttons don’t work. This article will explain the fundamentals of UI testing, why it’s important, and supply a UI testing checklist and examples to help you get started.

What is UI Testing?

UI testing is the process of validating that the visual elements of an application perform as expected. In UI Testing, graphical components such as text, radio buttons, checkboxes, buttons, colors, images and menus are evaluated against a set of specifications to determine if the UI is displaying and functioning correctly.

Why is UI Testing Important?

UI testing is an important way to ensure an application has a reliable UI that always performs as expected. It’s critical for catching visual and even functional bugs that are almost impossible to detect using other kinds of testing.

Modern UI testing, which typically utilizes visual testing, works by validating the visual appearance of an application, but it does much more than make sure things simply look correct. Your application’s functionality can be drastically affected by a visual bug. UI testing is critical for verifying the usability of your UI.

Note: What’s the difference between UI testing and GUI testing? Modern applications are heavily dependent on graphical user interfaces (GUIs). Traditional UI testing can include other forms of user interfaces, including CLIs, or can use DOM-based coded locators to try and verify the UI rather than images. Modern UI testing today frequently involves visual testing.

Let’s take an example of a visual bug that slipped into production from the Southwest Airlines website:

Under a traditional functional testing approach this would pass the test suite. All the elements are present on the page and successfully loaded. But for the user, it’s easy to see the visual bug. 

This does more than deliver a negative user experience that may harm your brand. In this example, the Terms and Conditions are directly overlapping the ‘continue’ button. It’s literally impossible for the user to check out and complete the transaction. That’s a direct hit to conversions and revenue.

With good UI testing in place, bugs like these will be caught before they become visible to the user.

UI Testing Approaches

Manual Testing

Manual UI testing is performed by a human tester, who evaluates the application’s UI against a set of requirements. This means the manual tester must perform a set of tasks to validate that the appearance and functionality of every UI element under test meets expectations. The downsides of manual testing are that it is a time-consuming process and that test coverage is typically low, particularly when it comes to cross-browser or cross-device testing or in CI/CD environments (using Jenkins, etc.). Effectiveness can also vary based on the knowledge of the tester.

Record and Playback Testing

Record and Playback UI testing uses automation software and typically requires limited or no coding skill to implement. The software first records a set of operations executed by a tester, and then saves them as a test that can be replayed as needed and compared to the expected results. Selenium IDE is an example of a record and playback tool, and there is even one built directly into Google Chrome.

Model-Based Testing

Model-based UI testing uses a graphical representation of the states and transitions that an application may undergo in use. This model allows the tester to better understand the system under test. That means tests can be generated and potentially automated more efficiently. In its simplest form, the approach requires the steps below:

1. Build a model representing the system
2. Determine the inputs
3. Understand the expected outputs
4. Execute the tests and compare the results against expectations

Automated UI Testing vs Manual UI Testing

Benefits of Manual UI Testing

Manual testing, as we have seen above, has a few severe limitations. Because the process relies purely on humans performing tasks one at a time, it is a slow process that is difficult to scale effectively. Manual testing does, however, have advantages:

• Manual testing can potentially be done with little to no tooling, and may be sufficient for early application prototypes or very small apps. 
• An experienced manual tester may be able to discover bugs in edge-cases through ad-hoc or exploratory testing, as well as intuitively “feel” the user experience in a way that is difficult to understand with a scripted test.

Benefits of Automated UI Testing

In most cases automation will help testing teams save time by executing pre-determined tests repeatedly. Automation testing frameworks aren’t prone to human errors and can run continuously. They can be parallelized and executed easily at scale. With automated testing, as long as tests are designed correctly they can be run much more frequently with no loss of effectiveness. 

Automation testing frameworks may be able to increase efficiency even further with specialized capabilities for things like cross-browser testing, mobile testing, visual AI and more.

UI Testing Checklist of Test Cases

On the surface, UI testing is simple – just make sure everything “looks” good. Once you poke beneath that surface, testers can quickly find themselves encountering dozens of different types of UI elements that require verification. Here is a quick checklist you can use to make sure you’ve considered all the most common items.

UI Testing Checklist – Common Tests

• Text: Can all text be read? Is the contrast legible? Is anything covered by another element?
• Forms, Fields and Pickers: Are all text fields be visible, and can text be entered and submitted? Do all dropdowns display correctly? Are validation requirements (such as a date in a datepicker) upheld?
• Navigation and Sorting: Whether it’s a site menu, a sortable table or a multi-page form, can the user navigate via the UI? Do all dropdowns display? Can all options be clicked/tapped, and do they have the desired effect? 
• Buttons and Links: Are all buttons and links visible? Are they formatted consistently? Can they be selected, and do they take the user to the intended pages?
• Responsiveness: When you adjust the resolution, do all of the above UI elements continue to behave as intended?

Each of the above must be tested across every page, table, form and menu that your application contains. 

It’s also a good practice to test the UI for specific critical end-to-end user journeys. For example, making sure that it’s possible to journey smoothly from: User clicks Free Trial Signup (Button) > User submits Email Address (Form) > User Logs into Free Trial (Form) > User has trial access (Product)

Challenges of UI Testing

UI testing can be a challenge for many reasons. With the proper tooling and preparation these challenges can be overcome, but it’s important to understand them as you plan your UI testing strategy.

• User Interfaces are complex: As we’ve discussed above, there are numerous distinct elements on each page that must be tested. Embedded forms, iFrames, dropdowns, tables, images, videos and more must all be tested to be sure the UI is working as intended.
• User Interfaces change fast: For many applications the UI is in a near-constant state of flux, as frequent changes to the text, layout or links are implemented. Maintaining full coverage is challenging when this occurs.
• User Interfaces can be slow: Testing the UI of an application can take time, especially compared to smaller and faster tests like unit tests. Depending on the tool you are using, this can make them feel difficult to run as regularly.
• Testing script bottlenecks: Because the UI changes so quickly, not only do testers have to design new test cases, but depending on your tooling, you may have to constantly create new coded test scripts. Testing tools with advanced capabilities, like the Visual AI in Applitools, can mitigate this by requiring far less code to deliver the same coverage.

UI Testing Example

Let’s take an example of an app with a basic use case, such as a login screen.

Even a relatively simple page like this one will have numerous important test cases (TC):

• TC 1: Is the logo at the top appropriate for the screen, and aligned with brand guidelines?
• TC 2: Is the title of the page displaying correctly (font, label, position)?
• TC 3: Is the dividing line displaying correctly? 
• TC 4: Is the Username field properly labeled (font, label, position)?
• TC 5: Is the icon by the Username field displaying correctly?
• TC 6: Is the Username text field accepting text correctly (validation, error messages)?
• TC 7: Is the Password field properly labeled (font, label, position)?
• TC 8: Is the icon by the Password field displaying correctly?
• TC 9: Is the Password text field accepting text correctly (validation, error messages)?
• TC 10: Is the Log In button text displaying correctly (font, label, position)?
• TC 11: Is the Log In button functioning correctly on click (clickable, verify next page)
• TC 12: Is the Remember Me checkbox title displaying correctly (font, label, position)?
• TC 13: Is the Remember Me checkbox functioning correctly on click (clickable, checkbox displays, cookie is set)?

Simply testing each scenario on a single page can be a lengthy process. Then, of course, we encounter one of the challenges listed above – the UI changes quickly, requiring frequent regression testing. 

How to Simplify UI Testing with Automation

Performing this regression testing manually while maintaining the level of test coverage necessary for a strong user experience is possible, but would be a laborious and time-consuming process. One effective strategy to simplify this process is to use automated tools for visual regression testing to verify changes to the UI.

Benefits of Automated Visual Regression Testing for UI Testing

Visual regression testing is a method of ensuring that the visual appearance of the application’s UI is not negatively affected by any changes that are made. While this process can be done manually, modern tools can help you automate your visual testing to verify far more tests far more quickly.

Automated Visual UI Testing Example

Let’s return to our login screen example from earlier. We’ve verified that it works as intended, and now we want to make sure any new changes don’t negatively impact our carefully tested screen. We’ll use automated visual regression testing to make this as easy as possible.

1. As we saw above, our baseline screen looks like this:
   
2. Next, we’ll make a change by adding a row of social buttons. Unfortunately, this will have the effect of inadvertently rendering our login button unusable by pushing it up into the password field:
   
3. We’ll use our automated visual testing tool to evaluate our change against the baseline. In our example, we’ll use a tool that utilizes Visual AI to highlight only the relevant areas of change that a user would notice. The tool would then bring our attention to the new social buttons along with the section around the now unusable button as areas of concern.
   
4. A test engineer will then review the comparison. Any intentional changes that were flagged are marked as accepted changes. On some screens we might expect changes in certain dynamic areas, and these can be flagged for Visual AI to ignore going forward.
   
   We need to address only the remaining areas that are flagged. In our example, every area flagged in red is problematic – we need to shift down the social buttons, and move the button out of the password field. Once we’ve done this, we run test again, and a new baseline is created only when everything passes. The final result is free of visual defects:
   
   

Why Choose Automated Visual Regression Testing with Applitools for UI Testing

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy and includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

The Applitools Ultrafast Test Cloud includes unique features like the Ultrafast Grid, which can run your functional & visual tests once locally and instantly render them across any combination of browsers, devices, and viewports. Our automated maintenance capabilities make use of Visual AI to identify and group similar differences found across your test suite, allowing you to verify multiple checkpoint images at once and to replicate maintenance actions you perform for one step in other relevant steps within a batch.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Read More

• How do Baseline Comparisons Work in Applitools? [Video, 2:11]

• What is Visual Regression Testing [blog]
• Top 9 Tips to Increase UI Automation Speed [blog]
• What is Visual Testing [blog]
• What is Visual AI [blog]
• Overview of visual UI testing [docs]
• The Impact of Visual AI On Test Automation [webinar recap]
• Modern Functional Test Automation Through Visual AI [free course]

The post UI Testing: A Getting Started Guide and Checklist appeared first on Automated Visual Testing | Applitools.

What is Functional Testing?

Functional testing is a type of software testing where the basic functionalities of an application are tested against a predetermined set of specifications. Using Black Box Testing techniques, functional tests measure whether a given input returns the desired output, regardless of any other details. Results are binary: tests pass or fail.

Why is Functional Testing Important?

Functional testing is important because without it, you may not accurately understand whether your application functions as intended. An application may pass non-functional tests and otherwise perform well, but if it doesn’t deliver the key expected outputs to the end-user, the application cannot be considered working.

What is the Difference between Functional and Non-Functional Testing?

Functional tests verify whether specified functional requirements are met, where non-functional tests can be used to test non-functional things like performance, security, scalability or quality of the application. To put it another way, functional testing is concerned with if key functions are operating, and non-functional tests are more concerned with how the operations take place.

Examples of Functional Testing Types

There are many types of functional tests that you may want to complete as you test your application. 

A few of the most common include:

Unit Testing

Unit testing breaks down the desired outcome into individual units, allowing you to test whether a small number of inputs (sometimes just one) produce the desired output. Unit tests tend to be among the smallest tests to write and execute quickly, as each is designed to cover only a single section of code (a function, method, object, etc.) and verify its functionality.

Smoke Testing

Smoke testing is done to verify that the most critical parts of the application work as intended. It’s a first pass through the testing process, and is not intended to be exhaustive. Smoke tests ensure that the application is operational on a basic level. If it’s not, there’s no need to progress to more detailed testing, and the application can go right back to the development team for review.

Sanity Testing

Sanity testing is in some ways a cousin to smoke testing, as it is also intended to verify basic functionality and potentially avoid detailed testing of broken software. The difference is that sanity tests are done later in the process in order to test whether a new code change has had the desired effect. It is a “sanity check” on a specific change to determine if the new code roughly performs as expected. 

Integration Testing

Integration testing determines whether combinations of individual software modules function properly together. Individual modules may already have passed independent tests, but when they are dependent on other modules to operate successfully, this kind of testing is necessary to ensure that all parts work together as expected.

Regression Testing

Regression testing makes sure that the addition of new code does not break existing functionalities. In other words, did your new code cause the quality of your application to “regress” or go backwards? Regression tests target the changes that were made and ensure the whole application continues to remain stable and function as expected.

User Acceptance Testing (UAT)/Beta Testing

Usability testing involves exposing your application to a limited group of real users in a production environment. The feedback from these live users – who have no prior experience with the application and may discover critical bugs that were unknown to internal teams – is used to make further changes to the application before a full launch.

UI/UX Testing 

UI/UX testing evaluates the graphical user interface of the application. The performance of UI components such as menus, buttons, text fields and more are verified to ensure that the user experience is ideal for the application’s users. UI/UX testing is also known as visual testing and can be manual or automated.

Other classifications of functional testing include black box testing, white box testing, component testing, API testing, system testing and production testing.

How to Perform Functional Testing

The essence of a functional test involves three steps:

• Determine the desired test input values
• Execute the tests
• Evaluate the resulting test output values

Essentially, when you executed a task with input (e.g.: enter an email address into a text field and click submit), did your application generate the expected output (e.g.: user is subscribed and thank you page is displayed)?

We can understand this further with a quick example.

Functional Testing Example

Let’s begin with a straightforward application: a calculator. 

To create a set of functional tests, you would need to:

• Evaluate all the possible inputs – such as numbers and mathematical symbols – and design assertions to test their functionality
• Execute the tests (either automated or manually)
• Ensure that the desired outputs are generated – e.g.: each mathematical function works as intended, the final result is given correctly in all cases, the formula history is displayed accurately, etc.

For more on how to create a functional test, you can see a full guide on how to write an automated functional test for this example.

Functional Testing Techniques 

There are many functional testing techniques you might use to design a test suite for this:

• Boundary value tests evaluate what happens if inputs are received outside of specified limits – such as a user entering a number that was too large (if there is a specified limit) or attempting to enter non-numeric input
• Decision-based tests verify the results after a user decides to take an action, such as clearing the history
• User-based tests evaluate how components work together within an application – if the calculator’s history was stored in the cloud, this kind of test would verify that it did so successfully
• Ad-Hoc tests can be done at the end to try and discover bugs other methods did not uncover by seeking to break the application and check its response

Other common functional testing techniques include equivalence testing, alternate flow testing, positive testing and negative testing.

Automated Functional Testing vs Manual Functional Testing

Manual functional testing requires a developer or test engineer to design, create and execute every test by hand. It is flexible and can be powerful with the right team. However, as software grows in complexity and release windows get shorter, a purely manual testing strategy will face challenges keeping up a large degree of test coverage.

Automated functional testing automates many parts of the testing process, allowing tests to run continuously without human interaction – and with less chance for human error. Tests must still be designed and have their results evaluated by humans, but recent improvements in AI mean that with the right tool an increasing share of the load can be handled autonomously.

How to Use Automated Visual Testing for Functional Tests

One way to automate your functional tests is by using automated visual testing. Automated visual testing uses Visual AI to view software in the same way a human would, and can automatically highlight any unexpected differences with a high degree of accuracy.

Visual testing allows you to test for visual bugs, which are otherwise extremely challenging to uncover with traditional functional testing tools. For example, if an unrelated change caused a “submit” button to be shifted to the far right of a page and it could no longer be clicked by the user, but it was still technically on the page and using the correct identifier, it would pass a traditional functional test. Visual testing would catch this bug and ensure functionality is not broken by a visual regression.

How to Choose an Automated Testing Tool?

Here are a few key considerations to keep in mind when choosing an automated testing tool:

• Ease of Use: Is it something easy for your existing QA team to use, or easy to hire for? Does it require an extensive learning curve or can it be picked up quickly?
• Flexibility: Can it be used across different platforms? Can it easily integrate with your current testing environment, and does it allow you the freedom to change your environment in the future?
• Reusability/AI Assistance: How easy is it to reuse tests, particularly if the UI changes? Is there meaningful AI that can help you test more efficiently, particularly at the scale you need?
• Support: What level of customer support do you require, and how easily can you receive it from the provider of your tool?

Automated testing tools can be paid or open source. Some popular open source tools include Selenium for web testing and Appium for mobile testing.

Why Choose Automated Visual Testing with Applitools

Applitools has pioneered the best Visual AI in the industry, and it’s able to automatically detect visual and functional bugs just as a human would. Our Visual AI has been trained on billions of images with 99.9999% accuracy and includes advanced features to reduce test flakiness and save time, even across the most complicated test suites.

You can find out more about the power of Visual AI through our free report on the Impact of Visual AI on Test Automation. Check out the entire Applitools platform and sign up for your own free account today.

Happy Testing!

Keep Learning

Looking to learn more about Functional Testing? Check out the resources below to find out more.

• Applitools Eyes: Modern Functional Testing [Video, 1:40]

• Modern Functional Test Automation Through Visual AI [Free Course]
• Why Should Software Testers Understand Unit Testing? [Blog]
• The end of Smoke, Sanity and Regression [Blog]
• The Impact of Visual AI On Test Automation [webinar recap]
• Cutting-edge Functional UI Testing Techniques [webinar recap]
  

The post What is Functional Testing? Types and Example (Full Guide) appeared first on Automated Visual Testing | Applitools.

A hands-on guide for getting started with visual UI testing for mobile apps with Applitools and Bitrise easily.

Mobile apps play a critical role nowadays in our life, especially during the pandemic period, like e-commerce, groceries, medical, or banking apps. As we know there are different mobile apps, technologies, SDKs, and platforms that make mobile testing have a lot of challenges and which need special technical skills from the testing team to tackle. 

And because most of the mobile companies aim to release their apps on a weekly or bi-weekly release cadence based on the team decision to deploy new features to the customer before their competitors, they are using different techniques. One of them is test automation.

Based on that, in this article, I’ll walk you through the most important aspects of automating the Visual UI testing for mobile apps and running it via CI/CD pipelines using Applitools and Bitrise. 

What’s Visual Testing?

Visual Testing is a testing activity that helps us to verify that the GUI (Graphical User Interface) appears correctly and as expected to customers, by checking that each element on a mobile app appears in the right size and position on different devices and OS versions. 

Why should we automate Visual UI Testing?

Mobile companies nowadays are applying Mobile DevOps and implementing CI/CD (Continuous Integration and Continuous Delivery/Deployment) pipelines to deploy their mobile apps consistently and frequently. 

One of the vital parts of this process is test automation, to make sure that app functionalities are working as expected and test that there are no blockers or critical issues in the early stages (Fail Fast) by running these tests on every pull request or as a nightly build. And if we add Visual UI testing with this pipeline we will make sure that there are no issues in the GUI and all the features are displayed correctly. 

In this tutorial, I will add the Visual UI Testing to my Android UI tests with Appium and then run my tests on Bitrise as CI/CD pipeline, covering the following steps:

1. Implement and run test and locally with Applitools Eyes SDK
2. Push the code to the GitHub repository
3. Create Bitrise account and add the project
4. Add the Applitools Environment Variables 

Let’s get started! 

To get started with Appium for Android applications, you should prepare your local machine with the following requirements:

• macOS High Sierra or Mojave or Windows 10
• NodeJS and NPM
• Java +8
• Android Studio includes Android SDK +16 (Android Emulator and command-line tools)
• Appium Desktop
• Appium
• Appium Doctor
• Maven or Gradle for your automation project

More information about installing Appium can be found in the free Appium Course on Test Automation University or in this blog post.

If you don’t have an Android application, you can fork the Sunflower app from GitHub. Sunflower app is a gardening app illustrating Android development best practices with Android Jetpack.

With Appium, you can test automation scripts or projects that are stand-alone. I created a Maven project (you can create a Gradle one if you’d like) with the following structure: 

1. POM.xml: I added the project dependencies such as testNG, Appium, Eyes Appium Java and created a Maven profile to run the tests from the command line with the maven surefire plugin.

2. Android_testng.xml: I added my testcase to run it with the maven command line using the maven command (mvn clean test -Pandroid)

3. FirstAndroidTest.java: it’s the test class where I added my test case and Appium desired capabilities with the following scenario: 

• Open App
• Click on Plant List
• Click on My Garden
• Click on Add Plant button
• Click on “Avocado”
• Click Back
• Click on My Garden and assert that the plant name is exist

The full project can be found on GitHub.

And now we can run the test using the following command:

> mvn clean test -Pandroid 

And the results will be like the following image:

Now it’s time to add the Visual UI validation using Applitools Eyes SDK for Appium. 

Applitools Eyes is powered by Visual AI, the only AI-powered computer vision that replicates the human eyes and brain to quickly spot functional and visual regressions. 

If you don’t have an account you can create a free account from here.

1. In the POM.xml file add the following dependency

2. In the FirstAndroidTest class, initialize the eyes SDK with the following command

Eyes eyes = new Eyes();

3. In the setUp() void, you can set the private Applitools API key (you can find it in your account) and disable take the full screenshot with the following code: 

eyes.setApiKey("YOUR_API_KEY");

eyes.setForceFullPageScreenshot(false);

4. In the add_plant_test() void, start the test with the following line:

 eyes.open(driver, "Sunflower", "Add My Plant");

5. After we open the app, we will add the following line to check that the Plant List screen is displayed correctly: 

eyes.checkWindow("Plant list",false);

6. Then after we clicked on My Garden, we can add this line: 

eyes.checkWindow("My Empty Garden",false);

7. After we add the plant in the Garden “Avocado” we will add this line:

eyes.checkWindow("Avocado",false);

8. At the end of the test, we will add this line to finish the test:

eyes.close();

9. In the tearDown() void, we will add this line to abort the test if not closed:

eyes.abortIfNotClosed();

10. Now we can run our test again with the same Maven command and check the results in the Applitools Test manager with the steps view. 

Or the Batch Summary View:

11. Try to run the test again and check if the results are the same or if there is an issue. 

In my case when I run the tests again the test failed because the Eyes detected differences between the base images or the first run/batch and the 2nd run. 

And I checked the dashboard to check the results and the differences:

And I found that I have a difference in the first image which was already highlighted. You can click on the image for more details or compare it with the base image to check the difference. 

Because in my case sometimes the images loaded slowly, there was a difference between the two images. 

And also you can check the Batch Summary View for more details. 

TIP: If you need to run on different mobile devices and OS versions in parallel, you can use Applitools Ultrafast Test Cloud. More info on that can be found here.

Configuring your CI for the Eyes

After we successfully run our tests locally, it’s time to integrate our tests with our Android CI using Bitrise. 

But first, we need to do a small change in the application path, Applitools API Key, and the Batch ID. Locally, we’re using the absolute path of the app and hardcoded values, but with Bitrise or our CI server, it should be as an environment variables with the following steps: 

1. I will set up the values of the environment variables APPLITOOLS_API_KEY and APPLITOOLS_BATCH_ID to provide the API key and batch ID respectively in the setUp() void using the following code:

2. I will change the APK path to the Bitrise ENV Var because we will build the app in the CI server and then passing it to the Appium test using the following line 

         caps.setCapability("app", System.getenv("BITRISE_APK_PATH"));

3. Push the code to the GitHub Repository

Now it’s time to integrate our Android app and the automation project with Bitrise.

4. First, you need to create a new Bitrise account (it’s free). You can use your email or any other account name you prefer.

5. After logging in, you will have an empty dashboard where you can start adding your projects.

6. Click on the Add your first app button to add the Sunflower Android app first. 

7. Select your account and in the SET THE PRIVACY OF THE APP section select Private or Public, then click Next.

8. Give Bitrise access rights to your source code provider and choose the repository. 

9.  Now you need to set up the repository access: click on auto-add SSH key or add your SSH key.

10. Choose Branch name, for example, master/main, and click Next.

11.  After this, Bitrise will run a validation with your repository and configuration.

12. After validating, the project build configuration will display successfully (for example, Android App).

13. Click on the Confirm button after reviewing the Workflow stack. You can add an app icon as well.

14. The final step is the Webhook setup. We just need to click on Register a Webhook for me! button and you will trigger your first Android build for your project 

15. You will be redirected to the build log where you can check the Steps and the build status.

16.  In the end, you will find a summary of all the tasks you ran during the build and how much time they took, along with the Step names. You can download logs or check the apps and artifacts here.

17. Now let’s open the Workflow Editor to configure the CI Workflow with the new steps required for Appium and Applitools integration. 

You can rename the workflow as you like, for example from primary to appium-ui-tests:

18. Add a Script step to clone the Appium repository by clicking on the + button and it will open the steps screen, then you can search about the script then select it. 

19. As we know, we are using Appium Desktop as a server and inspector locally, but with CI servers we can’t use it, so we need to install and run Appium Server from the command line. Here I’ll add another Script step to install and run the Appium server in the background.

20. You will notice that I’m passing –log appium.log with the Appium command to export the server log as a .txt file for more debugging purposes if any tests failed. 

21. Add the AVD Manager Step to create an Android Emulator to use with your tests. You can choose any API level. For example, you can change it to 28 but you need to change it in the Appium desired capabilities and commit the change to your GitHub repository as well.

22. Add the Wait for the Android emulator Step and wait for the emulator to be ready before running our tests on it.

23. Add another Script Step to run the UI tests. You need to switch inside the automation project to be able to run the maven command.

24. Add the Script Step to copy the Appium log file to the Deploy directory of Bitrise to be able to find it in the artifacts tab.

25. Add a Step to Export the test results to the Bitrise Test Report Add-on and specify the test results search pattern like this: (*/target/surefire-reports/junitreports/*)

26. Add the Applitools API Key and the Batch ID as environment variables, by click on the Secrets menu option and click add button then add your APPLITOOLS_API_KEY and APPLITOOLS_BATCH_ID like the following image: 

The final Workflow will look like the following:

To run the Workflow, go to the application page and click on the Start/Schedule Build button and select the Workflow then click on the Start Build button.

To check the test results and the Appium log file you can click on the Test Report add-on, and for the appium log and the app.apk file, you can click on Apps & Artifacts tab.

Finally, we successfully added the Visual UI testing for our Android app and we can run the tests on every pull request or as a nightly build to make sure that we don’t have something breaking our codebase and to keep our master branch always green. 

I hope you found this article useful and thank you for reading it! 

Code Repositories

• Sunflowers Android app including the bitrise.yml file
• Appium UI tests repository

Future Reading

• Using Eyes with GitHub
• Configuring your CI for the Eyes GitHub integration

Useful Links

• Sign up for a free Applitools account
• Learn more about the Ultrafast Test Cloud
• Find out more about Bitrise

The post Getting Started with Visual UI Testing for Android Apps with Applitools and Bitrise appeared first on Automated Visual Testing | Applitools.

Overview

Applitools is used by the world’s top organizations to help expedite time to market with Visual AI-powered functional & visual testing. Applitools easily fits into existing toolchains, turbo-charging CI/CD pipelines with AI-infused cross browser testing reducing test maintenance and slashing build times, allowing developers to test more, earlier.

With GitHub boasting over 56 million users as of September of 2020, it’s no surprise that a deep integration with Microsoft’s popular source control solution is critical to our customers’ app-dev workflows and thus success.

This article provides an overview of all the ways in which Applitools integrates with GitHub & GitHub Actions.

Use cases covered in this article:

• What is UI Version Control? (read more)
• Integration with GitHub via GitHub app (read more)
• Integration with On-prem GitHub server (read more)
• [Legacy Integration] with GitHub cloud and on-prem via direct integration (read more)
• Integration with GitHub Actions (read more)

What is UI Version Control?

By enabling the Applitools / GitHub integration, you instantly unlock UI Version Control. With UI Version Control enabled, developers can easily correlate code changes with UI/UX updates for Visual UI version control across web and mobile apps. As branches are created in GitHub, Applitools UI Version Control automatically mirrors the branch and allows for a new set of baseline images. This process simplifies the automated testing of feature branches without impacting the trunk and its tests. Next, as PRs are merged back, UI Version Control automatically updates the target branch with the source branch’s baseline images while ensuring that no conflicts exist between branches, eliminating test maintenance overhead.

Integration with GitHub via GitHub app 

With Eyes 10.11, Applitools now officially supports visual source control via the GitHub app. This interface requires fewer permissions than the Applitools direct integration and is thus the officially recommended integration by the Applitools product team. For step by step instructions on enabling the GitHub integration via GitHub all, visit the integration documentation.

Integration with GitHub server which is installed on-prem

Another great addition to Eyes 10.11 is the ability for GitHub on-prem servers to leverage a new proxy service to eliminate the security challenges IT teams previously encountered with the legacy “direct connect” integration. Setup is quick and easy, simply enable the integration, get the GitHub app, and install the Applitools GitHub Proxy Service.

[Legacy Integration] with GitHub cloud and on-prem via direct integration 

Existing cloud & on-prem deployments who have our legacy direct integration already enabled can continue to enjoy backward compatibility. 

Integration with GitHub Actions

Microsoft describes GitHub Actions as an easy way to automate all your software workflows including with world-class CI/CD. Build, test, and deploy your code right from GitHub. Make code reviews, branch management, and issue triaging work the way you want.

The Applitools integration requires your Applitools API Key and for you to pass Git Commit SHA as Applitools Batch ID. Once it’s all configured, GitHub Actions workflow understands when changes are made to code, and it automatically kicks off the workflow complete with your Visual AI powered test suite.

For more information, check out the article “How Do I Link GitHub Actions with Applitools?” from Satish Mallela 

Additional Links on GitHub & Applitools

• Get a Jump into GitHub Actions
• How to Setup GitHub Actions with Cypress & Applitools for a Better Automated Testing Workflow
• Webinar – Intro to GitHub Actions for Test Automation — with Angie Jones and Brian Douglas
• How to Test Your GitHub Pull Requests so You Don’t Break Your UI

Additional Source Control Integrations to unlock UI Version Control

• GitLab
• BitBucket
• Learn more about rich Applitools ecosystem by visiting our SDKs and Integrations page

(heart icon in header from Kiranshastry at www.flaticon.com)

The post Everything You Need to Know About Integrating Applitools with GitHub appeared first on Automated Visual Testing | Applitools.

Applitools provides an end-to-end software testing platform powered by Visual AI. It can be used by people in engineering, test automation, manual QA, DevOps, and Digital Transformation teams.

What does Applitools do? Find bugs that functional tests miss, like the one above.

That’s an actual visual bug from Instagram’s mobile app in March 2019. It’s a sponsored ad, meaning that one of Facebook’s customers paid good money for this placement.

How did this bug make it into their app?

Our bet is that Instagram has functional test automation for this screen. But guess what? All the expected text is on the screen. A traditional functional test wouldn’t find any issues.

Applitools finds visual bugs in your apps. It makes sure that no visual elements are overlapping, invisible, or off page. It makes sure that no new unexpected elements have appeared. All of which are things that you can’t automate with traditional functional tests.

Why use Applitools?

Let’s be clear: visual bugs aren’t some trivial issue for your business.

If you work with software — and frankly, which company doesn’t these days — you know that the rate of change is continuing to accelerate. Releases used to be once a quarter, now they’re once a day. Your customers expect constant updates to all the web and mobile apps they use — and yours are no different. Your competitors (the good ones at least) are constantly releasing upgrades.

This relentless pace of change strains QA teams worldwide, and I bet yours is no different. It’s never fun to sit in a release meeting and hear that a release can go out, “once QA has finished their testing” — and then have all eyes on the room look at you for an answer.

https://unsplash.com/@antenna

Problem is, integration and functional testing can only go so far. You need visual testing because your prospects and customers access your apps on a broad range of platforms and screen sizes. You know that “sloppy sites sink sales”, so your QA team still needs to manually check the visual layout of apps — which takes way too long and still misses crucial bugs.

It’s not your fault. Even the best digital companies in the world let visual bugs slip into production. Google. Amazon. And plenty more.

What you need is a test automation tool that lets you visually validate all your apps, on all the browsers and devices your customers use — super fast, super accurately.

So how does Applitools work?

At Applitools, we created the world’s first and only visual A.I. engine for automated visual UI testing and monitoring. Our product, Applitools Eyes, scans your apps’ screens and analyze them like the human eye and brain would, but with the power of a machine. That means Applitools Eyes will catch all the differences that are material — but also identify and ignore those which are not.

For example those simply due to rendering differences with different browsers and versions, operating systems, or screen sizes. With Applitools. We can say with confidence we are 99.9995% accurate, while other tools which use pixel by pixel, or dumb comparison, will show as much as an 80% false positive rate – far more harm than actual help.

So how can Applitools help you solve this problem?

A demo’s worth a thousand words, so check out this video to see how.

It’s just four minutes, but it could save you literally days.

Then setup a live demo with us, or if you’re the do-it-yourself type, sign up for a free Applitools account and follow one of our tutorials.

The post What is Applitools? appeared first on Automated Visual Testing | Applitools.

We are thrilled to share that Applitools has been named to CB Insights 2018 AI 100.

This recognition is part of a ranking of the 100 most promising private artificial intelligence (AI) companies in the world, and recognizes the teams and technologies that are successfully using AI to solve big challenges. We are honored to be recognized alongside many other organizations who are innovating in the realm of AI, machine and deep learning.

On top of making the general list, Applitools Eyes is the stand-alone finalist in the software development category. With other award categories in retail, media, cyber-security, agriculture, and more, it is clear that the ROI of AI and machine learning is real and organizations in every industry are finding innovative ways to leverage its benefits. 

Want to give Applitools Eyes a try? Enjoy a free trial of our easy-to-use SaaS solution

To calculate and determine the winners that made the AI 100 list, CB Insights uses a Mosaic scoring system, which is a quantitative framework to measure the overall health and growth potential of private companies. The Mosaic score evaluates companies based on three distinct factors: Momentum, Market and Money. After review of Applitools’ Mosaic score, we were selected as one of 1,000+ applicants to have made the list!

Why Applitools?
In today’s increasingly digital world, it is imperative that any company have a digital presence, let alone one that is visually flawless. With web teams buried in lines of code, it can often be challenging for organizations to find visual discrepancies on their apps and sites, and it often involves laborious manual processes to find/fix these discrepancies.

Thanks to Applitools Eyes, our AI-powered visual testing and monitoring solution, customers are able to automatically test the look, feel and user experience of their apps and sites without having to sacrifice their existing test framework.

All the while, Applitools Eyes’ AI-powered cognitive vision is constantly learning and improving from what are now millions of collected data points. This saves organizations countless hours and dollars spent using teams of manual testers tracking down visual imperfections and searching for flawed lines of code.

This journey has only just begun!
The increasing pressure from business executives to protect the brand experience, as digital transformation continues unabated, means we will see the emergence of the nascent Application Visual Management (AVM) category. This logical extension of Application Delivery Management (ADM), and Application Performance Management (APM), will become more a focus for enterprises as we head into 2018.

Make sure to stay tuned in the coming year for some more exciting announcements, updates to our solutions and the latest trends in test automation. And, thanks again to the CB Insights team for the accolade!

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Applitools Named to CB Insights’ 2018 AI 100 List appeared first on Automated Visual Testing | Applitools.