Applitools finds functional & visual bugs for the world’s largest organizations; Rally® is leveraged by the largest enterprises to manage their agile development at scale, including quality management and the tracking of bugs. We collaborated to design a new integration to make this process faster and provide developers with the in-depth details they need to reproduce and triage bugs faster than ever before. 

Rally & Applitools

Rally Software is an enterprise-class platform that is purpose built for scaling agile development practices. Rally serves as a central hub for teams to collaboratively plan, prioritize and track work, from strategy all the way down to stories and bugs through the entire SDLC.

Applitools is building the next generation of test automation platform for cross browser and device testing powered by AI assisted computer vision technology known as Visual AI. Visual AI helps Developers, Test Automation Engineers and QA professionals release high-quality web and mobile apps enabling CI/CD.

Better Together – A Transformative Workflow

Applitools & Rally share enterprise customers. Like all modern businesses, our shared customers are looking for ways to gain an edge over their competition – and oftentimes that edge is gained by improved workflows and automation that saves time and allows teams to release better quality software faster. It’s a never ending battle to improve time-to-market, and for our shared customers, we have just added an integration that can do just this.

The integration allows users to log a bug in Rally without ever leaving the Applitools Eyes user interface and workflow. This means no context shifting, no copy / paste, no additional logins or lost browser tabs. It’s designed for both workflow efficiency and to ensure that every detail gets logged so developers can have everything they need to triage the bug and move on. It’s not only developers and testers who benefit from this integration, now any stakeholder in the entire appdev process from product managers to UX designers can automatically pinpoint the exact cause of issues using the Applitools Root Cause Analysis capabilities, send the results instantly to Rally and have everything needed to reproduce and fix defects on the spot.  

Setting up the Rally Integration

Your Eyes admin can setup the Rally integration via the Admin screen →  Integrations tab. The process takes less than a minute. First you’ll enter their Rally server & API key – then a run through a quick authentication process requiring a Client ID & Client Secret.

Once authenticated, all Rally projects will be available to choose from, so simply choose the first project you will link to Applitools along with the default work item type for issues created from within Eyes (most likely it will be ‘defects’). You can optionally add one or more defined fields by simply choosing the field name and and the default value – repeat as needed. For detailed setup instructions, please visit our Rally integration documentation.

Automating the Defect Feedback Loop

The beauty of this integration is in its simplicity. When a bug is found by Applitools, Users can gather all relevant information including screenshot and full steps to reproduce with a single region-drag and click using the bug region feature. The detailed information is instantly sent to Rally for teams to immediately begin triage.

Applitools Integration Ecosystem

The Rally integration is a continuation of our commitment to extending the Applitools integration ecosystem and fitting seamlessly into customers’ existing workflows and tools. Applitools customers now benefit from seamless integrations with over 60 SDKs for you to choose from. This includes:

• Testing frameworks and languages such as Selenium, Cypress, Appium, and more
• Source control solutions like GitHub, GitLab and BitBucket
• CI\CD platforms like Jenkins, TeamCity and Travis
• Collaboration tools like Slack
• Agile planning and defect tracking tools like Jira

Now, with the Eyes 10.11 release, we’re excited to also include Rally on this list.

The post Transforming Software Development through a Modern Feedback Loop appeared first on Automated Visual Testing | Applitools.

Much is made of the dream flow from continuous integration to continuous testing to continuous delivery to production. Conceptually this sounds great! A developer checks in their code, the code is built, unit tests are executed, the code is deployed to a testing environment, and then the automated test scripts are executed. If nothing fails in the preceding steps, the code is then automatically deployed to the production environment. The users/clients/customers potentially get multiple releases per day, but also potentially get multiple features or fixes per day. We get all of this with no human delay or interaction once the code is checked in. Also, we get all of this with no human oversight and little human insight.

What could possibly go wrong? What, indeed!

Capturing Visual Issues

Let’s assume we have a production system with which humans will interact. One thing that could go wrong is the user interface could be all “messed up”. Perhaps all the text and the background are now royal blue. Blue on blue is difficult to read, to say the least; our users would not be able to use the site. It could be worse, though: what if our #1 competitor’s well-known logo and product color is royal blue. This would be awkward at best. At worst? Perhaps we are infringing on a competitor’s trademark and are now subject to legal action.

Oh, wait, automated visual testing would have caught that. Correct! Automated visual testing would have caught that plus a slew of other visual and formatting issues. But, like most testing and automation techniques, visual testing can’t catch everything a human would because automated visual testing isn’t, well, human…

Capturing Timing Issues

What if a transaction takes longer than it should or than it did last time? Oh, well, we can check for that in our scripts. Cool, cool. Do we have test scripts that pause mid-flow for 30 minutes because one of our kids threw a handful of Legos at the other and it was the end of the world as we know it? Uh…oh…no…, we can add a script for that. Can we add scripts to pause at every possible place in the flow? Probably not; we probably can’t even think of every possible place in the flow. Spoiler alert: our users do these things, so do their kids, pets, and roommates.

Can we think of everything? Likely not. We’re human.

Cost of Change and Cost of Failure

Now, there’s nothing inherently wrong with the process I outlined above. As always, it’s about being responsible. If developer-check-in-to-prod works for your organization, then you should do it…provided you understand the risk. This risk manifests in two big buckets: cost of change and cost of failure.

The cost of change is something that’s discussed a lot. What does it take to debug an issue, fix it, then run the fix through the pipeline to get it to production, and therefore, to the users? That’s a very basic cost of change calculation: the cost of the effort required to determine a fix and get that fix out of the door to the user. Oh wait, while we’re addressing this issue, we’re not addressing other issues and we’re not working on new capabilities. These are costs as well; they’re referred to as opportunity costs, i.e. the cost of not addressing Thing B because we are working on Thing A. The math on the cost of change starts to become more involved because we’re now accounting for the impact of addressing an issue.

Let’s take the story further; consider a retail company. Most retail companies that have a brick-and-mortar storefront also have an eCommerce capability via a website and a mobile app. The cost of change on the website is generally low. When a customer encounters an issue, the retailer may lose that customer or perhaps it will give a discount to that customer to make up for the poor experience, but the retailer can generally minimize the number of lost customers by quickly deploying a fix precisely because the cost of change is low.

Contrast that cost with the cost of change for this retailer’s point of sale (POS) system. Changes to a POS system have downstream impacts that aren’t immediately obvious. Consider a retailer that has a large number of stores but that has a very small per-store number of employees, say, ten employees per store. Each employee must be trained on new or changed POS features; the number of training and ramp-up hours per change may not be insignificant.

Now suppose we found an issue with the POS release and the resolution for this issue requires a change in the workflow. Though the training cost will likely not be as large as the original cost, there is still a non-zero training cost to deploying the new software for use. Of course, when the number of employees that use the POS system is larger, the training and retraining costs are larger as well. As we can see, the actual cost of change can be higher than it appears on the surface.

Imagine further that this POS software has an outage…on Black Friday. There will possibly be 10’s or even 100’s of people in a store, all of whom are all inconvenienced by the outage, and many of whom will be expressing their frustration on social media. Here, we see that there is a cost involved that’s in addition to the cost of change; issues, problems, and failures have a cost as well.

Framing Costs of Failure Issues

The cost of these issues, problems, and failures is something that seems to be discussed less frequently. What is the cost of failure? It’s the cost to the business of an issue, i.e. failure, in the system. In some cases, it’s probably rather small. If we’re producing a 99-cent mobile game and a user encounters an issue, we may lose that user but, in the grand scheme of things, we may not really care because we didn’t lose appreciable money. It’s ugly to say, but when discussing business, it’s generally accurate. We may lose that customer and perhaps some of their friends, but we’re probably not hinging our house payment on that one game sale.

Contrast the 99-cent game example with a package delivery company. This company will likely have a website for its customers to arrange pickups and deliveries. The cost of change is probably relatively low if there’s an issue on the website; the cost of failure, however, could be catastrophic. What if legal documents for a corporate sale were delayed, or worse, sent to a competitor by mistake? What if an undiscovered error in our system causes the company to fail to deliver life-critical medical supplies, or, say, a donor organ? Catastrophic, to be sure. The cost of this failure is unacceptably high.

Even in the cases where there is no loss of life, there can be tangible financial costs to failure as well. Companies often have contracts with their clients that contain service level agreements (SLAs). Violation of the SLAs can require penalty payments or a return of previously paid fees when our systems are “insufficiently performant”.

Conclusion – Assess Your Costs and Risks

As with most concepts in technology, there’s nothing inherently wrong with “check-in and deploy to production”. Companies are doing this and seem to be content-to-happy with the business results. That said, in all circumstances, we must be responsible with our automated approaches. We must understand the risks we are undertaking, and the risk tolerance we have. Without these self-assessments, we won’t be prepared for the consequences for the risks we unknowingly undertake.

Like this? Catch me at an upcoming event!

The post Production Deploy with Every Check-In? You Gotta Go TWO Low! appeared first on Automated Visual Testing | Applitools.

“Watch this deep-dive session, where I demonstrate multiple methods you can use for collecting production data, as well as give real-life examples of how production data can help you find bugs, and even enable running tests in production.” — Amber Race

Our software doesn’t run in a test environment with mock users — it runs out in the real world, being used by thousands or even millions of real people, with thousands of variations of devices, screen sizes, and operating systems.

Learn why you need to be collecting this data — and how you can make it actionable for continuous testing, learning and improving, as well as how it can help you make your own services more transparent and user-friendly to your customers.

Watch this webinar by expert Amber Race — Sr. SDET @ Big Fish Games — and learn:

1. What monitoring and logging tools are available
2. How to decide what to track
3. Ways to create meaningful reports with monitoring results
4. How to use production data to create more realistic tests
5. Strategies for using real-time monitoring to run tests in production

Amber’s Slide Deck:

Full webinar recording: 

Additional reading and recommended resources:

1. 16 reasons why to use Selenium IDE in 2019 (and 2 why not) – post by Al Sargent 
2. Which programming language is most popular for UI test automation in 2019? – post by Angie Jones 
3. The New Selenium IDE: See How It Can Turbo-charge Your Testing Efforts – webinar with Dave Haeffner and Tomer Steinfeld
4. Collaborating with Developers: How-to Guide for QA Pros & Test Engineers – webinar by Gil Tayar
5. Confident React – Kent C. Dodds Speaks about Frontend Testing – webinar with Kent C. Dodds
6. Release Apps with Flawless UI: Open your Free Applitools Account, and start visual testing today.
7. Test Automation U — the most-talked-about test automation initiative of the year: online education platform led by Angie Jones, offering free test automation courses by industry leaders. Sign up and start showing off your test automation certificates and badges!

— HAPPY TESTING — 

The post The Joy of Testing in Production – with expert Amber Race appeared first on Automated Visual Testing | Applitools.

This Friday, we’re joining Agile, Lean, and Kanban experts to share best practices in software development at the Virginia Crossings Hotel & Conference Center for Innovate Virginia 2019.

What is Innovate Virginia? Innovate Virginia (@InnovateVA, #InnovateVA) is an annual event hosted by Agile Richmond, bringing together dozens of industry thought-leaders and more than 400 Agile practitioners for a day of networking, interactive sessions, presentations, and fun! This year’s conference is taking place on March 22.

This unique one-day event is a time to share the use of innovative methods of software delivery among an impressive lineup of speakers. Attendees will hear from industry leading representatives from companies including Mavenlink, Frontier Project, Leading Agile, CarMax and many more. We’re certain that we will learn a lot from this event and hope to take home some refreshing new insights and excitement about the latest innovations in software delivery.

As Gold Sponsors, we’re proud to support Innovation Virginia 2019. You can find us at the expo hall during exhibit hours. We’re looking forward to sharing the latest from our team at Applitools where we have newly released Applitools Root Cause Analysis, helping front-end developers and QA professionals pinpoint the cause of bugs in application code within minutes, eliminating hours from traditional bug diagnosis practices and enabling companies to keep software projects and digital transformation initiatives on track.

We are also excited to share with the community our new AI-powered visual testing integration with Selenium IDE! The integration enables QA and development teams to check responsive software applications using Visual AI across dozens of browsers and devices.

Additionally, Applitools Eyes offers UI baseline management, team collaboration, a wide range of integrations with testing frameworks, and extensive analytics dashboards — eliminating the stress of manual visual testing through its AI features. This helps ensure a visually perfect user experience on every single browser and device.

We hope to see you there – come to meet the team, and get your “Visually Perfect” swag!

If you don’t get a chance to connect with us at Innovate Virginia, reach out or sign up for a free Applitools account.

The post Innovate Virginia 2019 – See You There! appeared first on Automated Visual Testing | Applitools.

Later this week we will be joining global DevOps leaders at the DevOps Talks Conference (DOTC) in Melbourne, Australia!

What is the DevOps Talks Conference? The DevOps Talks Conference (@DevOpsTalks, #dotc19) brings together DevOps leaders, engineers, and architects who are implementing DevOps principles and practices in startups and leading enterprise companies from around the globe. The conference boasts speakers from companies such as: Google, Microsoft, JFrog and more! This year the event is being held at the Melbourne Convention and Exhibition Centre (MCEC) March 21-22!

In today’s competitive IT industry, companies have to deliver products and services to their customers faster than their competition. Agile delivery practices are helping companies collaborate and deliver faster. The maturity of DevOps practices is an essential ingredient to speed up time-to-production for each feature, idea and/or market test. DevOps is the engine of Continuous Integration and Continuous Deployment, but it requires significant changes in culture and personal attitude.

That is why we are so excited to be a Platinum Sponsor of the DevOps Talks Conference 2019 in Melbourne. Where we will not only be sharing the latest visual testing and monitoring techniques that support automation and improved visual UI testing through the Application Visual Management (AVM) approach but contributing to the DevOps world at large and working together to create the changes needed make these practices the new standard.

Applitools Co-founder and COO, Moshe Milman, will be presenting “Fixing Your Automation Challenges in the Era of CI/CD” on March 22 @ 12:05 pm (local time). In his talk, Moshe will cover how Continuous Delivery is now the holy grail of IT organizations, but why many companies are still struggling with the transition into shorter release cycles and faster, more frequent deployments. A key challenge that companies are facing in that transition lies around test automation, and in this talk, he covers these challenges in detail and demonstrates how successful companies are addressing it.

We’re looking forward to sharing the latest from our team at Applitools were we have newly released Applitools Root Cause Analysis, helping front-end developers and QA professionals pinpoint the cause of bugs in application code within minutes, eliminating hours from traditional bug diagnosis practices and enabling companies to keep software projects and digital transformation initiatives on track.

We are also excited to share with the community our new AI-powered visual testing integration with Selenium IDE! The integration enables QA and development teams to check responsive software applications using Visual AI across dozens of browsers and devices.

Additionally, Applitools Eyes offers UI baseline management, team collaboration, a wide range of integrations with testing frameworks, and extensive analytics dashboards — eliminating the stress of manual visual testing through its AI features. This helps ensure a visually perfect user experience on every single browser and device.

We hope to see you there — stop by our table in the expo hall to meet the team, and grab your “Visually Perfect” swag.

If you don’t get a chance to connect with us at DevOps Talks Conference 2019 in Melbourne, reach out or sign up for a free Applitools account.

The post DevOps Talks Conference (DOTC): Melbourne 2019 – See You There! appeared first on Automated Visual Testing | Applitools.

Next week, we’re joining agile and test automation leaders at the Dorint Sanssouci Berlin/Potsdam in Potsdam, Germany at Agile Testing Days 2018!

What is Agile Testing Days? Agile Testing Days (@AgileTD, #AgileTD), now in its 10th year, is an annual conference that brings together global agile testing professionals to discuss the latest trends in testing including DevOps, security, AI, culture and more. This year’s conference is taking place in Potsdam, Germany November 11-16.

We are looking forward to taking in a range of opportunities to learn – from breakout sessions and keynotes to workshops and networking events, we can’t wait to join this community for an immersive week of sharing and collaborating.

We are also scheduled to deliver three talks.

The first will be delivered by Applitools’ Senior Developer Advocate and test automation thought leader, Angie Jones. Angie presents “The Reality of Testing in an Artificial World” on Tuesday, November 13 @ 10:25 a.m. CEST. One of the strongest assets of being a tester is being able to discover and report on the unknown, so, if you want to learn the skills needed test in today’s increasingly artificial world and how to to make an impact, don’t miss out on this session.

Next is Gil Tayar, Senior Architect, who presents “Not Only Cars! AI, Please Test My App” on Thursday, Nov 15 @ 11:10 am CEST. If you want to learn more about the role of Artificial Intelligence in Testing and Test Automation, where AI can take us in the near future, as well as the 6 levels of AI in automated testing — this talk is for you!

Gil will also present our all-time most favorite talk: “Intro to Visual UI Testing” on Wednesday, November 14 @ 11:55 a.m. CEST. During this session, Gil demonstrates how you can easily avoid front-end bugs and visual regressions, as well as substantially increase coverage, by adding scalable automated visual testing to your existing automated. He will end the talk with a live visual test run by Applitools Eyes!

Agile Testing Days 2018 is an outstanding conference, and we are honored to sponsor and support it. You can find us at our booth during exhibit hours. We’re excited to share the latest visual testing and monitoring techniques that support automation and improved visual UI testing through the Application Visual Management (AVM) approach.

We’re also looking forward to sharing the latest from Applitools Eyes, our Automated Visual AI Testing Platform. Learn about the recent updates to our AI powered visual UI testing platform, Eyes. The latest features include Cypress and Storybook SDKs that are engineered to run visual tests instantly as developers write code, enabling them to identify visual bugs as soon as they are introduced. The result is more productive engineering teams, delivering higher-quality apps faster than ever before.

Additionally, Applitools Eyes offers UI baseline management, team collaboration, a wide range of integrations with testing frameworks like Selenium, and extensive analytics dashboards — eliminating the stress of manual visual testing through its AI features. This helps ensure a visually perfect user experience on every single browser and device.

We hope to see you there!

If you don’t get a chance to connect with us at Agile Testing Days 2018, reach out or sign up for a free Applitools account.

The post Agile Testing Days 2018 Germany – See You There! appeared first on Automated Visual Testing | Applitools.

Next week, we’re joining test automation and agile software development leaders at the 2018 Agile and Automation Days Poland located at Park Inn by Radisson Krakow in Krakow, Poland. 

What is Agile and Automation Days Poland? Agile and Automation Days Poland (@AADaysPoland, #aadays2018), now in its third year, is a two-day software testing conference featuring interactive sessions and workshops geared towards advancing the test automation and agile practices in Poland and beyond. This year’s conference is taking place in Krakow, Poland from October 15-16.

We will be giving our own presentation alongside representatives from Facebook, Pivotal, ThoughtWorks, Royal Bank of Scotland, and many more.

Amit Zur, a Sr. Software Developer at Applitools — and the man behind our new Cypress and Storybook SDKs — is scheduled to deliver his presentation on October 15 at 11:55 a.m CEST. He will present “Async/Await: The Promised Land” where he discusses quirks, best practices and use cases for the new syntax Async/Await. By the end of this talk, you will be able to rush back to your computer and start refactoring all your code. And to whomever already uses async/await we will discuss the quirks and tips of the syntax, as well as use cases for incorporating it in tests.

If you don’t get a chance to catch up with Amit during the show, we encourage you to check out the latest updates to Applitools Eyes, our Automated Visual AI Testing Platform. Some of the latest features include UI Version Control that lets you build software smarter by capturing the entire visual history of your apps so you have a record of what’s been changed, by whom, and when — and what should be rolled back if necessary.

Additionally, Applitools Eyes offers UI baseline management, team collaboration, a wide range of integrations with testing frameworks like Selenium, and extensive analytics dashboards — eliminating the stress of manual visual testing through its AI features. This helps ensure a visually perfect user experience on every single browser and device. You can find us posted up in the foyer giving away some visually perfect swag – hope to meet you in Krakow!

If you don’t get a chance to connect with us at Agile and Automation Days Poland, reach out or sign up for a free Applitools account to start creating your own visual UI tests.

The post Agile and Automation Days Poland – See You There! appeared first on Automated Visual Testing | Applitools.

In a previous blog post of mine “Embracing Agility: A Practical Approach for Software Testers”, I discussed how an agile team changes the role of the software tester. Instead of being a gatekeeper that tests the product after it is developed, the software tester is part of the development process.

I also described how the shift-left movement enables the software tester to concentrate on the more interesting tests, and leave the boring tests (“this form field can only accept email addresses”) to the software developer. But those interesting tests, at least the ones we decided to automate, need to be written, and written quickly.

In this blog post I will discuss a technique whereby you can use visual UI testing to make writing those tests much easier. 

Visual UI Testing vs Functional Testing

Visual tests are usually used to check the visual aspects of your application. Makes sense, no? To automate tests that verify that your application looks OK, you write visual tests, and use visual testing tools such as Applitools Eyes. But visual testing tools can also be used for functional tests.

Really? How can visual testing help us write functional tests, tests that check how the application behaves? First, let’s understand how visual testing and functional testing of applications work.

Functional Testing

A functional test (in this blog post! I am sure that you can find other people that define it differently) is a test that starts an application in a certain state, performs a set of actions, and tests that the application is in the correct state after those actions.

So a functional test looks like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Assert that the actual state of the application is the expected one

Yes, you might argue that it is not the test that starts the application, and that the test should always create the specific set of data using other actions. You could, but that’s bikeshedding. True, no functional tests look exactly like this, but the idea is correct:

• action, action, action
• assertion, assertion, assertion
• action, action, action
• assertion, assertion, assertion

Let’s take an example of a to-do list application:

One functional test would be:

• Action: check a todo
• Assertion: it is checked in the “All” tab
• Assertion: it is gone from the “Active” tab
• Assertion: it is checked in the “Completed” tab

Implementing functional assertions

How would one go about implementing the assertions? If it’s a web application, and we’re using a browser automation library like Selenium Webdriver, then the tester would need to check whether the specific todo’s checkbox is checked, and also check that the other checkboxes are not checked. Also, check that the text of the todo is striked out, while the others are not striked out.

Oh, and don’t forget to check the “2 items left” text in the bottom left of the list.

And each such check is a pain in the butt — the tester needs to determine what the id of the control they are checking is. And sometimes there is no specific id (or class), and so the tester needs to go to the developer and ask them to add an id for testing, or if the tester is in an agile team, maybe add it themselves!

Also: this needs to be done across hundreds of tests, and thousands of assertions. This is practically impossible, and definitely impossible to maintain. What usually happens is that the developer chooses a subset of assertions, and tests just those. In our example, it would only assert that the checkbox of the todo item is checked, and forego the other tests.

Visual Testing

We’ve seen what functional tests are, and one of the problems associated with them. Let’s move on to visual tests: a functional test (in this blog post! I am sure that you can find other people that define it differently) is a test that starts an application in a certain state, performs a set of actions, and tests that the application looks good after these actions.

See how similar they are? A visual test looks like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Assert that the application lookscorrect

This is exactly like functional tests, except that the assertions are different. So let’s look at how assertions are performed in visual tests. How does a visual test go about testing that an application looks correct?

Implementing visual assertions

How do visual assertions work? Well, first of all, we take a screenshot of the screen — we are doing a visual assertion after all. But this screenshot, alone, is not enough. What we do is compare this to a baseline screenshot that we took in a previous run of the test. If the algorithm comparing the current screenshot to the baseline one tells us that they’re the same, then the assertion passes, otherwise it fails.

And how do we generate the screenshot? In a two phase process:

1. We define the screenshots that are created by the assertion of the first test the baseline screenshots. The tester manually checks each one of those baseline screenshots.
2. Each time the test failsit could be because there is a bug in the software, but it could also be because the software changed and this is the new The tester in this case manually checks each one of those new screenshots and declares them as the new baseline.

How does visual comparison work?

Unfortunately, visually comparing two screenshots seems easy, but it’s not. Theoretically, it should be as easy as comparing pixels, but this, unfortunately, is not the case. Different operating systems, browsers, even different versions of the same browser generate slightly different images. Moreover what if the screenshot includes the current time? You would need to check everything except that region of the screen.

There are even more concerns, which we should outline in a future blog post. Fortunately, tools like Applitools Eyes hide us from these concerns.

Using Visual Assertions in Functional Tests

Now that we understand how functional testing works, we understand that there’s no difference between functional and visual tests in how they are structured — the only difference is in what they test and in how they implement the assertions:

• Functional assertions check each and every control on the page to ensure that they have the correct value
• Visual assertions compares a screenshot of the page against a manually verified baseline screenshot.

Visual assertions seem pretty easy to use. Just manually verify the first screenshot, and just take screenshots of the pages after the actions in the test. No need to check each control, determine the id of the control, or ask the developers to add ids to controls. Just take a screenshot and let visual testing frameworks like WebDriverCSS or Applitools Eyes compare them.

And we’ve seen how functional assertions are hard. But what if we can use visual assertions in functional tests? Something like this:

1. Load a specific set of application data
2. Start the application
3. Execute a set of actions on the UI (or the REST API!) of the application
4. Take a screenshot to verify that the data is correct

Think of the todos page after checking the todo item:

With one screenshot, we can check that the check was checked (ouch, too many “check”-s), verify that the other todo items weren’t, that the “2 items left” is correct, that the todo item is in strike-out, and probably other stuff which we didn’t even notice.

Many front-end developers use this kind of “testing against a baseline”, but what they test is the HTML of the components, and not how they look like. But the idea is similar. The difference is that it is much nicer to check the difference from a baseline between two images, than to compare a textual diff of HTML.

Pleasant side benefit of using visual assertions: you get to check the visual look of your application for free!

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Visual UI Testing as an Aid to Functional Testing appeared first on Automated Visual Testing | Applitools.

The Agile Manifesto is 16 years old. In the home page, it defines a very simple set of values:

• Individuals and interactions over processes and tools
• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan

These four simple values have sparked a revolution in how we write and deliver software. I like to call them the “cut the bulls**t” values. To understand how these values affect testers, let’s dispense with the ceremony, let’s dispense with lengthy design documents, and, yes, lengthy test plan documents, and focus on understanding the core — writing the software.

And the way to do that is to deliver working software from the start (this is the “Working software over documentation” part of the manifesto). The agile manifesto asserts that the product needs to work from the beginning, and not wait for a lengthy and cumbersome “integration phase” when all the work on the product is integrated together. Instead of many team silos, where everybody is working on one piece of the puzzle (and the architect is the only one aware of how everything will be pieced together), we start with a small team that builds an initial working software (MVP, or minimal viable product) and gradually grows it to a bigger product with a bigger team.

This team includes everybody — the customer, product manager, developers, and testers (this are the “Individuals and interactions” and “customer collaboration” values of the agile manifesto). This integrated team delivers a minimal version of the product in a very short time, and… waits. Waits for what? For the customer to start working on the product and figure out the next step for the product. In other words, agile teams don’t build a specification for the whole product, but rather start small and incrementally determine how the product should look like (this is the “Responding to change over following a plan” part).

Yeah, yeah, yeah. Bla, bla, bla. I got that. But how does it affect my work as a software tester? It does. Immensely. I have one sentence for you:

Embrace agility: shift left and shift right.

But, unfortunately, to explain this, and to see how this affects your work as a software tester, I need a bit more explanations, so hang in there.

Testing in the Days of “Release Early, Release Often”

The agile manifesto also says: “Deliver working software frequently”. And today’s software development community has taken it to the extreme. Some companies release every week, some every day, some don’t even have the concept of “release”. They just release anytime a feature is ready.

Moreover, agility forced the backend to come up with an alternative to the “monolithic backend”. The concept of “microservices” has replaced the “monolith”. This concept enables large companies to break up their backend into many small services, each one responsible for a small piece of the puzzle, and to deploy each microservice independent of each other.

These two things means that in many companies there is no “release”. It’s a fluid set of changes that just happen to the product in ongoing process.

But if there is no release, how will software testers test? I have one sentence for you:

Embrace agility: shift left and shift right.

Shifting Left

What does shifting left mean?

The software development software, in the days before the process became agile, looked like this:

Software testers used to be at the right of this process. They used to be the gatekeepers that bar the gate to releases that are not qualified for deployment.

This process actually hasn’t changed much in the days of agile, except that now we have many of these pipelines in parallel, and each takes days, and not months:

But how can a software tester be a gatekeeper in this model? It’s practically impossible. Once the implementation is done, there is no “time” for testing, as the team has gone on to the next thing it needs to do. In the agile model, delivering software incrementally is paramount. Time for testing is not included.

To solve this, the tester needs to shift the testing left. It needs to be part of the implementation, and not a separate step:

This has many practical consequences, consequences that affect you as a software tester:

Software developers need to test too

Shifting left means that a lot of the burden of testing needs to be dealt with by the software developer and not by the tester. A tester should not need to check that a certain form field needs to be numeric. A tester should not need to check that the value in this table is correct. All those boring tests that occupy a lot of the tester’s time need to sit “close to the code”, and should not be dealt with by the tester. The software developer can use unit testing to check those boring checks in much faster and more reliable manner than the tester, and free up the tester’s time for the more interesting and holistic tests, whether manual or automated.

Automated tester tests need to sit close to the code too

As discussed above, agile software is now deployed piecemeal-wise, and not in one big deployment bang. This means that the software testers tests need to test not just the whole big product, but also parts of it.

This means that if the microservice your team is writing is a web service that communicates via REST API, then your tests should check that REST API in isolation from the rest of the product. And if your team is building UI components that the other parts of the company uses, those components should be tested in isolation from the rest of the product.

In other words, software testers should not only do end to end tests, but should also check parts of the software, and not the whole thing. Which parts? A good rule of thumb is that if your team is building components and microservices that are part of a bigger whole, you should test those components in isolation and not only as parts of the bigger whole.

Automated tester tests need to be part of the CI process

Software developers that operate as part of a true agile process frequently use CI tools to help them cope with the fantastic pace of software development. For software developers, the CI tool isthe gatekeeper for their software change.

What is a CI tool? CI means Continuous Integration, and a CI Tool is a tool that listens to changes in the source code, builds it, runs the tests, and if the tests pass, mark the changes “green”. If they fail, they usually email the person that submitted the software changes. The CI tool helps the software developer ensure that all the tests they write are executed for those specific changes. Examples of CI Tools are Jenkins, TeamCity, Travis, Bamboo.

These are fantastic tool, and are definitely part of the shift-left movement, as the tests runs for every code change. A software developer should ensure that not only do the software developer tests run, but also their own tests. A code change needs to be green only if the developer tests pass and the tester tests pass.

The three rules above are, for me, the essence of the shift-left movement. To be agile, automated tests should not be a separate stage of the development, but should be part and parcel of it. And to implement this should be the responsibility of both the tester and the developer.

And this responsibility brings us to what is for me, the holy grail of the shift-left movement:

Automated tester tests need to be part of the source code

There should be no difference between where the developer tests reside and where the tester tests reside. They should be part of the same source code, and as we said above, run as part of the same CI process. I will dedicate a blog post in the near future to this important idea (“Speaking the language of frontend developers”).

Remember, in an agile process, the tests become as important and integral as the development itself. Embrace agility.

Shifting Right

But if the shift-left movement means that a tester does not check the whole product holistically, then does that mean that it does not need to be checked as a whole?

Of course it has to be checked as a whole. The big question is when? If there is no release, then when do we test the whole product? The answer may surprise you and is part of the shift-right movement.

And the answer is — whenever you want to. Once there is no release, and software changes keep pouring out of the pipe, and they are well-tested due to the shift-left movement, then you can do E2E testing whenever you want to.

Run your end to end tests continuously

If you can run your end to end tests whenever you want, then just run them continuously. Yes, source code keeps pouring in from the left, and deployments keep pouring out from the right, but that doesn’t mean that we should give up and not test. It actually means that we should run them all the time.

No, we will never be testing what is exactly going to be in production in an hour. We will always be testing an approximation. But if this run of e2e tests doesn’t uncover the bug, the next one will. And, yes, it will be too late, because that bug is already in production, but that’s OK — because we’re agile, fixing the bug also takes a very short amount of time.

In an agile world, it’s OK for bugs to sometimes reach production. This is what is meant by Facebook when they say “Move fast and break things”. While an extreme sentence, it does hold the important truth that trying to stop all bugs from reaching production doesn’t help development, it harms it.

Run your end to end tests in production

Some companies take this to the extreme, and have software testers test the full product not in a testing or staging environment, but actually in production, after it’s released! (this is why it’s called “shift right”). And incredibly enough, this is not as dumb as it sounds. Testing in production enables us to uncover bugs that are due to:

• bad data that can only be found in production
• the scale and amount of people using the application in production
• the deployment system used for production
• and bugs that are in the infrastructure (servers, networks, database, etc) used in production

Embracing Agility

So yes, embrace the agile manifesto. Agility changes and amplifies the role of the software tester. The shift left movement encourages the software developer to take on the role of the tester, and encourages the software tester to participate in the software development process, while the shift right movement encourages the software tester to take on some of the responsibility of monitoring the application.

And this is a good thing — the tester should not be a gatekeeper, a judge that is aloof and separate from the software development and deployment process. The tester should not be the bad guy saying, “thou shall not pass” and delaying the release of the process.

What the tester should be is an integral part of the software development process. And embracing agility allows them to do that.

Yes, But How?

Embrace agility. Shift left:

• Software developers need to test too: shifting tests to the left means that software developers should test their code too. Educate your consumers. Work with them to make this happen. Fortunately, ongoing education is moving software developers to this understanding.
• Automated tester tests need to sit close to the code too: stop thinking as gatekeepers and start thinking as developers. Test parts of the code and not just the final product.
• Automated tester tests need to be part of the CI process: the CI tool is the new gatekeeper, so your tests should be part of the CI.
• Automated tester tests need to be part of the source code: if you can, have your tests be part of the source code, and not a separate piece.

Embrace agility. Shift right:

• Run your end to end tests continuously: there is no release. Continuously test the “current” product, no matter whether it is not yet in production or already there.
• Run your end to end tests in production: there is no release. The best place to run your tests is in production, and you will get a ton of added benefits from it.

What’s in it for Applitools?

In the end, I am a developer advocate for Applitools, a company that builds a Visual UI Testing platform for testers and developers. So, what’s in this movement for Applitools?

Plenty. We believe visual UI testing tools are crucial for shifting left and shifting right. But that’s for another future blog post.

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Embracing the Agile Manifesto: A Practical Approach for Software Testers appeared first on Automated Visual Testing | Applitools.

Once an application is internationalized, the engineering effort required to maintain its localized versions, and to localize it to new languages, is drastically reduced. However, the same is not true for UI localization testing.

In the first part of this post we reviewed common localization bugs, and the challenges involved in finding them. In this part, we show how Visual Test Automation tools can be applied to UI localization testing, eliminate unnecessary manual involvement of testers and language experts, and drastically shorten test cycles. 

Visual Software Testing is the process of validating the visual aspects of an application’s User Interface (UI).

In addition to validating that the UI displays the correct content or data, Visual Testing focuses on validating the layout and appearance of each visual element of the UI and of the UI as a whole. Layout correctness means that each visual element of the UI is properly positioned on the screen, is of the right shape and size, and doesn’t overlap or hide other visual elements. Appearance correctness means that the visual elements are of the correct font, color, or image.

Visual Test Automation tools can automate most of the activities involved in visual testing. They can easily detect many common UI localization bugs such as text overlap or overflow, layout corruptions, oversized windows and dialogs, etc. All a tester needs to do is to drive the Application Under Test (AUT) through its various UI states and submit UI screenshots to the tool for visual validation.

For simple websites, this can be as easy as directing a web browser to a set of URLs. For more complex applications, some buttons or links should be clicked, or some forms should be filled in order to reach certain screens. Driving the AUT through its different UI states can be easily automated using a variety of open-source and commercial tools (e.g., Selenium, UFT, etc.). If the tool is properly configured to rely on internal UI object identifiers, the same automation script/program can be used to drive the AUT in all of its localized versions.

So, how can visual test automation tools be used to simplify UI localization testing?

• Preparation – in order to provide translators with the context required to properly localize the application, screenshots of the application’s UI are often delivered along with the resource files to be localized.The process of manually collecting these screenshots is laborious, time consuming, and error prone. When a visual test automation tool is in place, updated screenshots of all UI states are always available and can be shared with translators with a click of a button. When an application changes, the tool can highlight only those screens (in the source language) that differ from the previous version so that only those screens are provided to translators. Some visual test automation tools also provide animated “playbacks” of tests showing the different screens, and the human activities leading from one screen to the next (e.g., clicks, mouse movements, keyboard strokes, etc.).  Such animated playbacks provide much more context than standalone screenshots and are more easily understood by translators, which are usually not familiar with the application being localized.Employing a visual test automation tool can substantially shorten the localization project’s preparation phase and assist in producing higher quality preliminary translations, which in turn can lead to fewer and shorter test cycles.

• Testing localization changes – visual test automation tools work by comparing screenshots of an application against a set of previously approved “expected” screenshots called the baseline.After receiving the translated resources and integrating them with the application, a visual test of the updated localized application can be automatically executed using the previous localized version as a baseline. The tool will then report all screens that contain visual changes and will also highlight the exact changes in each of the changed screens. This report can then be inspected by testers and external language experts without having to manually interact with the localized application. By only focusing on the screens that changed, a huge amount of time and effort can be saved. As we have shown in the first part of this post, most UI localization bugs are visual by nature and are therefore sensitive to the execution environment (browser, operating system, device, screen resolution, etc.). Since visual test automation tools automatically execute tests in all required execution environments, testing cycles can be drastically shortened.

• Testing new localizations – when localizing an application for a new language, no localized baseline is available to compare with. However, visual test automation tools can be configured to perform comparisons at the layout level, meaning that only layout inconsistencies (e.g., missing or overflowing text, UI elements appearing out of place, broken paragraphs or columns, etc.) are flagged as differences. By using layout comparison, a newly localized application can be automatically compared with its domestic version, to obtain a report indicating all layout inconsistencies, in all execution environments and screen resolutions.

• Incremental validation – when localization defects are addressed by translators and developers, the updated application must be tested again to make sure that all reported defects were fixed and that no new defects were introduced. By using the latest localized version as the baseline with which to compare the newly updated application, testers can easily identify the actual changes between the two versions, and quickly verify their validity, instead of manually testing the entire application.

• Regression testing – whenever changes are introduced to a localized application, it must be tested to make sure that no localization bugs were introduced, even if no direct changes were made to the application’s localizable resources. For example, a UI control can be modified or replaced, the contents of a window may be repositioned, or some internal logic that affects the application’s output may change. It is practically impossible to manually perform these tests, especially with today’s Agile and continuous delivery practices, which dictate extremely short release cycles. Visual test automation tools can continuously verify that no unexpected UI changes occur in any of the localized versions of the application, after each and every change to the application.

• Collateral material – in additional to localizing the application itself, localized versions of its user manual, documentation and other marketing and sales collateral must be created. For this purpose, updated screenshots of the application must be obtained. As described above, a visual test automation tool can provide up-to-date screenshots of any part of the application in any execution environment. The immediate availability of these screenshots significantly reduces the chance of including out-of-date application images in collaterals and eliminates the manual effort involved in obtaining them after each application change.

Application localization is notoriously difficult and complex. Manually testing for UI localization bugs, during and between localization projects, is extremely time consuming, error-prone, and requires the involvement of external language experts.

Visual test automation tools are a new breed of test automation tools that can effectively eliminate unnecessary manual involvement, drastically shorten the duration of localization projects, and increase the quality of localized applications.

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Taking the Pain Out of UI Localization Testing – Part 2 appeared first on Automated Visual Testing | Applitools.

Watch this exclusive session and learn from top test automation experts about the top trends and tools that will dominate your work environment in 2018, and leading skills and best practices that can accelerate your career.

Automation Guild Conf founder Joe Colantonio will host a panel with 3 Dev/Test thought leaders: Angie Jones, Richard Bradshaw, and Gil Tayar.

They will discuss major automation skills that are in high demand when hiring test engineers, as well as tools and best practices required to help you deliver faster and better.

Listen to this session, and discover: 

• Top test automation strategies for 2018
• Big opportunities in test automation
• AI automation: how is it going to affect you and how to prepare for it
• Tools and technologies you should get familiar with right now

You can watch the full recording here:

Joe’s slide deck is here:

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post 2018 Top Test Automation Trends: Must-have Tools & Skills Required to Rock 2018 appeared first on Automated Visual Testing | Applitools.

Automating accessibility testing is not only about making it easier for those with disabilities to access your site and app, but also optimizing your overall automated testing efforts, in order to support speedy product delivery without sacrificing software quality.

Watch this webinar to learn how to effectively automate accessibility testing, including: industry standards, challenges, available tools, and see – step by step – how to write a proper automated test. 

Leading companies are currently moving into Continuous Delivery and DevOps model, aiming to speed up their release cycle with a variety of tools and processes that support Continuous Integration. To match this need for speed, a fast feedback mechanism – “Automated Testing” – is used to validate the build.

But functionally testing an app is not enough, as it ignores end users with disabilities. Today. app developers and test engineers must validate the accessibility of the app to ensure it can be accessed by everyone – and that includes people with disabilities, which account for a large percentage of the population

Listen to Test Automation and Selenium expert Manoj Kumar in this in-depth session, where he covers the essentials of Accessibility Testing, including:

• What is Accessibility testing
• The Accessibility standards
• Different tools available to automate Accessibility testing
• How to automate Accessibility testing, and what resources are needed
• How to implement an effective Accessibility strategy within your organizations

Manoj’s slide-deck can be found below:

And you can watch the full recording here:

To read more about Applitools’ visual UI testing and Application Visual Management (AVM) solutions, check out the resources section on the Applitools website. To get started with Applitools, request a demo or sign up for a free Applitools account.

The post Automating Accessibility Tests: Web is for Everyone appeared first on Automated Visual Testing | Applitools.