What The World's Top 12% Testing Teams Are Doing (That You Can Do, Too)

We know that you’re thinking, “Really? Another fluff post about the top people in my profession and what I need to do to be just like them?”

Listen, we hear you, but this is not some fluff piece with vague ideas of what might (or might not) work. Read on, follow along for the next few weeks, and we promise it will be worth your time! If not, write to us directly and tell us what we can do better next time. We will reply!

In case you missed it, on May 27th the 2019 State of Automated Visual Testing was released. Based on independent research sourced from over 350 testing teams around the world, we learned that 12% of you are getting much better results than the other 88%. We’re talking four times more successful as measured by the things you (and your boss, and bosses’ boss, and your bosses’ bosses’ boss) really care about – test coverage, release velocity, application quality, overall R&D teamwork, and cold hard cash! This is not our opinion. It’s not subjective. It’s objective data and information. Data and information that came from you and your peers.

In other words, we’re not asking you to take our word for it; we’re asking you to take your word for it. Take a moment to download the full report. We will be here when you get back!

Click here to download the report.

So what separates the top 12% from the other 88%?

Digital Transformation. Two words that have been baked into our world over the past 20 years. To such an extent that it may be fair to call it fluff? Fluff or not, IDC forecasts that worldwide spending on technologies and services that enable digital transformation will reach $1.97 trillion in 2022, per the (IDC) Worldwide Semiannual Digital Transformation Spending Guide.

(If you think a trillion is an abstract concept, check this out.)

Dang. That’s a lot of fluff! When your bosses’ bosses’ boss is spending that kind of cash, it’s always worth paying attention. It could be good for your career. As it turns out, 12% of the world’s testing teams did pay attention, and it’s their approach to managing the challenges of digital transformation that have set them apart.

The Testing World’s Digital Divide – Digital Transformation Quantified.

You can read up on The Enterpriser Projects CIO level take on Digital Transformation (warning: fluff alert) here, or you can quantify it for yourself with some simple math and see how you compare to other R&D and testing teams around the world. Got that calculator ready? Here we go…

How many applications do you have in production today? (Don’t forget those native mobile apps, they really add up).
How many pages or screens do you have in production on average for each of these applications? (Single Page Applications can be tricky we know, but give it your best guess).
How many viewport breaks do you support? (The market average is six if you are not sure.)
How many human languages do you support? (We’re talking about localization here – English, German, Spanish, Chinese, Japanese – not coding languages in case you’re wondering).

Now multiply those four numbers together. Congrats! You have just quantified the digital footprint of your business. People can write about digital transformation all they want, but you’ve just transformed all that fluff into something quite real. It’s probably a big number. Over 90,000 for a typical company and over 624,000 for the largest 30% of companies in the world. Here’s the kicker. You and your R&D team are responsible for managing the visual and functional quality of every single one of those pages and screens.

We categorized the number of screens in production by industry, based on your responses. Here’s what we discovered.

When functional test automation first emerged way back in 1989 with the launch of Mercury Interactive XRunner, it was a much different time. Browsers didn’t even exist. Since then, the browser wars have come and gone and standards are now in place. Applications have grown far more complex with native mobile, dynamic content, single page applications (SPA) and responsive design now an everyday reality. Digital transformation for any size company is officially past tense. We’re not transforming, we’re transformed. And now you have to deal with it, but how?

What Defines a Top 12% Global Testing Team?

Top 12% teams have overcome the massive challenge posed by digital transformation. Like any technical challenge we have ever faced, it started with someone on the team who felt the pain and set out to solve for it. And when they did, good things happened.

Test coverage increased by 60%.

Release velocity became 2.8x faster (even though coverage increased 60%)

Visual and functional quality improved 3x with far fewer escape bugs (even though they release 2.8x times faster)

R&D teams are 4x more satisfied with their visual and functional quality outcome (yea, I’d be more satisfied too with those types of results!)

All of this despite the fact that this 12% of testing teams were managing applications 2.2x larger than the other 88%. In all likelihood, they felt the pain before most of us, and have now led the way for the rest of us. We just need to follow.

Goodbye, But Only For A Week or So.

Today successful continuous management of application visual quality creates competitive advantage. Business leaders know this and are paying attention. As a result, testers are in a better position than ever to be heroes respected by their R&D teams for driving huge value for the companies they work for.

Over the next 10-12 weeks, Patrick and I will release a series of blogs that explain how these 12% of testing teams reinvented their testing approach to deal successfully with the challenges of Digital Transformation. We will get into the details as promised.

Or, if you simply can’t wait and want to ignore my spoiler alert, you can listen on-demand to the webinar Patrick and I hosted together entitled Wrong Tool, Wrong Time: Re-Thinking Test Automation, read the blog and view the slides here, or just reach out to us for help with your testing approach.

Until next time.

Yours Truly,

James the #NotSoEvilMarketingGuy

Patrick the #GuyWhoActuallyHasDoneTestingFor20Years