(Editor’s Note: This post has been recently updated for accuracy and completeness. It was originally published in March 2019 by Al Sargent.)
Have you tried using Selenium IDE for your QA test automation?
You can find lots of feedback from users around the world.
Still skeptical? That makes sense.
There’s been plenty of stigma around using record and replay tools like Selenium IDE rather than scripted QA automation tools like Selenium Webdriver, Cypress, and WebdriverIO. And, for seemingly good reason.
Traditionally, record and playback tools suffer from a litany of issues, including:
- No cross-browser support
- Brittle tests
- Difficult to wait for app under test
- No conditional logic
- Chaining one test script to call another not available
- Unable to embed code into recorded scripts
- No way to edit scripts once recorded
- Lacking a script debugger
- No way to run scripts in parallel
- No way to run tests from Continuous Integration build scripts
- Lack of integration with source code control systems
- No plugins to extend functionality
- No way to do visual UI testing
- Poor support for responsive web
- No way to quickly diagnose front-end bugs
- Unable to export tests to languages like Java
- No way to enable data-driven tests
Revising Selenium IDE
Back in 2019, Applitools helped revise the Selenium IDE project. Two years earlier, the project had effectively died. Selenium IDE only ran on Firefox. With Firefox 55, Selenium IDE broke, and there seemed to be no motivation to fix it.
Plenty of articles back then explained why Selenium IDE was bad. There was this Quora thread comparing Selenium IDE with Selenium Webdriver. And plenty of issues listed in the Selenium IDE questions on Stackoverflow. Plus this top 10 list of issues with record & replay.
However, Applitools engineers got involved and addressed the bugs – as well as some of the shortcomings. In a major enhancement, Applitools made it possible to run Selenium IDE on both Chrome and Firefox. The team expanded the code export functionality from IDE-captured tests. Also, the team provided code hooks allowing others to write their own export hooks.
With great Applitools integration, Selenium IDE can help engineers with or without coding skills build effective tests quickly.
Sixteen Reasons Outlined
Here’s a list of 16 reasons why — and one why not – to try Selenium IDE. Read them, and let Applitools know what you think.
Let’s dive in.
#1: Selenium IDE is cross-browser
Selenium IDE first came out in 2006.
It was a different time. iPhones didn’t exist, the Motorola Razr flip phone was the must-have device, and Borat topped the movie box office. Firefox was the shiny new browser, and Chrome wouldn’t come out for two more years.
So it’s no surprise that Selenium IDE hitched its wagon to Firefox. Unfortunately, it remained that way for over a decade, frustrating the heck out of users with its single-browser support.
No more.
Selenium IDE runs as a Google Chrome Extension…
….and Firefox Add-on:
Even better, Selenium IDE can run its tests on Selenium WebDriver servers. You can do this using Selenium IDE’s command line test runner, called SIDE Runner.
You can think of SIDE Runner as blending elements of Selenium IDE and Selenium Webdriver. It takes a Selenium IDE script, saved as a .side file, and runs that using browser drivers should as ChromeDriver, EdgeDriver, Firefox’s geckodriver, IEDriver, and SafariDriver.
SIDE Runner, and the drivers above, are available as a straightforward npm installs. Here’s what it looks like in action:
#2 Robust Tests
For years, brittle tests have been an issue for functional tests — whether you record them or code them by hand. A huge contributor to this problem has been object locators. These are how your QA automation tool identifies which field to fill, or which button to click. These can be a button label, an XPath expression, or something else.
Developers are constantly sadistically tormenting QA teams releasing new features, and as a result, their UI code is constantly changing as well. When UI changes, object locators often do as well.
Selenium IDE fixes that by capturing multiple object locators when you record your script. During playback, if Selenium IDE can’t find one locator, it tries each of the other locators until it finds one that works. Your test will fail only if none of the locators work.
This doesn’t guarantee scripts will always playback but it does insulate scripts against many changes. Here’s a screenshot of how it works. As you can see, Selenium IDE captures linkText, an XPath expression, and CSS-based locators.
Imagine building this functionality in Selenium Webdriver. You’d have to first gather up all potential Xpath locators, then all CSS locators, then iterate through each until you find an object match. It’d be a huge chunk of time to automate just one interaction, and you’d be left with a mess of hard-to-maintain code.
Selenium IDE provides an alternative that is fast, resilient, and easy-to-maintain.
#3 Wait For Your App
When running tests, it’s essential to give your application time to catch up to your test automation tool. This can include time for backend operations, fetching page elements, and rendering the page. It’s especially necessary when running on staging servers that are under-resourced.
Why does waiting matter? If your test script tries to interact with some page element (field, button, etc.) that hasn’t loaded, it will stop running.
Thankfully, the new Selenium IDE knows automatically wait for your page to load. Also, commands that interact with some element wait for the element to appear on the page. This should eliminate most, if not all, of your explicit waits.
But, if that’s not enough, the new Selenium IDE gives you other options.
In the new Selenium IDE there’s a global set speed command that you can use to pause after every test step. Even better, you can set this from the toolbar in the new Selenium IDE. Check it out below.
Between automatic waits and global set speed, you should have a lot fewer pause commands. That means your tests will be simpler and easier to maintain.
If you need more fine-grained control, Selenium IDE lets you insert steps to wait for an element to meet some condition: editable, present, or visible — or the opposite (not editable, not present, or not visible).
Finally, there’s the pause command that you can insert after individual steps. Selenium IDE has had this for a long time; feel free to use if you’re feeling nostalgic.
#4 Conditional Logic
When testing web applications, your scripts have to handle intermittent user interface elements that can randomly appear in your app. These are those oh-so-helpful cookie notices, as well as popups for special offers, quote requests, newsletter subscriptions, paywall notifications, and adblocker requests.
Conditional logic is a great way to handle these intermittent UI annoyances features. You want your scripts to say, If X appears, click the link to make it go away.
You can easily insert conditional logic — also called control flow — into your Selenium IDE scripts. Here are details, and how it looks:
#5 Modular Test Scripts
Just like application code, test scripts need to be modular. Why?
Many of your test scripts will have steps to sign into your app, sign up for an account, and sign out of an app. It’s a waste of time to re-create those test steps over and over.
Selenium IDE lets one script run another. Let’s say you have a login script that all your other scripts call. You can easily insert this step into Selenium IDE. Here’s how it looks:
This way, if your sign in, sign up, or sign out functionality changes, you only have one test script to change. That makes test maintenance a lot easier.
Here’s a quick demo of this in action:
#6 Selenium IDE supports embedded code
As broad as the Selenium IDE API is, it doesn’t do everything. For this reason, Selenium IDE has execute script and execute async script commands that lets your script call a JavaScript snippet.
This provides you with a tremendous amount of flexibility by being able to take advantage of the flexibility of JavaScript and wide range of JavaScript libraries.
To use it, click on the test step where you want JavaScript to run, choose Insert new command, and type execute script or execute async script in the command field, as shown below:
#7 Scripts Can Be Edited
In the old Selenium IDE, scripts couldn’t be edited. For this reason, Selenium IDE tests were considered throwaway scripts: if they didn’t work, you’d have to delete them and re-record a test.
With the new Selenium IDE, you can easily modify your tests. Insert, modify, and delete commands, as you can see below. No more throwaway scripts!
#8 Available Debugger
Pretty much every IDE on the market has combined an editor and a debugger. (That is, after all, what’s meant by Integrated Development Environment.)
But not the old Selenium IDE. It had no debugger. (Whoops.)
The new Selenium IDE lives up to its name, and provides a way for you to set breakpoints in your script. Just click on the left margin of your test.
This way, you can inspect your browser’s state when your script stops due to a breakpoint. Here’s how it looks:
This makes it a lot easier to troubleshoot issues. (Speaking of troubleshooting, check out #16 below.)
#9 Run Scripts In Parallel
With the old Selenium IDE tests could only be run one at a time. This made tests take much longer. Alternatives like Selenium Grid were only available when used with Selenium WebDriver.
Selenium IDE can run tests in parallel. This lets you get through your test suites much faster.
To run multiple SIDE Runner tests in parallel, just tell it the number of parallel processes you want. Here’s an example of running three tests at once:
No, that’s not a Bandersnatch reference…
Here’s a quick video of this in action (view in full screen since the fonts are small):
#10 Run From CI Build Scripts
Because SIDE Runner is called from the command line, you can easily fit into your continuous integration build scripts, so long as your CI server can call selenium-ide-runner and upload the .side file (your test script) as a build artifact. For example, here’s how to upload an input file in Jenkins, Travis, and CircleCI.
This means that Selenium IDE can be better integrated into your DevOps toolchain. The scripts created by your less-technical QA team members — including business analysts — can be run with every build. This helps align QA with the rest of the business and ensures that you have fewer bugs escaped into production.
#12 Selenium IDE scripts can be managed in a code repository
Other record and replay tools store their tests in a range of binary file formats. (For example, here are UFT’s binary file formats.) You could check these into a source code repo, such as GitHub or GitLab, but it wouldn’t be all that useful since you couldn’t inspect test scripts, compare differences, or selectively pull in changes.
In contrast, the new Selenium IDE stores test scripts as JSON files. This makes them easy to inspect, diff, and modify. Here’s a script I recorded, viewed in Sublime text editor. You can easily change the starting URL, window size, and object locators.
If you manage your Selenium Webdriver scripts in GitHub, GitLab, Bitbucket, Azure DevOps, AWS CodeCommit, Google Cloud Source, or some other source code repo, you can now do the same for your Selenium IDE scripts.
#12 Extensible With Plugins
Unlike the old Selenium IDE, the new Selenium IDE supports third-party plugins to extend its functionality. Here’s how to build your own Selenium IDE plugin.
This is pretty exciting. You can imagine companies building plugins to have Selenium IDE do all kinds of things — upload scripts to a functional testing cloud, a load testing cloud, or a production application monitoring service like New Relic Synthetics.
Plenty of companies have integrated Selenium Webdriver into their offerings. I bet the same will happen with Selenium IDE as well.
Speaking of new plugins…
#13 Do Visual UI Testing
We here at Applitools have built a Selenium IDE plugin to do AI-powered visual validations on Selenium IDE, called Applitools for Selenium IDE. (Imaginative, right?)
To get it, head to the Chrome and Firefox stores, do the three-second install, plugin your Applitools API key, and you’re ready to go.
Create a Selenium IDE script, choose Insert new command, type eyes (that’s the name of our product), and insert a visual checkpoint into your test script. Like this:
Visual checkpoints are a great way to ensure that your UI renders correctly. Rather than a bunch of assert statements on all your UI elements — which would be a pain to maintain — one visual checkpoint checks all your page elements.
Best of all, Applitools uses visual AI to look at your web app the same way a human does, ignoring minor differences. This means fewer fake bugs to frustrate you and your developers — a problem that often leads simple pixel comparison tools to fail. When Applitools finds a visual bug, it’s worth paying attention to.
Here’s an example of Applitools Visual AI in action, finding a missing logo on a GitHub page. We didn’t have to create an assert statement on the logo; Applitools visual AI figured this out on its own.
#14 Visually Test Responsive Web Apps
When you’re testing the visual layout of your responsive web apps, it’s a good idea to do it on a wide range screen sizes (also called viewports) to ensure that nothing appears out of whack. It’s all too easy for responsive web bug to creep in.
And when they do, the results can range of merely cosmetic to business-stopping. Here’s Southwest Airlines putting the kibosh on their checkout process with a responsive bug that covers up the Continue button:
Not good, right?
When you use Applitools for Selenium IDE, you can visually test your webpages on Applitools Ultrafast Grid. This cloud-based testing service has over 100 combinations of browsers, emulated devices, and viewport sizes. This lets you do thorough visual testing on all your web apps.
Here’s how you specify which combinations to test on:
Once your tests run on Ultrafast Grid, you can easily check your test results on all the various combinations, like this:
Your responsive web bugs can run but they cannot hide…
#15 Pinpoint The Cause Of Front-end Bugs
Every Selenium IDE script you run with Ultrafast Grid can be analyzed with our Root Cause Analysis.
This matters because, to bastardize Jerry Seinfeld, it’s not enough to FIND a bug. You have to FIX the bug.
Like the Seinfeld car rental company, every testing tool I know of finds bugs, but doesn’t tell you how to fix them.
Except Applitools.
When you find a visual bug in Applitools, click on it, and view the relevant DOM and CSS diffs, as shown below:
I want to point out that we don’t show all DOM and CSS diffs — just the handful that are likely to have caused a visual bug. This makes debugging visual bugs go much faster.
We covered a ton of different ways Selenium IDE and Applitools work together. Here’s a visual summary:
#16 Export Webdriver scripts
Originally, Selenium IDE could export to Webdriver Java, but the 2019 refresh required additional coding. That code has been written for the following exports:
- C# NUnit
- C# xUnit
- Java JUnit
- JavaScript Mocha
- Python pytest
- Ruby RSpec
Additionally, you can create and contribute your own code export package. You can find the instructions in the Selenium IDE documentation.
Selenium IDE Limitations
Since this document first got posted, the two limitations have been addressed substantially. Originally, code export needed to be completed, and it was – with Java support in early 2019. As mentioned above, anyone can contribute scripting export code to the project, which is how the export set has grown.
Selenium IDE doesn’t support data-driven scripts directly
In the original design, Selenium IDE could not import a bunch of tabular data, like a CSV file or database table, and then run a parameterized test once for each row of data. The direct feature is still of interest – but remains blocked by a bug. You can track progress here.
However, intrepid engineers have proposed a work-around using SIDE Runner. Contributor PawelSuwinski writes:
“With SIDE runner is a just matter of side file preprocessing before running. I did it in some php project as part of a composer script, I do not have any JS npm run-script working example but would use templates concept this way:
- For CSV data use something like csv2json to get data in JSON format
- Creating template SIDE file use store json with Target like ex. %DATA%
- In preprocessor replace all ‘%DATA%’ in template side file with target data (ex. using rexreplace) and save it as a target side file (in cache/ tmp area)
- Run side runner on target side file”
Work on this feature continues. Let Applitools know if you have tried the workaround successfully.
Summary
Here’s how Selenium IDE compares to traditional record & replay:
| Capability | Traditional Record & Replay | Selenium IDE |
|---|---|---|
| Cross-browser support | No | Yes |
| Resilient tests | No | Yes |
| Automatically wait for app under test | No | Yes |
| Conditional logic | No | Yes |
| Run one test from another | No | Yes |
| Embed code into scripts | No | Yes |
| Edit scripts | No | Yes |
| Debug scripts | No | Yes |
| Run scripts in parallel | No | Yes |
| Run scripts during CI builds | No | Yes |
| Manage scripts in source code repo | No | Yes |
| Plugins to extend functionality | No | Yes |
| Visual UI testing | No | Yes (w/ plugin) |
| Responsive web support | No | Yes (w/ plugin) |
| Diagnose root cause of visual bugs | No | Yes (w/ plugin) |
| Export tests to code | No | Yes |
| Data-driven tests | No | Workaround proposed |
‘Less is more
Selenium IDE is part of a larger trend of software making life simpler for technical folks. One example: the broad range of codeless tools for developing applications.
Other examples: Serverless offerings like AWS Lambda make it easier to write just the code you need to get a job done. And Schemaless databases like MongoDB provide architects with much more flexibility to innovate versus tightly constricted SQL databases.
Codeless, serverless, schemaless — and now scriptless, with Selenium IDE. We might be seeing a trend here.
Go deeper
To get started, check out this tutorial on Selenium IDE. It’s done by Raja Rao, a former QA manager who’s been using Selenium Webdriver for over a decade. So he knows what he’s talking about.
Beyond that, here’s a fairly complete list of resources to learn the new Selenium IDE in 2021:
Selenium IDE pages
- Documentation
- Tutorial
- Plugin on Google Chrome Store
- Plugin on Firefox Store
- GitHub page for Selenium IDE
- SIDE Runner npm page
- Stackoverflow on Selenium IDE
- Quora on Selenium IDE
- Selenium IDE on Reddit
Applitools for Selenium IDE pages
- Product Page – Applitools for Selenium IDE
- Documentation
- Google Chrome Store – Applitools for Selenium IDE
- Firefox Store – Applitools for Selenium IDE
Videos
- Selenium IDE tutorial video playlist
- Dave Haeffner presenting Selenium IDE at Selenium Camp
- Dave Haeffner presenting the new Selenium IDE launch webinar
- Tomer Steinfeld on how he built the new Selenium IDE
- Joe Colantonio presentation on Selenum IDE, part one and part two
- Simon Stewart: Selenium IDE – the next generation
- Other Selenium IDE videos
How do you plan on using Selenium IDE? Let us know!
