How (Not) To Choose a Test Automation Framework
by Devon Jones
Early in my QA career my roles mostly consisted of contributing to and refactoring a legacy UI Automation framework. I started with Java/Selenium, then JavaScript and Nightwatch.js, then Ruby/Capybara with Cucumber, learning the ins and outs of different frameworks as I came across them. When issues arose, I eventually started taking note of what could be traced directly back to the test framework. When I first started my QA journey I hadn't been in the room when most of my previous companies had selected their UI automation frameworks, but once I became a consultant and a Lead QA, I would often have a seat at the decision making table. As a result, I can now share exactly how a lot of framework selection comes about. The conversation usually starts with something like this:
“So I was thinking we should automate some tests. Not unit tests, end to end tests, for regression purposes”
“I agree! What should we use?”
And here are the absolute worst ways to pick an automated test framework:
{Some Engineer} Has Used it Before and is Willing to Build It!
“I’ve used { TestFramework } before, I could put something together…”
“Sounds great!”
Ok, so. While I understand the desire to leverage existing resources, going with a framework just because someone at the company has used it will only save you time and effort in the short term. Especially if “someone” is a developer as opposed to a quality engineer, although quality engineers are also guilty of just sticking with what they know regardless of use case.
Aside from the obvious programming language and test code syntax, frameworks vary widely in what browsers they can run against, what functionality they are able to automate, and the level of stability, documentation and support they provide. The odds that your co-worker’s former business had the exact same needs as yours are very low. And they may have been using the wrong tooling too!
It’s in the same language as our dev code!
“Let’s just go with { TestFramework } since it’s also written in { developer coding language }”
Think about it- In software engineering, no company would suddenly decide to build an app in a programming language without carefully examining and debating speed, security, compatibility with other tooling and libraries, documentation, and overall stability and support. But in the QA space, it happens all the time.
UI automation code approaches the application from the perspective of an end user (black box), so the use case and requirements of the programming language are very different. Test code is more narrowly focused and needs to be very concise and readable so that it’s clear what is being tested. It’s also usually maintained and written by QA Engineers, who have a different technical skill set from developers. Lastly, it needs to be easy to maintain so that the regression suite can be both consistent and reliable in finding bugs before production.
Keeping this in mind, look at all of the new (Playwright, Cypress), old (TestCafe, WebDriver IO), and deprecated (Protractor, PhantomJs) Node.js based JavaScript UI testing frameworks. They not only clutter up test code with needless async/awaits or promise chains, but Node itself is constantly evolving. For example, Python was released in 1991 and it’s on version 3.13, Ruby was released in 1996 and it’s currently on version 3.4. Node came out in 2011 and it’s on (as I’m writing this) version 21. This can throw a test suite into chaos as things that were once supported stop being supported. I remember when a change to Node broke WebDriver.ios popular “synchronous mode”, and all existing tests had to be updated to async syntax. It’s hard enough keeping a test suite up to date with changes to the app itself, much less having to constantly monitor risks from the programming language.
It Will Save Us Time With [Insert Supposed Shortcut]!
The saying “if it sounds too good to be true, it probably is” applies to all things, and testing frameworks are no exception. When Cypress came out, it was touted as an automation framework “By Developers, For Developers” that would allow them to quickly spin up end to end tests. Being able to interact with the development code was another bonus for developers, you could just get your page into the state you want and test it. There were other advertised perks, and Cypress quickly became the “must know” tool of QA test automation, despite its initial use case as being for developers . As a consultant at the time, I was excited to try it, and spun up a POC to show upcoming clients.
Or… tried to. Since Cypress wasn’t written “by and for QA professionals” I had a hard time trying to understand the syntax and the design pattern Cypress had established. I didn’t have a choice of assertion library, I couldn’t run against Safari (our second most used browser), I had to fight the framework to use an object oriented design pattern (Page Object Model) for readability and reusability of common methods. As I wasn’t a developer, I couldn’t access the development code to make any quick and easy magic happen, and since the tests should black box, it’s not a reliable or realistic testing environment anyway.
Would it be fast for a JavaScript developer? Probably. Cypress picked a set of functionality and tooling and packed it together, so it’s ready to use out of the box, but that fixed set of tools isn’t going to work for every use case. Even after I figured out how to structure my tests and got more familiar with how Cypress worked I’d still hit a wall when an application would need something automated that Cypress didn’t support or needed a special workaround/specific plugin to implement. I have made test POCs in Cypress for a few companies along with a list of capabilities and limitations and it was passed over each time in favor of more robust options.
It’s The Hot New Framework Everyone is Talking About!
I remember the first time I heard “So there’s this new framework that everyone’s talking about…”
I was working at my very first QA job as a dedicated Test Automation Engineer, still learning to navigate my team’s Selenium/TestNg/Java repo. During our morning standup, one of my coworkers proposed creating a POC of an innovative new test framework, to see if it would be a worthwhile addition to our comprehensive regression suite. The framework? Cucumber.
The basic idea behind Cucumber is to map test code to regular words, enabling non-technical team members to easily create test scenarios. It was also supposed to make tests easier to read. Three of us volunteered to take a stab at automating some basic tests during the next sprint and report back at our retro.
Now, I know there are companies that happily and successfully use Cucumber on a regular basis, despite all of the newer, shiner, more trendy frameworks making the rounds. It may be the perfect framework for their needs and resources. Some may even use it for the use case it was created for; allowing non-technical contributors to create tests using the written test steps. And automation is like sunscreen - the best one is the one you will actually use.
That being said… For us, it was terrible. First we had to ramp up on Regex. No biggie, we all used it to some extent anyway. But this was a LOT of abstraction. We then added the Regex files to the standard test files. Then we had to agree on what words to use for each of the steps. How granular did we want to be? Should we use “click” or “select”?
Then we debated on how to communicate what words mapped to what steps for our Product Owners and Project Managers. A Confluence page glossary? Team Wiki? We went back and forth on that for a while, and then finally called them over to our area. We excitedly told them that we’d be demoing a new way of testing that would allow them to make tests on their own, explained how it worked, and asked them what language they’d prefer. They stared at us for a beat, glanced at each other, and said “Yeah… we aren’t going to do that.”.
Which is fair. Not only do they have their own responsibilities, but we already had a fully built out regression suite, an effective system for requesting new automation, excellent automated reporting, and a low bug rate. Implementing Cucumber would just mean more work for both groups and no tangible benefit besides being able to say “We use Cucumber too!”
When I started looking for my next opportunity I couldn’t believe how many companies wanted Cucumber experience. As I had some, I usually got at least a first round interview, and one of the first questions I always asked was “Why Cucumber?” I got lots of explanations regarding its readability and its behavior-driven-ess, but when I asked if non-technical stakeholders were using it to construct tests, the answer every time was a resounding “No.” At this point I’d usually follow up with “Then why are people that know how to code writing Regex and creating additional files to map code to words? Can’t they just use good naming conventions and documentation to make any framework readable?”. And the answer to that usually boiled down to somebody making an executive decision that Cucumber was going to be used based on industry buzz.
Cucumber has mostly passed out of fashion now… I haven’t seen it as a desired skill in a job posting in a while.
After Cucumber came Cypress (see the above point), and the most recent star in the testing sky is Playwright (yet another node based framework), which is currently a must-have skill in a lot of QA related job posts.
The point of this article isn’t to dissuade anyone from any specific framework. With the right use case and implementation, any framework can work well for a company’s needs. It’s companies deciding to implement a tool without evaluating if it’s the right tool for their needs and if they have the resources to adequately implement and support it. And when the hastily chosen framework is hard to learn, hard to maintain, hard to debug, or doesn’t support the functionality or browsers they need it to support… they blame test automation as a whole and contribute to the assumption that automated tests are difficult to write, flakey, and not worth the effort. Which inevitably leads to the invention of yet another shiny new (likely node based) framework. And the cycle continues.
While I have experienced all of the above first hand, and could state the issues I’ve had with various frameworks personally, one of my favorite voices in the QA industry, Zhimin Zhan, has beautifully laid out the business case for avoiding JavaScript frameworks. I highly recommend his resources and books, which I will link for you below!
🔗 Why Javascript is not a Suitable Language for Real World Test Automation
