What teams said their biggest test automation challenges were
During a live webinar with our integration partner TestRail, Global App Testing recently had the opportunity to ask an audience of QA experts what their biggest automation challenge was.
We wanted to ask this question on the back of the bombshell research from TestRail– which found that businesses systematically overestimated how much they would automate year on year. Businesses say they will automate more tests, and never quite seem to do so.
So, what’s stopping them? Here’s what our audience of QA engineers said.
PS: If you want to watch the webinar replay, it's available here. Additionally, you can check out our answers to the Q&A here or access our free trial offer with TestRail.
1/ Flaky tests due to a changing product (28%)
More than any other factor, respondents named flaky tests due to a changing product as the most important blocker stopping them from automating more.
On the one hand, this wasn’t the result we expected. Businesses are sometimes coy about the number of tests which flake, perceiving it as a failure. But it’s testament to how widespread automation difficulties are that it came out on top.
After all, this is a problem that plagues even the biggest businesses. One recent paper tells of how “At Google, of the 115,160 test targets that had previously passed and failed at least once, 41% were flaky. Of the 3,871 distinct builds sampled from Microsoft’s distributed build system, 26% failed due to flakiness.”
It’s very clear that the maintenance costs of tests are at least as significant as the costs of building them. A speaker at the webinar recounted an anecdote: “two automation engineers left a company and stopped maintaining their tests. Two months after they were gone from their posts, the company was fully manual again. There was no automated test which lasted for more than two months.”
2/ Not enough time (26%)
Time is the most finite resource in the universe. The second most frequent answer was that they just hadn’t got round to automating all the tests.
This is hardly surprising. After all, in our experience, falling behind your target as an internal QA team generally creates even more problems: the less you successfully automate, the more you have to test manually, further squeezing the time you had budgeted to automate more. This vicious cycle could be at the crux of why many businesses consistently target high percentages of automated tests than they achieve – one small slip and the team creates problems they need to spend their time solving.
3/ Lack of internal skills (19%)
In the middle were the lack of internal skills, with one fifth of respondents naming it the main reason they hadn’t automated more.
There are fewer complex dynamics here – you either have the skills or you don’t. But it’s worth noting that this is likely to be bias towards smaller businesses (we are not able to identify who voted what). For those segments, it then seems probable that this is the top reason why they do not automate more tests. Engineering labour is expensive; and, while “we didn’t have the budget” was the least commonly selected of all our answers, the cost of hiring engineers and quality engineers still seems like tough strategic decisions for any CTO.
4) Product is poorly suited to automation (12%)
It’s perhaps a slight selection bias that people who did not believe their product could have lots of automated tests did not show out in a webinar about how to automate more. Products which are complex and reliant on multiple integrations; which involve or relate to hardware steps; or where the most important testing from a business standpoint is not purely functional; are all poorly suited to automated tests.
5) Unable to secure budget for efforts (6%)
Software remains an extremely lucrative industry, and it’s no wonder that an inability to get budget was the lowest ranked of all the available choices. But could this be a symptom of QAs perceiving QA as cheaper? Watch the webinar via the link below and let us know what you think.