How to calculate whether QA tests should be automated or manual

P.S. If you're here because you want to automate more tests, you should check out our solution immediately below this.  It's a non-automated solution, but one reason lots of customers use our services is to help them focus in this area. Check it out. 

In a recent webinar with the easy CI/CD tool Buddy Works, we looked at how businesses can calculate the true cost of testing and use it to determine whether tests should be automated or manual. You can check out our thinking on the subject below .👇

Why do businesses believe they will automate so many tests?

In TestRail’s first annual survey in 2018, businesses set out their plans for test automation. The 6,000 respondents automated 42% of their tests and planned to automate a further 21% next year. 

But they didn’t. In the 2019 survey, the same 42% of tests were automated, and this time, businesses said they would automate 61% in 2020. By the most recent survey in 2021, just 38% of tests were automated. By now, the pattern is consistent. Businesses systematically overestimate the amount they will automate. 

But why?

Why businesses like test automation

Teams tend to like the idea of automating tests. That’s because:

• You can run automated tests whenever you like
• Automated tests return results instantly
• Automation is perceived as a one-time investment, which would make it cheaper to automate over the long term. (In our experience, this is only sometimes true.) 

And then together these factors lead to even better second-order effects: 

• You can remove bottleneck slowing down your releases if your tests are instant 
• You can improve your DORA metrics as you measure your progress. 

But the reality of testing difficulty belies this. We ran a survey during a separate webinar about the top reasons businesses felt they couldn’t automate more tests. And here’s the TLDR: 

• The top result (28%) of respondents cited flaky tests due to a changing product. The second result (26%) is not enough time to automate.  
• Both answers are time. “Flaky tests due to a changing product” really refers to the time investment of maintaining your tests. “Not enough time to automate” refers to the time investment of setting them up.
• Businesses are underequipped to calculate the time costs of building and maintaining tests, or the other time demands which will be made of them in the cut and thrust of product development. 

What’s the equation to calculate whether a manual or automated test is better?

ST + (ET x N) = the true time cost of testing. 

You can check this for automated and manual tests to identify whether it’s cheaper for your business to execute a test manually or to automate it. 

ET is the execution time. We know that automation is much faster here, and it’s the main metric businesses focus on when they want to automate all their tests. For Global App Testing, we offer 2-6 hour test turnaround with real time results. Tests land in tester inboxes straight away, so in many cases the first results come through much faster.

ST is the setup time including any maintenance time investment. It takes more time to automate a test script than it does to quickly test something or to send it to a crowdtester like Global App Testing. Setup time is also the second barrier to setting up tests, so it’s worth running this algorithm twice – one to add up which is more expensive, and one with adapted algebra to calculate the maximum time your business can invest in one go. 

N is the number of times a test will be used before it flakes. It’s great that execution on an automated test is very rapid; but the saving is immense on a test used 1000s of times. If the test will be used twice before it flakes, the return is less impressive.

A final note is to ensure you know what you’re optimizing for. Is time or money more important? The labour costs of the individuals setting up the automated test (developers) versus the labour costs of individuals executing tests (global QA professionals) could be different; and try running this algorithm with both units plugged in. 

Where can I watch the webinar? 

You can watch it below.