Test automation ROI: how to calculate it

823 0

Calculating the ROI for test automation is not a simple task. Investing in automation can be expensive and success depends on many factors which are highly related to your specific business and QA requirements.

Everyone agrees that test automation brings proven benefits. Test automation can help to:

  • increase requirements or user story coverage;
  • release software faster and more frequently;
  • reduce the risk of finding defects in production;
  • reduce the effort and focus manual testing where it’s really necessary;
  • help detect defects earlier in the software development life cycle.

So if the benefits are clear, what’s the issue? The challenge can arise when you submit a budget request for test automation software or service: the budget owner will consider every cost, including yours, based on how much money it will make or save the company.

Calculating the ROI for your test automation strategy is fundamental to figure out if this transition is worth the change and to compare the suitable solution providers on the market. Before executing, or even thinking of building out an automation strategy, the very first thing you have to do is to calculate the net gain you’ll see from transitioning.

At Qualibrate we have been helping QA organizations implement automated software testing for over ten years. We have learned that test automation can provide a relevant return on investment (ROI). And we have learned some things about how to quantify the ROI to help present an effective business case for automation.

In this article we will guide you through the calculation of this figure, to come up, eventually, with an accurate result for your own business. We will start understanding what the ROI is. Then we will briefly explain the common mistakes QA managers fall into, when finding the result. After it we will go through every single factor that you need to take into consideration.

 

What is ‘Return on Investment (ROI)’?

Return on Investment (ROI) is a measure of profitability that evaluates the performance of an initiative by dividing net benefits by net investment.

Test automation ROI formula

Test automation ROI formula

As ROI is measured as a percentage, it can be easily compared with returns from other investments, allowing one to measure a variety of types of investments against one another.

 

Qualibrate ROI calculator

ROI reportTo make your life easier we created an online calculator based on our expertise. All you need to do is fill in the form with your own input and you will receive a full report with:

  1. ROI for year 1, year 2 and year 3
  2. Payback period
  3. Total savings

If you want to understand how we created this calculator, keep reading and we will go through every single factor to take into consideration and how it affects the final result.

 

Common mistakes

Before analyzing the components on cost and invests let’s see the common mistakes when calculating this figure:

  • Considering automating 100% of your test cases.
    Manual testing will always be important: there are scenarios that will always require manually executed test cases. This means you’ll still have to create, run, and maintain those tests.How to choose what to automate then?
    The benefit of test automation is proportional to how many times a given test can be repeated. Tests that are only performed a couple of times a year won’t bring you a positive ROI. The first candidates for test automation are tests that you run frequently and require a big amount of data to perform the same action. Or the most used functionality that introduces high-risk conditions and also tests that take a lot of effort and time when manual testing. Check this article to choose the test cases you want to automate.
  • Not accounting for test maintenance.
    As the business evolves and adjust to customer & legal requirements, the test cases also have to be updated to reflect these changes. Hence, after creating and implementing your automation strategy, you will need to continuously maintain and update your tests. When you build new features and continue to make product improvements, your test cases and regression suites will grow. Ensuring these are usable over months or years will require continuous maintenance.
  • Not taking into consideration the extra benefits of test automation
    There are some extra benefits when implementing a test automation strategy that are hard to calculate. However, it is always good to have them in mind as intangible gains:

    • Not depending on expertise: with the help of a friendly test automation provider, like Qualibrate, anyone can create and run test scripts. That means you won’t need to rely on long and difficult lines of code and on tester expertise. You just need to make sure to create a strong basis upon your test automation strategy and you will be stress-free also when your best tester will leave the company.
    • Reduce project slippage and faster time to market: with a good QA strategy and the support of test automation, you will be able to deliver faster, increase releases per year and avoid annoying project delays due to bug fixing. If you play in a very competitive sector, being able to be the first on the market will give you a strategic advantage on competitors.
    • Team morale increase: executing repetitive, manual test cases can be frustrating. Leveraging on test automation gives your team time to spend on more challenging and rewarding projects.

Savings – factors

Let’s see, in practice, how to calculate the savings:
 
1) Time saved for regression testing

Regression testing is fundamental for the success of your QA activity, as it will help you ensure that your application is bug-free and that product features that were already validated continue to work properly.
Over time, test cases to run will grow and will take longer to execute. By implementing test automation you will most benefit by saving time for testing. In Qualibrate we’ve helped customers to release from weeks to hours, as you can see here in this business case.

To come up with how much you save you need to consider the following key variables that make your organization unique before getting started:

  • Business Processes in scope, including variants.
    For Business Process we mean the end to end scenarios you run in your application.
    The more business processes and complex you have, the more you will save. You should consider that, growing your business and adding a new feature to your product means that you will increase your Business Process in scope over time. On average test automation will make sense to you if you have at least 50 business processes in scope.
  • Test automation coverage (%).
    As mentioned below this percentage will highly depend on your business type. Consider the test that you want to keep manual and the ones that are not worth to automate to come up with the percentage of test automation coverage for all your business processes.
  • Releases per year.
    The more releases you have the more you’ll save with test automation. On average if you release at least 4 times a year, test automation can make a great impact on you.
  • Test cycles per release, including retesting for defects.
    How many test cycles do you run per release? Consider at least two in case you detect any bug, even the smaller or if you make any change, even the slighter.
  • The number of processes impacted during a release (%).
    Are your business processes highly dependent one from the other? Are the functionalities you release affecting the functionalities already existing? If that’s your cause the percentage is high and you’ll most benefit for test automation. A reasonable percentage of impact, based on our experience, is usually between 5 and 35%.
  • Cost of tester per hour.
    This is another figure which makes the difference in your overall result. If you have highly skilled testers you will greatly save for test automation.
    You need to consider that, after implementing test automation, you can have business people to create and run regression tests. It depends on the expertise required for using the test tool you’ve chosen. That’s the case for Qualibrate: no coding skills are required. This means that you can have your best-skilled testers to focus on manual testing critical functionalities.
  • Average time to create, execute and maintain a manual test
    Based on our customer experience, we know that with test automation you can reduce 80% time spent on manual testing., or even more. The more it takes to execute a manual test the more you will be able to save.

Cost for testing

All you have to do is applying the formula to the current scenario, the cost for manual testing, and take off the cost for test automation.

The time to create and maintain an automated test case highly depends on your business process and on the tool you are using. If you want to create a proper test case, which requires low maintenance, it will usually take the same time than creating it manually. What you’ll save is the time for maintenance, which will be pretty low, and the entire time for execution, as the test will run automatically.

Don’t forget to take into consideration that not all your business processes will be automated. Make sure you only apply test automation to the percentage of desired coverage.

Savings in testing - formula
 
2) Time saved for documentation and training

Don’t forget that you can benefit from implementing test automation tools in other time-consuming activities, often underestimated. That’s the case of Qualibrate: while recording your business process and saving all the user actions and input data on our cloud, you can use the stored information to create business process documentation and end-user training.

To calculate how much you save you will need to know the

  • cost of the business analyst, who usually creates documentation
  • cost of the training manager, that provides the onboarding manuals for new users
  • time to create and maintain documentation and training for business process

You can apply the same formula we used for testing. For documentation and training you will only consider the time to create and the time to maintain.
 
3) Reduction of risk and remediation costs

The biggest ROI of automated testing is the reduction of risk and remediation costs.
With test automation you will be able to hit a test coverage close to 100% and lower the risk to introduce a defect in the production environment. You will be able to find hidden defects earlier in the life cycle, that otherwise you would have only found into production, causing business disruption.

Business disruption means a loss. This loss needs to be considered in the ROI formula for test automation: we will calculate the gain in terms of money if a bug is discovered post delivery/implementation. So, in this case, the gain is the loss the company could face if automation testing was not implemented.

As you can imagine this is a figure that highly depends on your own business. Think of Amazon, or Booking: what would happen if their systems were not working, even for few minutes, and people from all over the world would not be able to buy anything on these two platforms. And how about banks, financial institutions, cryptocurrency trading platforms or ad networks? Introducing bugs into production would mean a big loss of money and, most of all, on credibility and trust.

Reaching a test coverage of nearly 100%, with test automation, means that you significantly lower the risk that this happens. How we calculate this “loss” gain then?

These are the questions you have to reply, first:

  • What defines risk in your company? For instance, a bank’s customer-facing application for conducting transactions cannot go offline for any significant period of time and must have rigid security protections.
  • How many people will be affected by a platform issue?
  • What is the likelihood of worst-case scenarios happening?
  • What is the potential market loss of application downtime?
  • What is the potential impact in the case of buggy software that annoys customers?

The difference between manual testing and test automation is the likelihood that bugs will be pushed into production. The likelihood, multiplied for the potential loss, will give you the saving.

Savings in risk reduction

Investment – factors

 
1) Tool license
The license cost is the most impactful investment. The vendors have different subscription plan. Make sure you don’t submit a long life subscription.

There is an increasing number of automated testing tools available on the market today, and choosing the one that is right for you can be difficult: check this guide we created to help you choose.
 
2) Resources for installation and configuration
When comparing the tools, make sure you are also aware of the time required for installation and. There are some famous solutions on the market requiring huge downloads and frequent updates.

Another important factor to be considered as an investment is the learning curve. How long does it take for your team to get familiar with the tool? How long will it take your team to fully ramp up and be comfortable executing tests with the product? Is it an easy-to-use tool for everyone, or will some team members need additional training?

Some automated testing tools rely heavily on scripting tests. We are not big fans of complex solutions in Qualibrate, that is why we created an online platform. All you have to do is to login with your credentials and you will be able to automate a business process in minutes, thanks to our friendly UI not requiring a single line of code. We just started a project in New Zealand where the customer was able to execute their first end to end regression cycle only after 3 weeks of starting from scratch.
 

Final formula

To sum up let’s see the initial formula again.

Test automation ROI formula

Where:

  • SAVINGS: savings in testing + savings in risk reduction and remediation costs
  • INVESTMENT: tool license + installation and configuration

Are you still struggling?

We hope this article helped you to understand how important it is to calculate the ROI for your test automation strategy.

If you are not sure about the result you came up with you can still double check it by filling our calculator or contact us. We will be more than happy to help you and share with you our expertise.

Leave a Reply