ROI of Automation Effort

Automation effort, as any effort in software production process must be analyzed according to business criteria, in many cases the ROI is the first one. Is it worth ? Well, there is no single answer to that, or single formula. What metrics and formulas can be used in calculating ROI of an automation effort ? The simplest formula can be:

Em = Time it takes to execute the cycle manually

Mm = the sum time of updating the manual testing documentation after each run, per cycle

N = number of cycles that need to be executed

Au = Time it takes to automate the cycle

Ea = time it takes to execute full cycle automatically

Ma = time it takes to modify the automatic framework on changes between cycles.

So, according to time factor, automation is suitable if: N * (Em+Mm) > Au + N * (Ea+Ma)

But still this formula doesn’t give us a direct answer, weather to do automation or not. The next factor that should be tested is Cost. Let’s take the regression task for example:

Pm = regression cost on creating and executing manual testing (include number of workers, working hours, working place and terms) Pmu = update regression cost of manual testing Pa = regression cost of automating the cycle (tools price, developer working hours and terms, automation station cost) Paex = regression cost of executing the cycle Pau = update regression system between cycles (developer hours, automation station cost ) So according to cost criteria, automation is worth if N * (Pm + Pmu) > Pa + N * (Paex + Pau)

Since much of the Automation frameworks is done, when lifting up the automation platform, the ROI is even getting better on the next releases. What about testing the automation ROI against finding defect criteria ? Well I believe that it will be possible, only if the testing strategy is to test it all. Meaning full 100% coverage, of the new and old features, and since most of the software companies are working on testing according to probability chances, Risk Based Testing, most of the added value of the Automation system is to verify that the old features were well protected and not damaged because of the new features. Regression testing, Acceptance testing and verification testing cannot be measured by finding defect criteria. More advantages that were not taken in the previous formula:

1. When having Automation capabilities, you activate them much more often, making sure quality remains after every change.
2. Faster feedback for the developers teams, sometimes means less defect and much more faster manufacturing process.
3. Higher overall testing capabilities even means that developing process become more object oriented, more modular, making sure that every change doesn’t reflect other existing features. Developing process gets much more improved and efficient.
4. Manual test switch their focus only on new features and edge situation testing which will cause the product quality level to jump much higher.
5. Manual testers will catch much more defects, QA work will be much more efficient.
6. Return defects level will be much less, or defects caused during work to fix others will be much less, I call it Backlash work, and there is even Backlash work on second level and third.
7. Human resource costs are dropping down, even if you retain your manual testing power, you will need it to be more professional on the new features, and less on the overall product structure, learning curve will be dropped and getting professional will be much more faster.

Summary You can start with the simple RIO formula, but then you have to take in consideration all the other criteria according to your product, service and targets. You will have to accumulate your product values and targets into numbers, for example: How much it is worth to you to be first in quality level ? How much it is worth to you to be first and fast into market ? How complicated is your product, can you keep the high quality with profession less resources.? Are you thinking about using Outsourcing for development or testing or both ? etc. The answers for these questions are changing from one manufacturing process to another, and have a huge influence on the ROI that you will have on your Automation platform.