Survey shows skepticism on ROI calculators

Recently we discussed Enterprise Management Associates’ approach to analyzing return on investment and invited readers to participate in an ROI study. Today we’ll share some of the findings and reveal the winner of the promised prize.

We asked participants whether they trusted the ROI calculators provided by vendors. Seventy-nine percent said they don’t. When we categorized the data by management level, the numbers were even more striking. Every one of the executives said they don’t trust vendor ROI calculators, while 87% of midlevel management and 33% of low-level or technical management said they don’t trust such calculators.

When asked about the use of ROI, there was a general consensus that ROI is a valuable – in fact, nearly essential – tool for investment decision-making. It is just the ROI analysis of vendors that is suspect. This actually conforms pretty closely to other research EMA has conducted and reinforces a conclusion we have made in the past: ROI is not a simple metric and should not be used as such.

ROI, in fact, is sensitive to the environmental conditions in which it is computed or predicted. For executives that are leery of calculators that oversimplify the environment and produce extraordinary ROI figures, this research provides some vindication. To trust an ROI calculator, you must trust that the calculator is focused on the right metrics and that those metrics are consistent with your own IT environment. Otherwise, the old saying of “Garbage in, garbage out” applies.

Calculators (EMA prefers the term “estimators”) should be used only as a component of ROI analysis. A salesperson should be able to clearly articulate how ROI is generated and should be able to show how the calculator maps to the buyer’s environment. If the salesperson doesn’t have a clue what ROI means aside from how it is spelled, that should be a warning sign that the ROI may be superficial or that the process is gamed to produce only positive ROIs. One test of this is to see if a less-than-stellar ROI can be generated at all. If the conditions for low to no ROI are at an unreasonably low threshold, that is a warning sign as well. The bottom line is that ROI collateral, like any other kind of collateral, can be an asset or a liability to decision making: skepticism is a good thing to apply to any of it.

And now, the winner of our contest: John Sitek. He receives a month of EMA analyst access and the ROI study that we referenced in this column. John, drop us a line by e-mail and we will set you up.