Business value can’t be measured by ROI analysis alone

While 85% of respondents in an Enterprise Management Associates survey earlier this year indicated that they rely on return on investment calculations to measure value from technology, 78% said they don’t trust ROI calculators provided by vendors.

This admittedly might be a dumb question, but why is this? Many probably suspect that vendors go out of their way to tweak or even misrepresent data, but this isn’t generally the case. Many vendor calculators aren’t dishonest so much as, to be blunt, ridiculous. Yet if the math is good, the data sound, the samplings justified – and often done so through outside analyst firms – how can this be?

One answer might be that ROI is itself an overly simplistic concept. But does that mean that other forms of measurement are better, such as net present value or total cost of ownership? No, in my opinion. While a mix of these and other approaches will naturally provide more shading, too many calculations – with too much of a pretension to detail – will not, in the end, bring much more enlightenment.

The issue with a lot of these approaches is that they don’t necessarily measure what counts – and understanding what really counts for a given organization is far from trivial.

In a way, ROI and business impact analyses have issues that are similar to what we found in early service-level agreements. SLAs often targeted nearly pointless technical metrics that were chosen primarily because they were easy to quantify. Having SLAs (as EMA has observed in customer environments) around “buffer miss ratio,” “CPU utilization” or “allocation failures,” is as ridiculous as many ROI calculations. A reduction in trouble tickets is clearly valuable, but understanding its true value in context with broader organizational and operational efficiencies – not to mention improved service – is a more complex dynamic.

I’d like to suggest that the first place to start with any business impact assessment is with the first word: business. What is the value of a technology investment in terms of real business goals? How and why is the IT organization and the business trying to evolve?

It may be that improved collaboration is the biggest requirement. And while many technologies may enable this, measuring “improved collaboration” per se is never going to be perfectly scientific. On the other hand, I would argue that to discard it would be to miss the point.

There are ways to integrate easily quantified value with more subjective – but possibly far more important metrics – through modeling. The result isn’t a ridiculous six-digit calculator, but a multi-dimensional view of business impact that allows the buyer to see and measure value – even if it’s not empirically perfect.

Timing – that is, getting the technology most needed at the right time to help an organization evolve – is an example of something that’s valuable yet generally non-quantifiable. Imagine an entrepreneur introducing an online shopping capability for office supplies and computers. The entrepreneur’s IT shop develops an application that allows a way to evaluate everything from furniture to computers – but it is a complex application that runs the risk of responding slowly and discouraging buyers. Superior management analytics for optimization and preventative diagnostics ultimately enable this entrepreneur to deliver this app with a competitive response time. The result is many millions in revenue and the first-phase success of a new business.

So what’s the ROI on the relevant management products? With timing, objectives and full business impact taken into account, it’s priceless.

What I’m advocating isn’t a single system for calculating value, but a dual approach. Model context and value without worrying about Excel first. Then apply the Excel metrics that are measurable and place them within that context. The result will be something of a hybrid that tries to reconcile two fundamentally irreconcilable worlds. However, it is in the tension between these two approaches that real insight into the value of an investment can be achieved.

What do you think? I welcome your examples and your thoughts.