It’s been a little more than a decade since then-rookie analyst Rob Enderle penned a five-year forecast of the operating system market that raised the ire of vendors and colleagues. His report forecast a dramatic drop-off of the Macintosh operating system, along with declines in NetWare and Unix. …
It’s been a little more than a decade since then-rookie analyst Rob Enderle penned a five-year forecast of the operating system market that raised the ire of vendors and colleagues. His report forecast a dramatic drop-off of the Macintosh operating system, along with declines in NetWare and Unix.
“That was the year that every CEO from the head of HP to the head of Sun wanted me to find a new career in toilet cleaning, hopefully someplace where the toilets were particularly nasty,” recalls Enderle, who is now president and principal analyst at Enderle Group. “I don’t think there was a month that went by when I didn’t think someone would fire my butt.”
Along with angering vendors, the report prompted one of Enderle’s colleagues at the time to proclaim him an idiot, in a Wall Street Journal story, for saying Macintosh users would ever migrate away from their beloved platform.
While many of the shifts Enderle predicted came to pass, the report was not completely on target, Enderle says. “I was too high on the adoption of Windows 95 – I was too high by a lot.”
Just as it’s easy to point the finger at weather forecasters when unexpected atmospheric conditions dampen a day, IT analysts take abuse when expected market dynamics don’t pan out. The nature of the discipline, combined with the shaky relationship between analyst firms and IT vendors – which are both market sources and research sponsors – add to the challenge analysts face.
“No question that no matter what anybody says, this is for the most part more of an art than a science, doing forecasting,” says Jeff Kaplan, managing director of Thinkstrategies.
“Especially when you look at segments that are for the most part supported by vendors and service providers who don’t have to report specific revenue or shipment figures by product or service line,” he adds.
Kaplan was on the number-crunching, market-sizing side of research before founding consulting research firm Thinkstrategies, which isn’t in the business of quantifying markets.
The art of market research involves taking on faith the estimates vendors provide, correlating those with customers’ stated buying intentions, and factoring in macroeconomic and geopolitical issues, Kaplan says. “That composite picture represents a best effort at charting a path for where a market might go,” he says.
But even that often isn’t enough. In 1994, Enderle forecast the market for Win 95 would hit $40 million in its first year, but it only reached about $24 million. “It just never crossed my mind when I did that prediction that Microsoft would have this big launch party and then stop marketing. Sales dropped off a cliff,” he says.
Earl Perkins, an analyst in the security and risk strategies group at Meta Group, finds two extremes in analyst tactics. There are the really meticulous, military types who gather data like squirrels gathering nuts for winter, Perkins says. “They’re really not satisfied until their nest is so full of nuts, they can’t climb in the nest. But if you do that, the event has already passed. So then here you are a historian rather than a predictor,” he says.
At the other extreme are analysts who read one thing, talk to one person and make a guess, Perkins says. Meta Group – which Gartner is in the process of acquiring – looks to operate in the middle, balancing data with insights gleaned more from customers than vendors, he says.
When Sun acquired Waveset Technologies in 2003, Perkins predicted Waveset would disappear, and it didn’t. “I thought they would pull a typical Sun, gut the company, take the technology and let everybody walk,” Perkins says. Instead Sun deviated from tradition and formed a division, he says.
The best way to respond when a prediction doesn’t bear out is to own up, Perkins says. “I go back and say I was wrong,” he says. “Customers prefer people who are honest and admit their mistakes.”
Getting it right
Coming up with predictions is a collaborative process, says Jasmine Noel, a principal analyst at Ptak, Noel & Associates. “I never try to sit in a room by myself and come up with these things,” she says. “To me, it’s the conversations with people outside of my area of expertise that can really help predictions.”
She’s learned over the years that predictions have to be focused to be realistic. A couple of years ago Noel predicted application management tactics would change dramatically in 2005 to handle the influx of smaller, more modular applications built on top of Java 2 Platform Enterprise Edition and .Net platforms.
But slower-than-expected adoption of new application platforms has lessened the need for a new management approach. While industries such as financial and retail, which modify applications frequently, have adopted new development methods, other industries with more static application environments haven’t seen the need to, Noel says. “I didn’t take into account the fact that people’s budgets would still be this tight,” she adds.
Tempering predictions and not expecting all industries to behave as one is critical, Noel says. “The entire computing industry is not uniform,” she says.
Looking ahead, Noel’s predictions – such as the convergence of configuration management and performance management technologies – are predicated on a series of interconnected developments in the broader IT industry.
“It took me a while as an analyst to recognize that these other things have to happen in order for predictions to come true,” Noel says. “Analysts as an industry have gotten better at doing that. Fewer people are making predictions that are just way out there.”
Sometimes being a responsible analyst means being patient. Thomas Nolle, founder and president of CIMI, refused to forecast the market for VPNs or VoIP technologies until 2002. There simply weren’t enough buyers and therefore enough data available to make a legitimate forecast, he says.
Over the past 20 years, Nolle, a former computer programmer, has honed a computer forecast model he built that simulates buyers’ technology-purchasing decisions using econometric indicators and a base of 20-plus years of survey data.
His reports aren’t always popular because they don’t tend to over-inflate market estimates, Nolle says. “The truth is frequently not tremendously interesting,” he says.
In the past, not all research firms were as diligent about trying to weigh all the appropriate factors as they needed to be, Kaplan says. “This industry has been plagued for many years by analyst firms that are market makers. That is, they’re trying to create high-growth markets that they can follow and advise people about,” he says. “Exaggerated forecasts have been the reason for so many disappointments in this industry and actually a lot of company failures.”
Firms that are in the business of forecasting market sizes have a lot of credibility to regain and as a result are reassessing their forecasting methodologies, Kaplan says.
“They’ve all been forced to become much more conservative, because people do blame them for over-expectations that took place during the Internet boom and bust,” he says. “They were part of the hype machine that unfortunately took a lot of customers as well as vendor organizations over a cliff.”
Regaining trust of IT users and vendors isn’t something that will happen overnight. “Most people I know both on the user and the vendor side don’t have a lot of faith in forecasts anymore,” Kaplan says. “They’re kind of a necessary evil.”
