Microsoft report prompts Forrester policy change

Forrester has changed its policy toward vendor-sponsored research following last month’s publication of a controversial Microsoft-funded study that compared the cost of developing applications on Linux and Java to a Microsoft-based approach.

Forrester Research has changed its policy toward vendor-sponsored research following last month’s publication of a controversial Microsoft-funded study that compared the cost of developing applications on Linux and Java to a Microsoft-based approach.

The policy change was announced in a letter written by George Colony, the CEO of the Cambridge, Mass., company, and posted to the Forrester Web site late last week.

“We will no longer accept paid for, publicized product comparisons,” Colony said in an interview. “The best example of that would be the Microsoft report.”

The Microsoft-funded study found that developing certain types of Web applications was 40% more expensive when Java and Linux were used rather than Microsoft software.

The survey was widely seen as a blanket statement about the cost of ownership of the platforms, rather than a more qualified statement about their relative costs for running certain types of applications, according to Forrester analyst John Rymer, the author of the report.

“There was a huge outcry about the Microsoft study,” said Rymer, who blamed the way the media covered the report for much of the criticism. “‘Microsoft cheaper than Linux:’ That was the basic headline. There were a dozen variations on that. Obviously, if you read the report, the conclusions in the report are much more qualified than that,” he said.

In addition to Microsoft having funded the study, critics also took exception to the small sample size (12 companies) that Forrester’s results were based on, as well as the study’s methodology.

With the policy change, Forrester will still do product comparisons for vendors, Colony said, but it will not allow them to publicize the results, as was the case with the Microsoft-funded study, as well as a recent customer satisfaction study funded by Peoplesoft Inc. that was also released in September.

Both studies were promoted by the vendors’ respective public relations departments. In Microsoft’s case, the study was offered as the latest data point in the company’s “fact-based” campaign against Linux, an initiative promoted by Microsoft’s general manager of platform strategy, Martin Taylor.

“Martin Taylor went out and visited with a bunch of reporters, and he was referring to the study and using it to advance his case that Linux doesn’t have a lot of advantages,” said Rymer. “George (Colony) was uncomfortable with this.”

Forrester stands behind the results of the study, but the company made a mistake in allowing it be publicized, Colony said. “It just looks bad. It’s a conflict of interest,” he said.

Paul Onnen, chief technology officer at WebMD in Elmwood Park, N.J., said that he had no problem with the findings of the Microsoft study, which he found to be “relatively fair.” But, he added, he generally doesn’t give much credence to vendor-sponsored reports. “I am, in general, very leery of sponsored research because there’s lies, damned lies and statistics, and you can make statistics look however you want them to,” he said.

“What we need to do in the IT community is stop pointing fingers between the Java community and the Microsoft community,” said Tim Brown, a systems architect with AirNet Systems in Columbus, Ohio. Brown believes a comparison of total cost of ownership (TCO) is so difficult to accurately quantify that research firms would be better off steering away from these sort of studies. “It’s really hard to prove how one operating system is better from a TCO perspective,” he said.