Do you believe in Magic Quadrants?

Gartner analysts say they have their trademarked Magic Quadrant down to a science. But plenty of vendors, gored by the quadrant’s twin axes of “completeness of vision” and “ability to execute,” insist it’s all just hocus-pocus.

Gartner analysts say they have their trademarked Magic Quadrant down to a science.

But plenty of vendors, gored by the quadrant’s twin axes of “completeness of vision” and “ability to execute,” insist it’s all just hocus-pocus.

Forum

Is the Magic Quadrant Bible or bunk? Add your thoughts

The idea of using a two-dimensional graph to summarize and visualize potentially complex values isn’t unique or new. Many of Gartner’s rivals use variations on the grid to evaluate vendors and technologies, and similar graphs are a staple of business schools.

But the Magic Quadrant – a term Gartner first used in 1992 and copyrighted in the late 1990s – has become a tool so widely known that it’s routinely parodied.

The Valley of the Geeks Web site (www.valleyofthegeeks.com) recently ran an updated and simplified “Gartner Magick Quadrante” that mapped a variety of celebrities, ranging from Bill Gates to Janet Reno, along with most of the characters from “The Lord of the Rings” and “Harry Potter” to two axes labeled “Payment to Gartner” and “Agreement with Gartner.” The quadrants are labeled Finders, Keepers, Losers, Weepers, which are pretty fair summaries of how many people read the Gartner graph.

In a story about “quadrant presentations” last year in the Venture Capital Journal, Dan Goodermote, a partner with ABS Capital Partners, illustrated how, and how easily, the idea is perverted. He showed a quadrant formed by a vertical axis from Evil to Good, and a horizontal axis from Incompetent to Expert. Not surprisingly, “My Company” ended up in the coveted top right panel – both Good and Expert.

Gartner goes to great lengths to dispel precisely this kind of image, at least among its clients. Over the years, says Gartner’s current research guru, Betsy Burton, the company has created a rigorous, standardized format and set of procedures, coupled with internal peer review, intended to ground the Magic Quadrant more on hard data and less on gut feelings.

The simple chart is the only visual relief in what is typically a six- to eight-page Magic Quadrant “Research Note,” periodically sent out by Gartner to its subscribers, which include technology companies, venture capitalists and other investors, and enterprise network and IT executives. The idea, Burton says, is to create a “picture of a specific market and where it’s going.”

A case in point is Richard Stiennon, Gartner’s research vice president, who spent weeks preparing a June 2003 Magic Quadrant on network firewall vendors. He followed what Burton says is a rigorous, standardized format that among other things spells out the detailed criteria, and their rationale, for judging a vendor’s “vision” and its “ability to execute.”

Stiennon collected a range of data, some quantifiable, some qualitative, from interviews with vendors and with Gartner’s end-user clients, which account for 62% of Gartner’s customers, according to the company. Much of the data is packed into spreadsheets, which generate the first outline of the new quadrant.

This time, Stiennon added a new, and critical, criterion: Did the vendor’s product do “deep packet inspection” of the network traffic? The change, in effect, directed attention at a just-emerging trend: the so-called application firewall.

“[Existing products] were doing nothing to stop the worms and the application attacks,” Stiennon says. “We saw a huge groundswell for a product that could go much deeper [than existing firewalls].”

Stiennon had to defend the addition, and his conclusions, before a peer group. Burton describes this as part of the company’s “Socratic method.”

The final step before publication is to circulate the conclusions to the vendors to catch any inaccuracies.

Stiennon’s quadrant triggered a minor earthquake in the firewall market when it was published. Of the nearly two dozen vendors, not one made it into the top right quadrant, where both execution and vision are highly rated.

“They said ‘No one can be a leader in this quadrant unless they have application firewall capabilities,'” says Andrew Stern, vice president of marketing for MagniFire, an Israeli firewall vendor with offices in New York. He adds that, incidentally, MagniFire was one of the first such vendors to claim that its software could do just that.

Most present and former Gartner analysts can recall, with a mixture of dread and survivor pride, calls from outraged vendors, some of whom counterattacked with screams, others with PowerPoint slides.

“If you downgraded someone, as soon as you published it, you knew you were going to be in for a week from hell,” says Skip MacAskill, a former Gartner analyst in the mid-1990s, and now vice president of marketing with Marconi. For him, covering the LAN market, it was always 3Com. “They were extremely aggressive and always willing to escalate things,” he says. “When I got a call from [3Com’s then-CEO] Eric Benhamou, I knew they had exhausted all their other options.”

Other vendors attack the basis of the quadrant – the complex structure that Burton says is the Quadrant’s great strength.

“The solutions [by the vendors] may be equivalent,” says a marketing manager at a privately held wireless LAN (WLAN) vendor who asked not to be named. “But how the vendor answers the questions and how [those answers are] interpreted can affect which quadrant you end up in.”

The executive notes that privately held companies don’t disclose a whole range of financial data that is critical for determining whether a company in fact has the “ability to execute.” “[Quadrant placement is] based purely on hype and speculation,” he says.

Not so, counters Gartner Vice President Ken Dulaney.

“The key message is that we are trying to assess more than technology, because we believe that all aspects of a company must have both vision and execution to succeed,” he wrote. Gartner analysts issue two groups of questions, one about “vision” and the other about “execution,” at five departments inside a vendor: finance, marketing, operations, sales and technology. Some of the questions are in questionnaires, others are aske face to face.

The simplicity of a graph that isn’t actually plotted according to mathematical values means it can be read in many ways, often far different from the way Gartner intends.

“It oversimplifies things, but that accounts for its effectiveness,” MagniFire’s Stern says. “It’s become the ‘two thumbs up’ of the IT world.”

Another executive, requesting anonymity, put it even more succinctly.

The Magic Quadrant has such a grip on the imaginations of venture capitalists and others “because it’s an easy way for them to ignore doing all their own homework,” he says. “If you’re up in the right hand corner, you’re good, and if you’re down in the bottom, you’re a bunch of schmucks.”