Skip Links

Research buries Microsoft’s Bing-vs.-Google claim

UPDATED: Yale economist finds nothing to support advertising that says people prefer Bing 2-to-1

By Paul McNamara on Wed, 10/02/13 - 11:12am.

bing

Microsoft's claim that people prefer Bing over Google by a 2-to-1 margin in blind taste tests always smelled fishy, but now a Yale professor and his students have shown it to be pure marketing malarkey by conducting their own survey ... and they did it using Microsoft's tool of choice.

Yale professor Ian Ayres, a lawyer and economist, writes on the Freakonomics blog:

So together with four Yale Law students, I set up a similar-sized experiment using Microsoft's own BingItOn.com site to see which search engine users prefer.  We found that, to the contrary of Microsoft's claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were "ties").  This is not even close to the advertised claim that people prefer Bing "nearly two-to-one."  It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who've taken the Bing-It-On challenge, if, as in our study, the millions of people haven't preferred Bing at a nearly a 2:1 rate.  ...

ayres

We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred.  We randomly assigned participants to search for one of three kinds of keywords: Bing's suggested search terms, popular search terms, and self-suggested search terms.  When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google).  But when the subjects in the study suggested their own searches or used the web's most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing.  These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results.

I've asked Microsoft's public relations firm for a comment. (Got it. See update below.)

Matt Cutts, head of the web spam team at Google and one of the company's high-profile bloggers, had a subdued reaction to the research by Ayers, writing on Google+:

I have to admit that I never bothered to debunk the Bing It On challenge, because the flaws (small sample size; bias in query selection; stripping out features of Google like geolocation, personalization, and Knowledge Graph; wording of the site; selective rematches) were pretty obvious.

On the other hand, Danny Sullivan at Search Engine Land sees more here than shady advertising practices, he sees a bright side for Microsoft:

There are two potential, contradictory reactions to the Ayers study:

  • It either conclusively or largely disproves the Bing preference claims;
  • Putting aside the Bing advertising claims, the search engine performed relatively well vs. Google.

Google won 53 percent of the time and Bing won 41 percent of the query tests, with a tie in 6 percent of instances. That suggests that Bing has the capacity to gain much more market share than it currently has (67 percent vs. 18 percent).

Ayers points out that the more assertive "prefer Bing 2:1″ claim has been replaced on the Bing It On website with the more limited claim that "people prefer Bing."

So they've replaced a whopper with a fib.

(Geek-Themed Meme of the Week Archive)

(Update: Just received a response from Microsoft. "The professor's analysis is flawed and based on an incomplete understanding of both the claims and the Challenge," says Matt Wallaert, a behavioral scientist for Bing. "The Bing It On claim is 100% accurate and we're glad to see we've nudged Google into improving their results.  Bing it On is intended to be a lightweight way to challenge peoples' assumptions about which search engine actually provides the best results. Given our share gains, it's clear that people are recognizing our quality and unique approach to what has been a relatively static space dominated by a single service.")

Welcome regulars and passersby. Here are a few more recent buzzblog items. And, if you’d like to receive Buzzblog via e-mail newsletter, here’s where to sign up. You can follow me on Twitter here and on Google+ here.