Microsoft Subnet An independent Microsoft community View more

Are Bing it on challenge claims a bunch of bunk?

Microsoft fires back after a Yale Law professor said Bing It On challenge results are deceptive.

You've seen the Bing It On "blind taste test" search engine commercials in which users decide if they favor the results from Bing or from Google. But what would life be without drama?

Bing vs Google for government shutdown search

When the Bing It On campaign launched in September of 2012, Bing claimed people prefer Bing 2:1 over Google. That was apparently too much for Yale Law professor Ian Ayres to swallow. In fact, it aggravated him so much that he dug into the claim. He wrote on Freakonomics, "I was slightly annoyed to learn that the 'nearly 2:1' claim is based on a study of just 1,000 participants. To be sure, I've often published studies with similarly small data sets, but it's a little cheeky for Microsoft to base what might be a multi-million dollar advertising campaign on what I'm guessing is a low-six-figure study."

Ayres, along with four Yale Law School students, conducted their own Bing It On taste-test challenge before publishing their conclusion "A randomized experiment assessing the accuracy of Microsoft's 'Bing it on' challenge claims [pdf]."

"We found that, to the contrary of Microsoft's claim, 53 percent of subjects preferred Google and 41 percent Bing (6 percent of results were 'ties')."

We also interjected a bit of randomness into our study to test whether the type of search term impacts the likelihood that Bing is preferred. We randomly assigned participants to search for one of three kinds of keywords: Bing's suggested search terms, popular search terms, and self-suggested search terms. When Bing-suggested search terms were used the two engines statistically tied (47% preferring Bing vs. 48% preferring Google). But when the subjects in the study suggested their own searches or used the web's most popular searches, a sizable gap appeared: 55-57% preferred Google while only 35-39% preferred Bing. These secondary tests indicate that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results.

The group concluded, "Google has a colorable deceptive advertising claim against Microsoft."

Matt Wallaert, Behavioral Psychologist for Bing, fired back in defense of Bing It On. He noted that Bing later morphed its claim into, "Wherever we go, people prefer Bing over Google for the web's top searches," and points to several posts that document the changes.

Next, Wallaert addressed the small sample size of 1,000 people to obtain the results. "A 1,000 person, truly representative sample is actually fairly large. As a comparison, the Gallup poll on presidential approval is just 1,500 people."

Ayres is irked that "Microsoft's ads encourage users to join the millions of people who have taken the challenge, but it will not reveal whether the results of the millions are consistent with the results of the 1,000." But Wallaert also seems irked, claiming that there is no way Bing can post those numbers because it does not track the results. He wrote:

The answer here is pretty simple: we don't release it because we don't track it. Microsoft takes a pretty strong stance on privacy and unlike in an experiment, where people give informed consent to having their results tracked and used, people who come to BingItOn.com are not agreeing to participate in research; they're coming for a fun challenge. It isn't conducted in a controlled environment, people are free to try and game it one way or another, and it has Bing branding all over it.

So we simply don't track their results, because the tracking itself would be incredibly unethical. And we aren't basing the claim on the results of a wildly uncontrolled website, because that would also be incredibly unethical (and entirely unscientific).

Ayres' final issue is the fact that the Bing It On site suggests queries you can use to take the challenge. He contends that these queries inappropriately bias visitors towards queries that are likely to result in Bing favorability.

First, I think it is important to note: I have no idea if he is right. Because as noted in the previous answer, we don't track the results from the Bing It On challenge. So I have no idea if people are more likely to select Bing when they use the suggested queries or not.

Have you tried the Bing It On challenge? You run five different searches, with the results divided into Bing and Google. You can choose the results on the right, or on the left, or "I can't decide, it's a draw." If you can't decide, maybe you should DuckDuckGo?

Like this? Here's more posts:

Follow me on Twitter @PrivacyFanatic

To comment on this article and other Network World content, visit our Facebook page or our Twitter stream.
Must read: Hidden Cause of Slow Internet and how to fix it
Notice to our Readers
We're now using social media to take your comments and feedback. Learn more about this here.