Bing it on! Yale professor calls shenanigans over Microsoft's search superiority claims

It’s no secret that Microsoft’s Bing search engine doesn’t get much love compared to Google. So to help poor Bing snag a few more users, Microsoft set out to show that its search product was just as good—if not better—than big bad Google. Given the chance in a Pepsi Challenge-styled blind test, Microsoft says, people actually prefer Bing over Google.

And so was born the Bing It On campaign, based on a Microsoft study of “nearly 1,000 participants” who ended up preferring Bing over Google two-to-one in blind tests. To drive home its message, Microsoft embarked on a major advertising campaign and set-up a website called BingItOn.com where anyone could take the Bing challenge.

Microsoft’s claims were entertaining, but they didn’t sit well with Ian Ayres, an economist and professor of law at Yale. On Tuesday, the good professor cried shenanigans against Microsoft.

“The 2-1 claim seemed implausible,” Ayres wrote on the Freakonomics blog. “I would have thought…it would be hard for people to distinguish between the two sets of results, much less prefer one kind 2:1.”

To find out if he was right, Ayres tested 1,000 Mechanical Turk workers—a similar sample size of Microsoft’s original Bing study—to see if he could duplicate Microsoft’s results using the Bing It On site.

Binging it on

As any scientist in the budding (and totally made up) field of “searchology” knows, the results of any experiment cannot be taken as fact unless they can be duplicated. So can we say with absolute fact that people prefer Bing or Google? Do we finally have enough data to create the Big Bing Theory?

Not according to Ayres, who found that most people, when left to their own devices, actually prefer Google over Bing. He did find that when people searched for a Bing-suggested search term on the Bing It On site, preferences for Bing and Google results were about even. But when people could use their own search terms or use terms pulled from a list of the Web’s most popular searches… it was the big “G” all the way.

Ayres argues that the second part of his study “indicate[s] that Microsoft selected suggested search words that it knew were more likely to produce Bing-preferring results.”

Ayres also suggests throughout the blog post that Microsoft has been misleading with its Bing It On ads. The problem, says Ayres, is that Microsoft has not released the results of the more than 5 million people who have taken the Bing challenge on BingItOn.com to see if they line up with Microsoft’s original study.

“It is misleading to have advertisements that say people prefer Bing 2:1 and also say join the millions of people who’ve taken the Bing-It-On challenge,” Ayres wrote. “If, as in our study, the millions of people haven’t preferred Bing at a nearly a 2:1 rate.”

Ayres goes even further arguing that “Google has a colorable deceptive advertising claim against Microsoft” that could be worth a lot of money.

Binging it on redux

Well, Microsoft is not a company willing to take allegations of legal liability lying down—especially when it comes to Google. The company returned fire shortly after Ayres post was published. Do the results of the millions of people who have taken the Bing challenge line up with Microsoft’s research?

Here's the thing: Microsoft has no idea. The company didn’t track user results on the Bing It On site, because that would have been unethical, Bing Behavioral Psychologist Matt Wallaert wrote on the Bing Search blog.

What? An online company didn’t track user data over ethics? What madness is this? “People who come to BingItOn.com are not agreeing to participate in research; they're coming for a fun challenge,” Wallaert wrote. “It isn't conducted in a controlled environment… So we simply don't track their results, because the tracking itself would be incredibly unethical.”

Wallaert then goes on to take a few more shots at Ayres’ findings and even suggests that Ayres results could—could—prove that Bing is better than Google at providing results for current news topics.

Binging it all together

While warring blogs are always good for a laugh, the reality is that whether you prefer Bing or Google really comes down to personal preference—blind taste test or no. Nobody searches in a vacuum of little blue links anymore, there all kinds of reasons to prefer one search engine over the other including interface elements, mobile features, social networking integration, privacy concerns, and location-specific results. (Bing, for example, just got a Modern makeover and smarter search results.)

Nevertheless, if you’re interesting in becoming a searchologist yourself to find out if there’s some ultimate truth in search then check ou tAyres’ study. Don’t forget to also read up on Microsoft’s methodology for its Bing tests.

Subscribe to the Today in Tech Newsletter

Comments