uTest holds quarterly crowdsourced competitions, probably to drum up publicity for its software testing business, and shares the results with vendors. Testers were judged on the “innovativeness” of the bugs, their understanding of the problems, and the quality of their reports. Rewards ranged from $400 for first prize to $50 for fifth place.
While Google earned the highest praise, almost a third of the testers were pleasantly surprised by Bing, and 10 percent said they’d make it their primary search engine, having tested Google and Yahoo as well. Still, the other 90 percent said they’d be sticking with Google, valuing its search accuracy above all.
Testers actually found the fewest bugs in Yahoo, with just 70 problems, though less participants focused on this departing search engine compared to the other two major search engines. Testers found 130 bugs in Google, with 8 percent classified as “showstoppers,” while 321 bugs were discovered in Bing, 14 percent of them showstoppers.
It’s not clear how uTest defines bugs, but they pertain to technical, functional, and user interface issues. Because known bugs don’t qualify, I’m guessing Google gets a pass here, as there could be well-reported problems that don’t show up in the contest results. By comparison, Bing is less than four months old, so there’s plenty of room for new problems to arise.
More interesting is the survey that uTest solicited from its testers after the contest. Money was also on the line here, with $400 going to whoever submitted the best feedback report.
Google dominated these surveys, as the top choice for accuracy, real-time relevance, and page load speeds. In all categories, Bing ranked second and Yahoo came in third. A small percentage of testers also tried Google Caffeine, and were generally impressed. One tester said results came back twice as fast as Google’s existing engine.
If the survey results are at all representative of all users, it’s a blow to Microsoft, which has tried to market Bing as turning up better search results. However, Microsoft may recognize it’s losing this battle, instead turning to new, flashy features such as visual search.