Search Engine Shoot-Out
How We Tested the Search Engines
We tested a total of 55 services in six categories: general (text info), video, mobile/local, news, images, and blogs. We conducted our tests over three weeks. We used ten terms in each category. For general text-information searches, for instance, our categories included technology, pop culture, research/academic topics, books, and travel/shopping. In each category we posed what we considered an easy query and a difficult one: Just about every search engine could find the 2007 Academy Award winners, for example, but only a few could locate a portrait of Italian poet-philosopher Giambattista Vico. We tested the engines at different times of the day--morning, afternoon, and night--on all seven days of the week. (However, the time of day and the day of the week had no impact on the results.) We occasionally changed our queries, particularly in the news and blog categories, to keep them timely. For instance, our easy pop-culture query changed from "Academy Award winners" to "2007 NCAA basketball tournament teams" later in testing.
We performed all of our testing on the same Windows system, a 2-GHz Celeron desktop with 512MB of memory and a 1.5-megabits-per-second (download speed) AT&T/Yahoo DSL Internet connection. We didn't test for query speed, but this typically varied by only fractions of a second. Among the most-popular search sites, we saw little difference in between the time we pressed <Enter> and the appearance of the search results. However, we did detect significant differences among the blog-search engines, with some smaller sites--such as Best of the Blogs--taking several seconds to post the results. For the local-search tests, we used a Nokia E62 smart phone.
Chart: Our Test Queries and Target Responses
We asked ten queries in each of six categories: text information, video, images, local information, news, and blogs. Within each category we sought both an easy and a difficult target in five different topics or subjects. In some instances our target was a specific Web page; but unless noted in the chart, any site that provided the target response was considered to have given a correct answer. For topical searches we changed the queries in each of the three rounds of testing. Click the icon below to see our complete list of test queries and the results we wanted for each.
| Search-Engine Shoot-Out: Our Test Queries and Target Responses | ||
|---|---|---|
| We asked ten queries in each of six categories: text information, video, images, local information, news, and blogs. Within each category we sought both an easy and a difficult target in five different topics or subjects. In some instances our target was a specific Web page; but unless noted below, any site that provided the target response was considered to have given a correct answer. For topical searches we changed the queries in each of the three rounds of testing. | ||
| Text-information search | Query | Target response |
| Technical (easy) | WiMax speed and range | 40 mbps per channel in a typical cell radius of 3km to 10km, as stated on the WiMax Forum |
| Technical (difficult) | Linksys WPA2 | Yes, the Linksys WRT54G wireless router does support WPA2 |
| Pop culture (easy) | First round: 2007 Academy Award nominees Second round: 2007 Academy Award winners Third round: 2007 NCAA basketball tournament teams |
First round: The award nominees in each category Second round: The award winners in each category Third round: The NCAA basketball tournament field of 64 teams |
| Pop culture (difficult) | Tigers may 1 | The Detroit Tigers play the Baltimore Orioles at Comerica Park on May 1, 2007 |
| Research/academic (easy) | Capital of Moldova | Chisinau (Kishinev) |
| Research/academic (difficult) | weight gnu yak | Male yaks weigh 1100 to 2645 pounds (500 to 1200 kilograms), and females weigh 660 to 770 pounds (300 to 350 kilograms); a gnu (a blue one) weighs 500 to 600 pounds (235 to 275 kilograms) |
| Book excerpt (easy) | excerpt Doctorow The March | An excerpt from the E.L. Doctorow book |
| Book excerpt (difficult) | No Exit | An excerpt from the Jean-Paul Sartre play |
| Travel (easy) | 2007 All Star Game | San Francisco's AT&T Park on July 7, 2007 |
| Travel (difficult) | Sanibel Beach | The location of Bowman (or Bowman's) Beach on Sanibel Island in Florida |
| Video search | Query | Target response |
| Internet video (easy) | Colbert speech 2006 White House Correspondents Dinner | The full 24-minute, 10-second video of Stephen Colbert's roast of the president |
| Internet video (difficult) | Food for Thought DeLuca | Food for Thought; short |
| Television clip (easy) | Samurai Delicatessen | Saturday Night Live skit starring John Belushi and Buck Henry |
| Television clip (difficult) | Gilda Radner | Saturday Night Live skit starring Gilda Radner and Steve Martin dancing |
| How-to video (easy) | how to carve a turkey | Several potential correct answers, one of which is Martha Stewart's instructions |
| How-to video (difficult) | upgrade notebook drive | Any video showing step-by-step instructions for installing a new hard drive in a notebook PC |
| News (easy) | First round: tornado in Enterprise, Alabama Second round: Australian cyclone Third round: Valerie Plame congressional hearing |
First round: Tornado in Enterprise, Alabama Second round: Australian cyclone Third round: Valerie Plame at congressional hearing |
| News (difficult) | First round: Dodgers home opener Second round: Chatsworth Third round: big rig 101 |
First round: "LA Dodgers Sell Out Home-Opener Tickets" Second round: "HazMat Team Called to Chatsworth" Third round: "Big Rig Crashes Through Center Divider on 101" |
| Animation (easy) | Gopher Broke | Nominee for 2005 Academy Award for animated short subject |
| Animation (difficult) | Maestro | Hungarian artist Geza M. Toth's Academy Award-nominated short |
| Image search | Query | Target response |
| Portrait (easy) | Rutherford B. Hayes | The nineteenth president of the United States |
| Portrait (difficult) | Vico | Italian poet-philosopher Giambattista Vico, 1668-1744 |
| Building (easy) | Pantheon | The Pantheon in Rome |
| Building (difficult) | Lincoln Hall Kentucky | Lincoln Hall, Berea College, Kentucky |
| Thing (easy) | platypus | The marsupial |
| Thing (difficult) | windform | A 20-foot horn |
| Place (easy) | Times Square | The New York City landmark |
| Place (difficult) | Asilomar | Asilomar State Beach, Pacific Grove, California |
| Event (easy) | Mardi Gras | The New Orleans Mardi Gras celebration, 2007 |
| Event (difficult) | Keeneland | Keeneland Racetrack (the building, not the horses or the track itself) |
| Local search | Query | Target response |
| Restaurant (easy) | Spago | The address and telephone number of the popular restaurant in Beverly Hills, California |
| Restaurant (difficult) | Chili My Soul | Information for the obscure restaurant in Encino, California |
| Gas station (easy) | gas station 91301 | The nearest station to the test location of Agoura Hills, California |
| Gas station (difficult) | cheap gas 91301 | The lowest-priced gas near the test location of Agoura Hills, California |
| Person's telephone number (easy) | [Author's own number] | The author's telephone number |
| Person's telephone number (difficult) | telephone Bill Smith Thousand Oaks | The telephone number of Bill Smith of Thousand Oaks, California |
| Driving directions (easy) | Agoura Hills to Spago in Beverly Hills | Directions for a short trip using mostly freeways from Agoura Hills to Spago in Beverly Hills |
| Driving directions (difficult) | Agoura Hills to Zuma Beach in Malibu | Directions for a longer trip using mostly surface streets and/or exurban roads from Agoura Hills to Zuma Beach in Malibu |
| Event/show/news (easy) | L.A. Lakers tickets | Any reputable ticket vendor |
| Event/show/news (difficult) | Calabasas election | March city election results for Calabasas, California |
| News search | Query | Target response |
| Technical (easy) | First round: RINBOT Second round: EU iTunes Third round: Microsoft antivirus |
First round: RINBOT virus outbreak Second round: European Union asks Apple to change iTunes copy protection Third round: Microsoft reports that its OneCare antivirus software needs work |
| Technical (difficult) | First round: 8800 Second round: Woosh Third round: March Vista update |
First round: nVidia driver update for Vista Second round: Woosh Wireless Broadband now compatible with Vista Third round: March 2007 Windows Vista Application Compatibility Update |
| International (easy) | First round: Airbus strike Second round: Morocco Third round: Russian crash |
First round: Workers strike over Airbus cuts Second round: Morocco cafe bombing on March 11, 2007 Third round: Plane crash in Samara, Russia |
| International (difficult) | First round: Zimbabwe Second round: Nigeria Third round: Ugandan troops |
First round: Australia cricket team defeats Zimbabwe in World Cup warm-up match Second round: Presidential candidate Umaru Yar'Adua returns to Lagos Third round: Ugandan ex-troops win $2 billion payout |
| Business (easy) | First round: Volt Second round: gas prices Third round: China interest rates |
First round: GM announces 2010 launch date for its Volt electric car Second round: Average gas prices rise 20 cents per gallon Third round: China central bank announces on March 17, 2007 (the day of testing) that it is raising interest rates to curb investment |
| Business (difficult) | First round: 7921G Second round: Alfa Third round: patriot blend troops |
First round: Cisco introduces the Unified Wireless IP Phone 7921G Second round: Russia's Alfa Group conglomerate raises its voting stake in Pelcom Third round: Newhall Coffee Roasting donates 12,000 pounds of its Patriot Blend to troops in Iraq and Afganistan |
| Popular/breaking (easy) | First round: Libby Second round: California fires Third round: war protest |
First round: Scooter Libby found guilty in CIA leak case Second round: Brushfires in Southern California Third round: Antiwar march in Washington, D.C. on March 17, 2007 (the day of testing) |
| Popular/breaking (difficult) | First round: Agoura water polo Second round: Oaks Christian Third round: Los Altos closure |
First round: Agoura High School's girls water polo team beats Laguna Beach in CIP Division II championship Second round: The Westlake Village, California, Oaks Christian High School's girls soccer team wins CIF title Third round: School board in Camarillo, California, votes to close Los Altos Middle School |
| Weather (easy) | LA weather | Weather forecast for Los Angeles |
| Weather (difficult) | Leadville weather | Weather forecast for Leadville, Colorado |
| Blog search | Query | Target response |
| Technical information (easy) | First round: RINBOT Second round: EU iTunes Third round: Microsoft antivirus |
First round: Blog discussing the RINBOT virus Second round: Blog discussing the European Union's beef with iTunes Third round: Blog discussing Microsoft's comment that its OneCare antivirus software needs work |
| Technical information (difficult) | First round: 8800 Second round: Woosh Third round: March Vista update |
First round: Blog discussing the nVidia driver upgrade for Vista (see news for details) Second round: Blog discussing Woosh's new Vista compatibility Third round: Blog discussing March 2007 Windows Vista Application Compatibility Update |
| Breaking news (easy) | First round: Libby Second round: Fred Thompson Third round: war protest |
First round: Blog discussing the Scooter Libby verdict Second round: Blog discussing Fred Thompson's possible 2008 presidential run Third round: Blog discussing the antiwar march in Washington, D.C., on March 17, 2007 (the day of testing) |
| Breaking news (difficult) | First round: Ballymena Second round: Nigeria Third round: Derbyshire cricket |
First round: Blog discussing the Irish Cup match between Linfield and Ballymena United Second round: Blog discussing the breaking news story, "Nigeria's Yar'Adua Blames Opposition for Death Rumour" Third round: BBC's Derbyshire Cricket Blog |
| Popular commentary (easy) | First round: Huffington Libby Second round: Huffington Hagel Third round: Malkin protest |
First round: Arianna Huffington opining on the Scooter Libby verdict Second round: Blog on Senator Chuck Hagel's presidential bid Third round: Michelle Malkin's blog of March 17, 2007, on that day's Washington D.C. antiwar protest and counterdemonstrators |
| Popular commentary (difficult) | Santa Paula | City of Santa Paula, California, blog written by its city manager |
| Product review (easy) | First and second rounds: iTunes Third round: Google Talk gadget |
First and second rounds: The iPod Blog Third round: Preview of Web-based module for Google Talk users |
| Product review (difficult) | First and second rounds: D40x Third round: EaglePicher |
First and second round: Preview of Nikon D40x digital camera (or similar) Third round: Preview of supersmall battery |
| Company profile (easy) | First and second rounds: Google Third round: Vonage and Covad |
First and second round: Inside Google Third round: Profile contrasting two popular VoIP providers |
| Company profile (difficult) | First and second rounds: Wetpaint Third round: Tello |
First and second round: Post on the wiki service Third round: Post on the demise of the pioneer VoIP service |
Add Your Comment