Search Engine Shoot-Out

How We Tested the Search Engines

Our goal was to replicate the real-world use of search engines. Most people aren't search gurus or reference librarians, and they typically don't use or understand Boolean commands (such as AND, OR, NOT); the average Web user simply goes to a search site, enters a word or phrase, and presses <Enter>. True, there are many ways to fine-tune a query--see our list of our favorite search tricks--but most folks usually can't be bothered. That's why we didn't place quotation marks (e.g., "hiking boots") around our test queries, even though doing so would almost certainly have produced more-accurate hits.

We tested a total of 55 services in six categories: general (text info), video, mobile/local, news, images, and blogs. We conducted our tests over three weeks. We used ten terms in each category. For general text-information searches, for instance, our categories included technology, pop culture, research/academic topics, books, and travel/shopping. In each category we posed what we considered an easy query and a difficult one: Just about every search engine could find the 2007 Academy Award winners, for example, but only a few could locate a portrait of Italian poet-philosopher Giambattista Vico. We tested the engines at different times of the day--morning, afternoon, and night--on all seven days of the week. (However, the time of day and the day of the week had no impact on the results.) We occasionally changed our queries, particularly in the news and blog categories, to keep them timely. For instance, our easy pop-culture query changed from "Academy Award winners" to "2007 NCAA basketball tournament teams" later in testing.

We performed all of our testing on the same Windows system, a 2-GHz Celeron desktop with 512MB of memory and a 1.5-megabits-per-second (download speed) AT&T/Yahoo DSL Internet connection. We didn't test for query speed, but this typically varied by only fractions of a second. Among the most-popular search sites, we saw little difference in between the time we pressed <Enter> and the appearance of the search results. However, we did detect significant differences among the blog-search engines, with some smaller sites--such as Best of the Blogs--taking several seconds to post the results. For the local-search tests, we used a Nokia E62 smart phone.

Chart: Our Test Queries and Target Responses

We asked ten queries in each of six categories: text information, video, images, local information, news, and blogs. Within each category we sought both an easy and a difficult target in five different topics or subjects. In some instances our target was a specific Web page; but unless noted in the chart, any site that provided the target response was considered to have given a correct answer. For topical searches we changed the queries in each of the three rounds of testing. Click the icon below to see our complete list of test queries and the results we wanted for each.

Search-Engine Shoot-Out: Our Test Queries and Target Responses
We asked ten queries in each of six categories: text information, video, images, local information, news, and blogs. Within each category we sought both an easy and a difficult target in five different topics or subjects. In some instances our target was a specific Web page; but unless noted below, any site that provided the target response was considered to have given a correct answer. For topical searches we changed the queries in each of the three rounds of testing.
Text-information search Query Target response
Technical (easy) WiMax speed and range 40 mbps per channel in a typical cell radius of 3km to 10km, as stated on the WiMax Forum
Technical (difficult) Linksys WPA2 Yes, the Linksys WRT54G wireless router does support WPA2
Pop culture (easy) First round: 2007 Academy Award nominees
Second round: 2007 Academy Award winners
Third round: 2007 NCAA basketball tournament teams
First round: The award nominees in each category
Second round: The award winners in each category
Third round: The NCAA basketball tournament field of 64 teams
Pop culture (difficult) Tigers may 1 The Detroit Tigers play the Baltimore Orioles at Comerica Park on May 1, 2007
Research/academic (easy) Capital of Moldova Chisinau (Kishinev)
Research/academic (difficult) weight gnu yak Male yaks weigh 1100 to 2645 pounds (500 to 1200 kilograms), and females weigh 660 to 770 pounds (300 to 350 kilograms); a gnu (a blue one) weighs 500 to 600 pounds (235 to 275 kilograms)
Book excerpt (easy) excerpt Doctorow The March An excerpt from the E.L. Doctorow book
Book excerpt (difficult) No Exit An excerpt from the Jean-Paul Sartre play
Travel (easy) 2007 All Star Game San Francisco's AT&T Park on July 7, 2007
Travel (difficult) Sanibel Beach The location of Bowman (or Bowman's) Beach on Sanibel Island in Florida
Video search Query Target response
Internet video (easy) Colbert speech 2006 White House Correspondents Dinner The full 24-minute, 10-second video of Stephen Colbert's roast of the president
Internet video (difficult) Food for Thought DeLuca Food for Thought; short
Television clip (easy) Samurai Delicatessen Saturday Night Live skit starring John Belushi and Buck Henry
Television clip (difficult) Gilda Radner Saturday Night Live skit starring Gilda Radner and Steve Martin dancing
How-to video (easy) how to carve a turkey Several potential correct answers, one of which is Martha Stewart's instructions
How-to video (difficult) upgrade notebook drive Any video showing step-by-step instructions for installing a new hard drive in a notebook PC
News (easy) First round: tornado in Enterprise, Alabama
Second round: Australian cyclone
Third round: Valerie Plame congressional hearing
First round: Tornado in Enterprise, Alabama
Second round: Australian cyclone
Third round: Valerie Plame at congressional hearing
News (difficult) First round: Dodgers home opener
Second round: Chatsworth
Third round: big rig 101
First round: "LA Dodgers Sell Out Home-Opener Tickets"
Second round: "HazMat Team Called to Chatsworth"
Third round: "Big Rig Crashes Through Center Divider on 101"
Animation (easy) Gopher Broke Nominee for 2005 Academy Award for animated short subject
Animation (difficult) Maestro Hungarian artist Geza M. Toth's Academy Award-nominated short
Image search Query Target response
Portrait (easy) Rutherford B. Hayes The nineteenth president of the United States
Portrait (difficult) Vico Italian poet-philosopher Giambattista Vico, 1668-1744
Building (easy) Pantheon The Pantheon in Rome
Building (difficult) Lincoln Hall Kentucky Lincoln Hall, Berea College, Kentucky
Thing (easy) platypus The marsupial
Thing (difficult) windform A 20-foot horn
Place (easy) Times Square The New York City landmark
Place (difficult) Asilomar Asilomar State Beach, Pacific Grove, California
Event (easy) Mardi Gras The New Orleans Mardi Gras celebration, 2007
Event (difficult) Keeneland Keeneland Racetrack (the building, not the horses or the track itself)
Local search Query Target response
Restaurant (easy) Spago The address and telephone number of the popular restaurant in Beverly Hills, California
Restaurant (difficult) Chili My Soul Information for the obscure restaurant in Encino, California
Gas station (easy) gas station 91301 The nearest station to the test location of Agoura Hills, California
Gas station (difficult) cheap gas 91301 The lowest-priced gas near the test location of Agoura Hills, California
Person's telephone number (easy) [Author's own number] The author's telephone number
Person's telephone number (difficult) telephone Bill Smith Thousand Oaks The telephone number of Bill Smith of Thousand Oaks, California
Driving directions (easy) Agoura Hills to Spago in Beverly Hills Directions for a short trip using mostly freeways from Agoura Hills to Spago in Beverly Hills
Driving directions (difficult) Agoura Hills to Zuma Beach in Malibu Directions for a longer trip using mostly surface streets and/or exurban roads from Agoura Hills to Zuma Beach in Malibu
Event/show/news (easy) L.A. Lakers tickets Any reputable ticket vendor
Event/show/news (difficult) Calabasas election March city election results for Calabasas, California
News search Query Target response
Technical (easy) First round: RINBOT
Second round: EU iTunes
Third round: Microsoft antivirus
First round: RINBOT virus outbreak
Second round: European Union asks Apple to change iTunes copy protection
Third round: Microsoft reports that its OneCare antivirus software needs work
Technical (difficult) First round: 8800
Second round: Woosh
Third round: March Vista update
First round: nVidia driver update for Vista
Second round: Woosh Wireless Broadband now compatible with Vista
Third round: March 2007 Windows Vista Application Compatibility Update
International (easy) First round: Airbus strike
Second round: Morocco
Third round: Russian crash
First round: Workers strike over Airbus cuts
Second round: Morocco cafe bombing on March 11, 2007
Third round: Plane crash in Samara, Russia
International (difficult) First round: Zimbabwe
Second round: Nigeria
Third round: Ugandan troops
First round: Australia cricket team defeats Zimbabwe in World Cup warm-up match
Second round: Presidential candidate Umaru Yar'Adua returns to Lagos
Third round: Ugandan ex-troops win $2 billion payout
Business (easy) First round: Volt
Second round: gas prices
Third round: China interest rates
First round: GM announces 2010 launch date for its Volt electric car
Second round: Average gas prices rise 20 cents per gallon
Third round: China central bank announces on March 17, 2007 (the day of testing) that it is raising interest rates to curb investment
Business (difficult) First round: 7921G
Second round: Alfa
Third round: patriot blend troops
First round: Cisco introduces the Unified Wireless IP Phone 7921G
Second round: Russia's Alfa Group conglomerate raises its voting stake in Pelcom
Third round: Newhall Coffee Roasting donates 12,000 pounds of its Patriot Blend to troops in Iraq and Afganistan
Popular/breaking (easy) First round: Libby
Second round: California fires
Third round: war protest
First round: Scooter Libby found guilty in CIA leak case
Second round: Brushfires in Southern California
Third round: Antiwar march in Washington, D.C. on March 17, 2007 (the day of testing)
Popular/breaking (difficult) First round: Agoura water polo
Second round: Oaks Christian
Third round: Los Altos closure
First round: Agoura High School's girls water polo team beats Laguna Beach in CIP Division II championship
Second round: The Westlake Village, California, Oaks Christian High School's girls soccer team wins CIF title
Third round: School board in Camarillo, California, votes to close Los Altos Middle School
Weather (easy) LA weather Weather forecast for Los Angeles
Weather (difficult) Leadville weather Weather forecast for Leadville, Colorado
Blog search Query Target response
Technical information (easy) First round: RINBOT
Second round: EU iTunes
Third round: Microsoft antivirus
First round: Blog discussing the RINBOT virus
Second round: Blog discussing the European Union's beef with iTunes
Third round: Blog discussing Microsoft's comment that its OneCare antivirus software needs work
Technical information (difficult) First round: 8800
Second round: Woosh
Third round: March Vista update
First round: Blog discussing the nVidia driver upgrade for Vista (see news for details)
Second round: Blog discussing Woosh's new Vista compatibility
Third round: Blog discussing March 2007 Windows Vista Application Compatibility Update
Breaking news (easy) First round: Libby
Second round: Fred Thompson
Third round: war protest
First round: Blog discussing the Scooter Libby verdict
Second round: Blog discussing Fred Thompson's possible 2008 presidential run
Third round: Blog discussing the antiwar march in Washington, D.C., on March 17, 2007 (the day of testing)
Breaking news (difficult) First round: Ballymena
Second round: Nigeria
Third round: Derbyshire cricket
First round: Blog discussing the Irish Cup match between Linfield and Ballymena United
Second round: Blog discussing the breaking news story, "Nigeria's Yar'Adua Blames Opposition for Death Rumour"
Third round: BBC's Derbyshire Cricket Blog
Popular commentary (easy) First round: Huffington Libby
Second round: Huffington Hagel
Third round: Malkin protest
First round: Arianna Huffington opining on the Scooter Libby verdict
Second round: Blog on Senator Chuck Hagel's presidential bid
Third round: Michelle Malkin's blog of March 17, 2007, on that day's Washington D.C. antiwar protest and counterdemonstrators
Popular commentary (difficult) Santa Paula City of Santa Paula, California, blog written by its city manager
Product review (easy) First and second rounds: iTunes
Third round: Google Talk gadget
First and second rounds: The iPod Blog
Third round: Preview of Web-based module for Google Talk users
Product review (difficult) First and second rounds: D40x
Third round: EaglePicher
First and second round: Preview of Nikon D40x digital camera (or similar)
Third round: Preview of supersmall battery
Company profile (easy) First and second rounds: Google
Third round: Vonage and Covad
First and second round: Inside Google
Third round: Profile contrasting two popular VoIP providers
Company profile (difficult) First and second rounds: Wetpaint
Third round: Tello
First and second round: Post on the wiki service
Third round: Post on the demise of the pioneer VoIP service

Subscribe to the Daily Downloads Newsletter

Comments