• PC World
• »Software
• »Browsers & Add-Ons

How We Tested the Search Engines

Our goal was to replicate the real-world use of search engines. Most people aren't search gurus or reference librarians, and they typically don't use or understand Boolean commands (such as AND, OR, NOT); the average Web user simply goes to a search site, enters a word or phrase, and presses <Enter>. True, there are many ways to fine-tune a query--see our list of our favorite search tricks--but most folks usually can't be bothered. That's why we didn't place quotation marks (e.g., "hiking boots") around our test queries, even though doing so would almost certainly have produced more-accurate hits.

We tested a total of 55 services in six categories: general (text info), video, mobile/local, news, images, and blogs. We conducted our tests over three weeks. We used ten terms in each category. For general text-information searches, for instance, our categories included technology, pop culture, research/academic topics, books, and travel/shopping. In each category we posed what we considered an easy query and a difficult one: Just about every search engine could find the 2007 Academy Award winners, for example, but only a few could locate a portrait of Italian poet-philosopher Giambattista Vico. We tested the engines at different times of the day--morning, afternoon, and night--on all seven days of the week. (However, the time of day and the day of the week had no impact on the results.) We occasionally changed our queries, particularly in the news and blog categories, to keep them timely. For instance, our easy pop-culture query changed from "Academy Award winners" to "2007 NCAA basketball tournament teams" later in testing.

We performed all of our testing on the same Windows system, a 2-GHz Celeron desktop with 512MB of memory and a 1.5-megabits-per-second (download speed) AT&T/Yahoo DSL Internet connection. We didn't test for query speed, but this typically varied by only fractions of a second. Among the most-popular search sites, we saw little difference in between the time we pressed <Enter> and the appearance of the search results. However, we did detect significant differences among the blog-search engines, with some smaller sites--such as Best of the Blogs--taking several seconds to post the results. For the local-search tests, we used a Nokia E62 smart phone.

Chart: Our Test Queries and Target Responses

We asked ten queries in each of six categories: text information, video, images, local information, news, and blogs. Within each category we sought both an easy and a difficult target in five different topics or subjects. In some instances our target was a specific Web page; but unless noted in the chart, any site that provided the target response was considered to have given a correct answer. For topical searches we changed the queries in each of the three rounds of testing. Click the icon below to see our complete list of test queries and the results we wanted for each.