• PC World
• »Web

How We Tested: Search Engines Compared

Our search engine tests covered five categories: the relevance of the first results returned by each search service, its advanced search features, its ease of use, the percentage of dead links it returned, and its ability to find links to information about a recent event (currency).

The relevance category was further broken down into five subcategories: product information (reviews of Pioneer DVD-Recordable drives), business information (Cisco Systems' first-quarter 2001 revenue), technology information (the maximum data-transfer rate of the Universal Serial Bus 2.0 standard), regional information (room rates at the Agate Cove Inn bed and breakfast in Mendocino, California), and hard-to-find information (the author of an out-of-print 1920s sci-fi novel, Eater of Darkness).

To test for currency we searched for information about an event that was topical at the time of our tests: musical artist and entrepreneur Sean Combs's decision to change his professional name from "Puff Daddy" to "P. Diddy."

We awarded from 0 to 40 points in the relevance and advanced search categories (averaging the relevance subcategories), and 0 to 20 points in the ease-of-use, dead links, and currency categories. In the first chart, we converted the number scores to one of four descriptions: excellent, good, fair, and poor. The second chart lists the number scores in each category, and the third chart shows the relevance test results when using each service's advanced search features. Note that Ask Jeeves and LookSmart are excluded from the third chart because they don't offer advanced search features.

In the relevance tests, the services earned 40 points when a link to the information we were looking for was among the first five results returned, 25 points when it was between the 6th and 10th results, 15 points when it was between the 11th and 20th results, and 5 points when it was between the 21st and 30th results.

For ease of use we subtracted points for anything that distracted us from: finding the site's search box on its home page, entering our search, or navigating through the service's list of results. These included obtrusive ads, poor placement of the search box, and any other design elements that made the site difficult to use.

We subtracted five points from the original 20 for each dead link in the first 20 results returned.

For our currency test we awarded 20 points when a link to a page with information about "P. Diddy" was among the first five returned, 15 points when it was between the 6th and 10th results, and 5 points when it was within the 11th and 20th results.