Search Sites Try to Give Instant Answers
Three search engines promise direct replies to queries (not just links), but results are spotty.
Steve Bass
Three search engines promise to serve up direct answers to so-called natural-language queries: natural-language veteran AskJeeves; GuruNet's Answers.com, a relative newcomer to Web search; and MSN Search, which recently beefed up its search results with information culled from Microsoft's Encarta encyclopedia. In our sample searches, however, the three services did much better at answering our general-information questions than they did at replying to technical, topical, or geo-specific questions.
We expected the answers to our questions to appear at the top of each site's initial results page, without our having to follow a link. We gave the sites some credit when the correct answer was on the first page in the search results (one click away), or was within the top ten links returned (two or more clicks away).
We posed dozens of questions and reduced our results to a representative 15 (see chart). AskJeeves produced four on-the-money answers; MSN Search and Answers.com answered three of our questions spot-on. (We included Google as a control; its results were almost identical to those of Answers.com.)
Of course, Web information is constantly changing. Our latest results varied considerably from those we got when we asked the same questions days earlier.
All three sites were on target with the question "What was John Dillinger famous for?" (though we had to click the top link at Answers.com), and they also aced "Who was the 17th president of the United States?" Yet none of the three could immediately answer a question about the speed of USB 2.0 versus ethernet.
Extra words--ironically, the natural language--often got in the way of good results. For example, entering the keywords "high tide Santa Monica California" returned better results than "When is high tide on October 15, 2005, in Santa Monica, California?"
Direct Answers: No Search Engine Aced Our Test (chart)
| Web search champ Google was as good at answering our natural-language questions as the three sites that promote their ability to respond to questions rather than keyword phrases. Our goal was to see the correct answer without having to follow a link on the results page (no clicks). We gave the sites partial credit for providing the answer on the top link's page (one click), and for having the answer in its top ten links (two or more clicks). Following are 15 questions that represent the dozens we posed to each service, along with the correct answers. Also listed are two keyword phrases that we submitted to the sites as a control. | ||||
| Question | Answers.com | Ask Jeeves | MSN Search | |
| Who was the 17th president of the United States? (Answer: Andrew Johnson) | No clicks | No clicks | No clicks | No clicks |
| What is the best way to start a barbeque? (Answer: barbecue safety tips) | Two or more clicks | Two or more clicks | One click | Two or more clicks |
| How do you light a barbeque? (Answer: barbecue safety tips) | Two or more clicks | One click | One click | Two or more clicks |
| What are roundheads? (Answer: supporters of the Parliamentarians in the 17th-century English civil war) | No clicks | Two or more clicks | No clicks | No clicks |
| What is the correct name of Mt. McKinley? (Answer: Denali) | Two or more clicks | One click | Two or more clicks | Two or more clicks |
| Which states use daylight saving time? (Answer: all but Hawaii and Arizona) | Two or more clicks | One click | One click | Two or more clicks |
| What was John Dillinger famous for? (Answer: Depression-era bank robber) | One click | No clicks | No clicks | No clicks |
| How do you make ceviche? (Answer: Mexican fish-dish recipe) | Two or more clicks | Two or more clicks | One click | Two or more clicks |
| Who were the wives of Henry VIII? (Answer: Catherine of Aragon, Anne Boleyn, Jane Seymour, Anne of Cleves, Catherine Howard, and Katherine Parr) | No clicks | One click | Two or more clicks | One click |
| What is Medigap insurance? (Answer: Medicare supplement insurance) | One click | No clicks | One click | One click |
| What is the semaphore timeout period has expired error? (Answer: Windows error message indicating a problem with a DVD burner) | Two or more clicks | Two or more clicks | Two or more clicks | Two or more clicks |
| What is the speed of USB 2.0 versus SATA? (Answer: USB 2.0, 480 mbps; SATA, 1.5 gbps) | Two or more clicks | (No answer in top 10) | Two or more clicks | Two or more clicks |
| What is the speed of USB 2.0 versus ethernet? (Answer: USB 2.0, 480 mbps; ethernet, 10 mbps to 100 mbps) | Two or more clicks | (No answer in top 10) | Two or more clicks | Two or more clicks |
| What is the temperature in Paris? (Answer: 66 degrees Fahrenheit with light drizzle, last time we checked) | Two or more clicks | No clicks | One click | Two or more clicks |
| When is high tide on October 15, 2005, in Santa Monica, California? (Answer: 8:32 a.m. and 8:41 p.m.) | One click | (No answer in top 10) | One click | One click |
| tide table October 15, 2005, Santa Monica, California | One click | (No answer in top 10) | (No answer in top 10) | One click |
| high tide Santa Monica, California | One click | Two or more clicks | One click | One click |
COMMENTS: Answers.com's results are nearly identical to those of Google, but with several general-interest questions leading to an "answer" page rather than to a list of Web results. When we tested, AskJeeves listed more sponsored links than nonsponsored ones, but subsequently the site has removed many of these ads. MSN Search returns a large number of sponsored links, but it was the fastest of the bunch. On some searches (such as "Where is Algerie?"), MSN corrected misspellings on the fly with no user intervention. On Google's results page, unobtrusive sponsored links are placed on the right side. On some searches (such as "Where is Algerie?"), Google corrected misspellings on the fly with no user intervention, and sometimes suggested the correct spelling. | ||||