The Straight Story on Search Engines

Not Always Commercial-Free

AOL and MSN aren't the only sites where commercial interests seem to affect the real results. Try using MetaCrawler to search for a topic like "Abraham Lincoln." Even once we'd looked past the paid links, we were presented with a remarkable list of links to sites selling Lincoln T-shirts, vases, and other memorabilia. Of the first ten real results, only three weren't trying to sell us something. And when we ran a search for "digital cameras" on MetaCrawler, the last handful of links appearing on each page were identical--all pointing to shopping sites.

InfoSpace's Steve Stratz says MetaCrawler has no policy to place commercial sites above real results. "The search providers on MetaCrawler currently are heavy on the paid side," says Stratz. "We are working hard to even this out by partnering with pure search providers." Stratz points to recent agreements with Ask Jeeves, FAST, and LookSmart. Search engine AllTheWeb is based on FAST's search technology.

Even computer scientists can't prove or disprove whether funny business goes on within relevancy rankings. "Many search engines really don't publish reliable scientific information about the quality of their results," says Amanda Spink, an associate professor of information sciences and technology at Pennsylvania State University who recently completed a five-year study of the search habits of Excite users. "Most search sites don't like to give out this kind of competitive information," she adds.

Search companies provided only basic information about how they rank results. Google employs a complex algorithm, or mathematical formula, to rank pages. Yahoo employs a team of editors to create useful categories of Web sites that they've reviewed for inclusion in the directory. Beyond that, however, search sites refuse to explain the ins and outs of their formulas.

Spink's study showed that about 70 percent of people don't scroll beyond the first or second page of search results. That could prove troublesome if results pages become front-loaded with sponsors or paid inclusion listings--links that Web site owners pay a yearly fee to have included in a search engine's results.