Are Search Results for Sale?

Until recently, it seemed difficult to find what you were looking for on the Internet. Now it seems you can only unearth what search engines want you to find.

Thank the growing popularity of the pay-for-placement business models adopted by most major search sites and portals like Yahoo, AOL, and MSN. These sites have inked deals with firms like Overture, which delivers "sponsored" search results alongside the Web site's own recommended and "Web Directory" results.

For a fee, companies can have their Web site ranked within the sponsored listings. Those sites show up above those ranked by relevancy in the search results. Critics view the trend as the transformation of search engines from neutral maps of the Internet into biased online brochures.

Most people know how search engines operate these days. Simply key in words like "car" or "travel" and you are whisked to a page of search results peppered with a mix of paid and hand-picked sites, supposedly ranked by relevancy.

At least one company is challenging this premise. Mark Nutritionals, a diet pill manufacturer, is suing search sites AltaVista, Overture, FindWhat, and Kanoodle for $440 million in federal court in Texas. Mark Nutritionals complains that people who enter queries for its product, Body Solutions, at these sites are misdirected to competing Web sites that have paid for top billing.

The lawsuit, believed to be the first of its kind, challenges pay-for-placement search engines that deliver top listings based not on relevancy, but on who has ponied up cash for a prime spot. The lawsuit raises the longstanding argument that search engines rank their own interests higher than yours.

Self-Involved Search

Some argue it's only a matter of time before search destinations become little more than online Yellow Pages. "These search engines have chosen crass commercialism over editorial integrity," says Gary Ruskin, executive director of Commercial Alert.

The Ralph Nader watchdog group has complained to federal authorities that search engines shouldn't mix paid and unpaid content in online search results without pointing out the difference to unsuspecting Web surfers. Ruskin says the creep of self-serving interests is hard to miss.

But is commercialism such a bad thing? After all, someone needs to pay the freight for costly search technologies.

"The fact of the matter is, what you're looking for may actually be the thing advertised," says Matthew Berk, an analyst with Jupiter Media Metrix.

Since the dot-com meltdown, a sharp downturn in online ad spending has taken its toll on search sites.

Last year Go, NBCi, and Excite relaunched with new missions. These sites have since dropped editorially compiled results and now deliver ad-dominated search results.

Recently, Terra Lycos became the latest search site to charge Web site owners for listing single Web addresses in its Web index. For about $30 yearly, Lycos lets Web site owners submit a single URL for guaranteed inclusion in the Lycos Web index within 48 hours. The fee does not guarantee a high ranking, however, only placement within the index.

Credibility Challenged

Lycos, which launched as a search engine, stopped using its own technology in 1999. It now relies on Fast Search and Transfer to spider and index Web pages on the Internet.

Search services need to be very careful about mixing paid search results with relevant Web sites, says Danny Sullivan, editor of SearchEngineWatch.com, a site devoted to how search engines work.

Testing identical search queries at both the MSN and AOL Web sites produces results that hint at bias, though site owners bristle at the suggestion.

"Every AOL search [result] features the most relevant content for AOL users, whether it is a partner or non-partner site," says Andrew Weinstein, an AOL spokesperson.

MSN, responding in a written statement, states: "MSN Search's ultimate goal is to provide people with the most useful and relevant search results from the entire Web."

Comparison Searching

Try searching for "new cars" on AOL Search, and also on MSN Search on the public Internet. Six out of the first ten search results (including sponsored and hand-picked sites) on MSN link to Microsoft-owned auto sites, including MSN Carpoint. Four out of the first ten links on AOL Search (including sponsored and hand-picked sites) point to AOL-owned car-buying sites. AOL doesn't suggest any Microsoft-owned sites on its first page of results, and MSN doesn't list any AOL sites on its first page either.

Microsoft's Carpoint Web site is the number four car destination on the Web, according to site rating service Jupiter Media Metrix.

Similarly, a search for "Olympics" produces perplexing results. Six sites out of the top ten sites on MSN Search link to an MSN or Microsoft partner site, including the official Olympics.com site. The first AOL Time Warner site that MSN lists is a Sports Illustrated 2002 Winter Olympics Web site, ranked in spot number 14.

In the same search on AOL's proprietary network, six of the first ten sites link to AOL-owned sites. The Official Salt Lake 2002 Olympic Web site doesn't make the first page. A link to ESPN Olympics coverage (hosted on MSN) is ranked 13.

One Source, Two Answers

It's easy to understand AOL and MSN recommending their own sites before others. But both search engines, which offer a mix of recommended, sponsored, and popular sites, use search technology from Inktomi to find and rank popular Web sites.

Inktomi sites appear as Web-indexed sites, listed below both the "sponsored sites" and the sites picked by portal editors. Users might assume those "Web Directory" sites delivered by Inktomi are free of commercial interests. They are not.

Inktomi provides its customers raw data of Web sites ranked by relevancy. The customers sometimes edit the list to eliminate sites like porn and gambling. Inktomi customers can also choose to block any site or otherwise manipulate the list, according to Inktomi. Inktomi refuses to let the public (or press) search its untouched Web directory.

Both AOL and MSN say they aim to deliver the "most relevant results possible." But both also have commercial interests to consider. They also both deny they block competing Web sites from Inktomi search results.

However, try searching for "travel" on those competing sites. The results, again, primarily link to the company's own or partner sites. Four out of the first ten travel sites that AOL Search produces link back to AOL. The Web's number two travel site, Expedia (a Microsoft partner), is ranked number 14 by AOL. Conversely, an AOL travel Web site doesn't show within the first 50 sites listed by MSN Search.

Also, a link to Expedia is missing from the Overture sponsored links section of AOL's Web-based search results for "travel." Expedia paid Overture to place a link to its Web site when someone searches for "travel" on any Overture partner site (including AOL). Overture declined to comment on its arrangement with AOL, but says a small fraction of its customers block sites from being advertised. AOL says it doesn't know why Expedia isn't showing up in its sponsored links section.

AOL acknowledges adding links to the "travel" findings from Inktomi's engine. MSN says its Inktomi search results are untouched for the identical search. Neither can explain how both sites could use Inktomi technology but produce different results for identical searches.