AlltheWeb.com Offers Near-Real-Time Indexing

Fast Search & Transfer will launch an upgraded version of its Alltheweb.com search engine on Monday that offers near-real-time searches in over 3000 online news sources.

Entering "anthrax," for example, can bring up news stories that were updated less than an hour ago from both national and local news organizations from various countries, and in a multitude of languages. The advanced search allows a user to set language preferences and also to pick what sections of the news to search.

Fast has added a new search, "vertical," which allows specific news searches. However, the most recent results from the news vertical are also displayed in a gray box on top of the general search results page.

"We want to show our customers that we can crawl fast, this is a subsecond index," says Rob Rubin, executive vice president and general manager of Fast's Internet division. "We are now continuously spidering and indexing over 3000 news sources."

Crawling, or spidering, is the process whereby a software agent follows links from one Web page to another and collects information on those pages. That information is then indexed to make it searchable.

Speeding Past the Competition

Fast sees crawling at high speed as its trademark. The general search index is refreshed every 9 to 12 days. Google, Fast's main competitor, reports it refreshes about once a month, and AltaVista admits it is months behind with refreshing, resulting in outdated search results.

"The news vertical is a significant freshness improvement," says Rubin.

Other new features on the site, which acts as a showcase for Fast's technology, are linguistic analysis of search queries, expanded customization options, and Fast Topics, which use the Open Directory Project, a human-edited directory run by volunteers, to bundle search results by topic.

Linguistic analysis, or antiphrasing, removes clutter from a search query to deliver better results. The query "What is the weather in Las Vegas?" for example, will be stripped of the "what is" and "Las" will be tied to "Vegas," with "weather" as an extra term.

Although Fast is not a widely known brand, it powers the search engines of Web portal company Terra Lycos, Europe's largest Internet service provider, eBay, and many others.

Fast last month announced an enterprise product, Fast Data Search, which allows large businesses to search the content on their own servers. Fast Data Search has been integrated, for example, with BroadVision's enterprise self-service applications.