Searching the Hidden Web
A universe of hidden resources exists online, in what some people call the "deep" or "invisible" Internet. This includes the rich databases of information from businesses, universities, government agencies, and other organizations that the Web's search engines can't spider and thus can't include in their results. How do you find the ones that are most useful to you?
A wise way to kick off your search: Look through a list of hidden resources, broken down into categories that you can browse for relevance to your work and interests. The Invisible Web Directory lists hidden sites in topic groups ranging from art to business. (Much of its material comes from The Invisible Web, a book by Chris Sherman and Gary Price.)
"It's not hard for search engines to find these databases, but it's very difficult for them to get past the search form and explore their contents," Sherman says. He estimates that the hidden Web is 2 to 50 times larger than the visible Web.
Direct Search, a site hosted by FreePint and run by Gary Price, gathers invisible Web databases into well-organized categories and allows you to simultaneously query both regular search engines and some facts databases.
For university or academic research sources, try Infomine, built by university librarians. The site links to databases, online journals, books, bulletin boards, mailing lists, articles, directories of researchers, and other online resources.
CompletePlanet, run by BrightPlanet, also links you to deep Web databases. But this site won't find a search term for you in one of these databases; it will only direct you to a possibly appropriate searchable database. Turbo10 also collects deep Web content and lists more than 1700 specific deep sources that hit everything from business data to libraries to government sources. If you like, you can create your own list of these engines to search with--say, a particular university's collections or a specific government database.
Metasearch sites Dogpile and Ez2Find query the popular search sites as well as some deep Web material, often yielding great results with a minimum of visual distraction. A different kind of metasearch engine is Vivisimo, which clusters results for easy selection. Type in a product name, and Vivisimo breaks down results by descriptions, reviews, and mentions in magazine articles, for example. The site is handy for researching a broad topic or new subject.