How to Stop Searching and Start Finding
Ask a couple of strangers for street directions, and you never know what you'll get. They may be rushed and give you only partial instructions; they may lead you astray with incorrect information; or they may not know the way at all and simply shrug you off with a smile. Still, when you find yourself going past the same mini-mart for the seventh time, you may have no choice but to ask for guidance. But who do you approach: The twenty-something gas-station attendant? The woman washing her car? The man walking his dog?
Searching for anything on the Web carries the same uncertainty. You may know exactly what you're looking for, but you don't know where to find it or who to ask. And when you do ask, you can't be sure you'll get the right answer.
The Web is crawling with search sites vying to show you the way. Conservative
estimates put the number at over 8000, including pure search engines, general-
and special-interest directories, and metasearchers (which query multiple
search sites at once). In addition, there are numerous alternative search
methods you could opt for, including utilities that work with your browser
to mine the Web for information (see
With so many choices, how do you decide which search site to use, and when? We tested over 20 engines, directories, and expert sites to see which ones produced the best results--that is, the most relevant links in the most logical ranking, with the least effort on our part.
We ran a series of queries, ranging from broad to specific, on each site. Some were business-related, such as one that tried to track down certified public accountants in California; others were scholarly, including one that sought information about a lost Hemingway manuscript (his wife left it on a train in 1922). We also hunted for information on pop-culture topics ranging from Pokémon to Queen (Freddie Mercury's band, not Victoria, Elizabeth, or Latifah).
Some sites did surprisingly well; others forced us to wade through swamps
of irrelevant links or pointed us to pages that no longer existed. (For highlights
of what we found, see
Despite the profusion of methods for searching the Net, the most common tools remain search engines and directories, each of which has its own merits. The main difference between the two is human intervention. Search engines use automated software "spiders" that crawl through the Web to collect and index the full text of pages that they find.
Directories, by contrast, rely on human editors to sift through pages,
winnowing out inappropriate ones and categorizing sites by subject. Nothing
goes into the directory unless an editor approves it, so you're unlikely to
find Christmas recipes in a Yahoo category about ham radios. But since directories
are crafted by hand, they are far less comprehensive than search engines.
Conservative estimates put the current size of the Web at a billion pages
(and gauge its growth at a million pages per day). Most engines don't come
close to indexing the whole Net, but at press time Google claimed to have
indexed a billion pages either partially or completely. The largest directory,
called the
Directories and search engines have different virtues, so most major search
sites meld aspects of each. AltaVista, for example, supplements its search
engine with a directory that uses listings from the
Of course, many search sites do a lot more than provide an engine and a directory--Excite, Go.com, Lycos, Yahoo, and others have morphed into portals offering an array of services, including stock quotes, news, e-mail, shopping, and anything else to dissuade you from going elsewhere on the Web. That scattershot strategy keeps visitors glued to pages long enough to notice the banner ads, helping the sites make a buck. But while much of the stuff portals offer is useful or fun, these sites can become so cluttered with peripheral features that the search tools seem an afterthought.
Some search sites are returning to the basics. Google started the trend
last year: Its sleek home page consists of little more than the Google logo,
the field where you type in your queries, and a couple of buttons. More recently,
AltaVista launched
Logically you'd expect the search engine that indexes the most pages to have the best chance of finding what you need. But if an engine doesn't properly organize the sites it finds, the one you want may be buried beneath thousands of irrelevant links. What good does it do for an engine to deliver 6000 environmental links when you really want a site for the rock group Green Day?
At Lycos, for example, we searched for
Of the sites we tested, Google provided the most consistently pertinent results--one reason it's our favorite engine overall. The site uses "page rank" technology to track the number of pages that link to a site. If a lot of pages link to a particular site on a specific topic, the reasoning goes, that site must be relevant to that subject. Consequently, Google gives it higher placement in the results.
Google has such confidence in this theory that it offers an "I'm feeling
lucky" button you can click to go directly to the site Google thinks is most
relevant to your search. But it might more appropriately be called
Google was less fortunate when we searched with the keywords
Another Google plus: Below each link it finds, Google provides a snippet of text with the word or words you searched for highlighted in bold. That helps you eyeball the results and gauge the relevance of each link quickly. (Most engines simply display the first line or two of text from each linked page, whether the text contains your search terms or not.)
Direct Hit takes a different approach to maximizing the relevance of results
by organizing them based on their popularity with previous searchers. For
instance, if few searchers clicked the first link for a search about computer
chips (say, because it is actually for a site about
In 1994 when the World Wide Web first took off, your choices for searching the Net were pretty limited: Yahoo and, uh... Yahoo. The site remains a cornerstone of Web research, but as a directory it faces stiff competition from the Open Directory and from LookSmart.
All three are useful, but on balance, we like the Open Directory Project best. Spearheaded by Netscape, this project relies on more than 24,000 volunteer editors worldwide who have indexed nearly 2 million sites in over 200,000 categories. Any other search site can license Open Directory and use its database for their own search results, and many of them do: AltaVista, HotBot, Lycos, MetaCrawler, and some 100 other sites dip into it for links.
You might expect the quality of the Open Directory's results to be erratic,
since the site relies on the work of volunteers. But in our tests, it produced
well-organized lists of pertinent sites, with clear descriptions of each link.
(In some cases, Yahoo offered only cryptic descriptions, or none at all.)
And the
The descriptions at LookSmart are also superior to those at Yahoo. But
LookSmart seems to have fewer listings for business sites than either Open
Directory or Yahoo--a problem if you're trying to track down an office design
firm in Detroit. And though it's not a full-service portal, LookSmart sometimes
lards its results with ridiculously off-topic shopping links. Do a search
for
No matter which directory you choose, all have one advantage over search engines: They're browsable, so you can wander through a series of related topics laid out in the site's menu. For instance, click on "Sports" at LookSmart and you get a drill-down list of topics like Baseball, Olympics, and Motor Sports. Click on "Motor Sports," and you can refine the search to a subtopic such as Formula One Racing or NASCAR.
Or suppose you need a company to make a sign for your Los Angeles-based business, but you know of only one supplier. You can perform a reverse query by running a directory search on the name of the company you know. Yahoo, for instance, will list the category that the company falls under ("Los Angeles>Business and Shopping>Business to Business>Signage"). Click on that category, and you'll get a list of sign manufacturers in LA. This approach can help you track down sites more quickly than you would by guessing at keywords.
It's bound to happen: AltaVista has a link to exactly the Web site you're
looking for. But you're searching at Google, which doesn't know that particular
site from
In the early days of the Web when no single search engine had indexed more than a few million pages, metasearchers were essential. Now AltaVista, Google, and other engines boast indexes of hundreds of millions of pages and let you comb a significant percentage of the Web all by themselves. Still, the best metasearch engines are worth trying, especially at times when your favorite single-engine search site strikes out. Unfortunately, metasearchers often fail to match the relevance of Google--probably because it's so tough to mesh links from disparate sites coherently.
Today, the Web has numerous metasearcher sites. Among the most prominent
are
The popularity of metasearchers has encouraged many traditional engines to acquire metasearch-like features. Ask Jeeves, for instance, provides results from About.com, AltaVista, Excite, WebCrawler, and 4anything Network in addition to the results from its own database.
MetaCrawler, our favorite metasearcher, draws on solid search sites such as AltaVista and Google, which helps it bring back plentiful and pertinent results. It trolls through all sorts of Net resources, too. Search for Beethoven and you can click on tabs to leap between engine and directory results, MP3 music files, images, newsgroup discussions, and even online auctions relating to the great composer and his music.
Another worthwhile metasearch site is Search.com, largely because of its specialized metasearchers on dozens of topics. The Automotive one, for instance, queries Autobytel, Car and Driver, CarPrices.com, and nine other car sites. But Search.com's general-purpose metasearcher often starts by returning a link from GoTo.com, a search engine that invites sites to pay for higher placement in its results. When we searched for Dell Computer at Search.com, it referred us to GoTo's paid link to a company that leases Dell PCs before it identified Dell's own home page. But at least Dell's site was the second link at Search.com; at Mamma.com, it was number 18.
One of our favorite search sites has indexed a grand total of just 1600
sites. That achievement might seem negligible, except for one thing: Each
of those 1600 sites is about beer. So if you seek a home brew club in Portland,
Oregon,
From
In addition, because these sites are so focused, the results are highly
relevant: If you search for
How do you find out if there's a search site devoted to, say, springer
spaniels? Simple: Consult a guide to topic-specific searchers. Two of our
favorites are
Only a few specialized search sites use true automated engines. (One of
these is Ditto.com, which helps you find images on the Web.) Most are actually
hand-compiled directories to sites on a specific topic. As such, they're better
for topic searches than for highly specific queries. For instance, search
Many specialty searchers are labors of love by one obsessed Netizen. Therefore, a site will likely grow stale if its proprietor loses interest or runs short of spare time. Look for a "Last updated" banner on the home page--and move on if the site seems to be in limbo.
For years, search-engine companies have tried to make their sites think like human beings (without much success). So why not turn the tables? That's the idea behind expert sites--information centers powered by the knowledge and opinions of real people.
Leading contenders in this area include
The drill at most expert sites is similar: Browse a Yahoo-style topic directory until you find the category you want. Then post your query and wait for answers from site visitors ("experts") who claim knowledge in that subject.
Much of the time, this system delivers savvy, personalized assistance of a sort no search engine can. When we asked about buying a commuting bike, for instance, an AskMe expert shared his favorite urban biking links, and a seasoned cyclist at Abuzz provided a checklist to bring to the bike shop. Though response times from such sites are not instantaneous, we often received answers within an hour or two.
In the end, we liked AskMe.com's easy interface and friendly, responsive community of experts best. But even there, some queries went unanswered, and pop-culture fans greatly outnumber literary historians. We asked one user who touted her expertise on Hemingway about the author's lost manuscript. She had never heard of it and then admitted she could name only two of his works.
When we asked the same question at LookSmart Live and Exp.com, we got accurate responses the same day. (At Exp.com, the answer cost us $3, but LookSmart performed just as well for free.) The lesson: No expert site has authorities on everything, so it makes sense to bookmark several sites.
In fact, that's essential advice for Web searching of all sorts: Don't depend on one site, or even two. Bookmark a bunch of them, get to know their strengths and limitations, and use them all. Even then, they won't always know the shortest route to the knowledge you seek: The Web is rife with detours and dead ends. But with the right sites to guide you, you'll spend less time driving aimlessly on the information highway.
You've got questions. Who's got answers? Start your quest at these sites--our picks as the best options in five key categories of search tools.
Besides using search sites on the Web, you can install search utilities on your hard drive. Here are three we recommend.
Though Copernic isn't a search engine per se, it raises online research
to new heights by offering impressive organization and collaboration tools.
You can save results to custom folders, e-mail results to others, eliminate
broken links by clicking on the Validate button, and sort entries by relevance
and other criteria. Like a metacrawler, the free version submits your query
to dozens of search engines simultaneously. The Plus and Pro versions ($40
and $80, respectively) work with hundreds of general and specialized engines,
and let you perform such advanced options as removing banner ads from results
pages and (in the Pro version) running unattended, scheduled searches. Get
a free version at
This tiny, free utility (available at
Combining the timeliness of online data and the speed of accessing 22 localized reference tools (dictionaries, encyclopedias, almanacs, and an atlas), X-Portal is ideal for the avid researcher. The $60 CD-ROM program needs about 200 megabytes of hard disk space; it integrates into Internet Explorer (version 4.01 or higher). X-Portal queries multiple search engines and its built-in reference works, then sorts results by relevance. The program weeds out many superfluous and not-so-relevant links from the list it presents to you at the end of the search.
