Netscape's Volunteer Army Indexes Web
It's true, there is strength in numbers. Just ask the folks at Netscape.
Once synonymous with the Navigator browser, America Online's Netscape Communications is gaining a reputation as a search leader. Through its Open Directory Project, Netscape harnesses the power of thousands of Internet volunteers to index the Web.
At last count, an army of 18,000 Netscape "editors" has indexed more than 1.2 million Web sites. That's enough to ruffle the feathers of the granddaddy of human-powered searches, Yahoo. It could also bode ill for other competitors say experts, who see Netscape as a formidable challenger in search technology.
Much of the Open Directory's success, including Yahoo, is traced to the frustration felt by the many Web surfers who can't find what they seek. Both the Open Directory and Yahoo assume humans, not machines, are better suited to finding and indexing relevant Web sites.
However, unlike Yahoo, About.com, and LookSmart, who pay squads of editors to lead searchers, Netscape depends on the efforts of loyal volunteers. Netscape, in a seemingly selfless act of socialism, lets everyday Net surfers build its directory. In turn, anybody can stop by DMOZ.com and download the Open Directory index free of charge, as long as they give Netscape credit.
That small price tag interests big portals like AltaVista, Lycos, HotBot, Netscape sibling AOL.com, and others that have cut their staffing costs by using the Open Directory.
It works like this. Instead of expecting staff motivated by salary to organize a section on, say, automobiles, Netscape tries to inspire auto enthusiasts to organize car site directories. The idea is that motivated specialists can find a better car site than a mercenary surfer who is charged with being an expert on many subjects.
At the same time, Netscape solves one of the biggest challenges facing search firms today: scaling to meet the enormous expansion of the Web. Ideally, the number of Open Directory editors will keep pace with the Web's growth. This means Netscape will always have enough "staff" to add sites to the directory.
Yahoo has an estimated 1.2 million sites in its index and 150 staff editors. But it draws criticism for being slow to list submitted sites in its index.
Netscape's Nintendo listing is typical of its directory. Three editors have ranked what they consider the most useful Web pages.
The "Forbidden Nintendo Information Repository" tops the list. The editors describe the site as having "lots of technical info, little-known facts, history, and oddities." Down the list is Nintendo's official home page along with scores of other sites.
Scott Schmucker, a 17-year-old high school student from Ohio who helps maintain the Nintendo list, says he surfs the Internet four hours a day and is a Nintendo fanatic. "Believe me, there are a lot of bad Nintendo sites out there. But when I find a good one, I'll get it on the index."
Schmucker says he doesn't mind working for free because he cares about pointing people to quality Nintendo sites. He says he's "proud" to be a part of the Open Directory.
"Sure, there are a lot of other search sites out there, but there is only one that is created by real Net users," Schmucker says.
Actually, Netscape's approach isn't unique. Go.com has created a similar staff of volunteers called Go Guides. Since its inception in August, Go.com has recruited about 6000 Guides and points to 190,000 sites. However, Go.com doesn't license its search technology to third parties.
Similarly, the HotLinks search engine requests you share your browser bookmarks, so others can search through them.
The communal zeal that has inspired so many to contribute to the greater good of the Web is similar to the "open source" movement, says Rich Skrenta, cofounder of the Netscape directory.
Netscape is riding the open-source wave with another Netscape project, the development of its next-generation Communicator 5.0 Web browser. The company turned its code over to a community of thousands of software engineers to tap their expertise and build a better browser. All modifications are made available to the public.
The open-source surge has also helped alternative operating systems like Linux shake up the software market by presenting a challenge to Microsoft--if not in volume, at least in enthusiasm.
"What we are building is a superior, participatory method for organizing content on the Web," Skrenta says. "Someday, Yahoo is going to realize that there's no way it can keep pace with the Web." That's the day Netscape wins the search war, he says.
Earlier this year Lycos replaced much of its directory listings with Open Directory, saying its staff of 200 editors couldn't keep up with the growth in the Web.
Quality, not quantity, counts, responds prime rival Yahoo. Representatives declined to comment any further.
Analysts say Yahoo shouldn't lose sleep.
"Yahoo isn't just about search anymore," says Emily Meehan, an analyst with The Yankee Group. "Yahoo is about content, commerce, and communication." Search is important, but the Open Directory will never threaten Yahoo, she says.
The market may have room for both--and other search sites, notes Malcolm MacLachlan, an analyst with International Data Corporation.
"It's not about putting Yahoo out of business," McLachlan says. "It's about creating a great directory and not giving people a reason to go to Yahoo."
The battle may be for mind share, but these free services do have prices attached. Netscape's Open Directory Project produces a Yahoo-like directory without the price tag. NetCenter and AOL.com both use the directory and benefit enormously, but don't have to shell out a nickel to build it.
Too, it's a matter of cyberturf. Still, if Netscape overtakes Yahoo in index size and popularity, that could be a tough turn for the search engine pioneer.