Using a Crowd to Troubleshoot the Web

It happened to Gmail users on Tuesday morning; it happens to everyone: You try to visit a Web site and get an error message. Why? Are you the only one who can't get through?

A new Web site, called Herdict Web, may have some answers for you. Herdict lets users share data on Web sites to build a global picture of which sites can and cannot be reached, and from where.

It is the brainchild of Jonathan Zittrain, a Harvard University law professor, who said he first began thinking about the idea in 2002 while researching Internet censorship in countries such as China and Saudi Arabia. Because the Internet is so decentralized, there's never been a good way of tracking which sites are down for whom. Sites can appear offline for many reasons: it could be due to a temporary networking bug at an ISP (Internet service provider), the Web site itself could be knocked offline, or it might even be censored by a national government.

Last year Zittrain co-authored a book, called Access Denied, that mapped out much of the Internet's censorship. But by the time the carefully researched book was published, its data was a year old. "I was eager to complement what we were already doing with a sense of real-time monitoring and reporting," he said in an interview Thursday. "You could aggregate ... reports and use those reports to invite others to test. Before you know it you'd start to have a map of what's inaccessible and where."

That's just what Herdict aims to do. Web surfers can visit the site to get information on whether certain Web sites are online in specific parts of the world, and they easily submit data on sites that may or may not be working for them. There's also a Herdict plugin for Firefox and Internet Explorer that lets users anonymously report how sites are working to Herdict's central database. Zittrain is meeting with Mozilla developers on Friday and says he'd "be delighted" to see it become part of the basic Firefox browser.

Herdict also has a mash-up with Google Maps that lets you see which sites are currently being reported as inaccessible in which parts of the world. On Thursday a site called, designed to skirt corporate Web-tracking software, was being widely reported offline.

Zittrain is also in the early stages of testing another project, called Herdict PC, which will compile the same kind of information about programs that users are installing on the Web. The idea is to use the same kind of crowd-sourcing, data-gathering techniques to see how well PCs are running compared to others and to get an idea of whether computers have been infected by a malicious program.

Launched Wednesday, Herdict Web doesn't yet have a lot of users, but Zittrain says that "the critical mass of just a few thousand will generate useful data."

The project could be used to track Internet censorship, but it could prove equally useful in tracking what pages are being taken down by Web site operators or being blocked within corporations, Zittrain said. "It may turn out that this becomes a really useful tool for tracking takedown requests on YouTube videos."

To comment on this article and other PCWorld content, visit our Facebook page or our Twitter feed.
Shop Tech Products at Amazon