Lessons Learned: IT's Biggest Project Failures

Sainsbury's Warehouse Automation

Sainsbury's, the British supermarket giant, was determined to install an automated fulfillment system in its Waltham Point distribution center in Essex. Waltham Point was the distribution center for much of London and southeast England, and the barcode-based fulfillment system would increase efficiency and streamline operations. If it worked, that is.

Installed in 2003, the system promptly ran into what were then described as "horrendous" barcode-reading errors. Regardless, in 2005 the company claimed the system was operating as intended. Two years later, the entire project was scrapped, and Sainsbury's wrote off £150 million in IT costs. (That's $265,335,000 calculated by today's exchange rate, enough to buy a lot of groceries.)

Lesson Learned

A square peg in a round hole won't fit any better as time goes on. Put another way -- problems that go unaddressed at rollout will only get worse, not better, over time.

Canada's Gun Registration System

In June 1997, Electronic Data Systems and U.K.-based SHL Systemhouse started work on a Canadian national firearm registration system. The original plan was for a modest IT project that would cost taxpayers only $2 million -- $119 million for implementation, offset by $117 million in licensing fees.

But then politics got in the way. Pressure from the gun lobby and other interest groups resulted in more than 1,000 change orders in just the first two years. The changes involved having to interface with the computer systems of more than 50 agencies, and since that integration wasn't part of the original contract, the government had to pay for all the extra work. By 2001, the costs had ballooned to $688 million, including $300 million for support.

But that wasn't the worst part. By 2001, the annual maintenance costs alone were running $75 million a year. A 2002 audit estimated that the program would wind up costing more than $1 billion by 2004 while generating revenue of only $140 million, giving rise to its nickname: "the billion-dollar boondoggle."

The registry is still in operation and still a political football. Both the Canadian Police Association and the Canadian Association of Chiefs of Police have spoken in favor of it, while opponents argue that the money would be better spent otherwise.

Lesson Learned

Define your project scope and freeze specifications before the requests for changes get out of hand.

Three Current Projects in Danger

At least Canada managed to get its project up and running. Our final three projects, courtesy of the U.S. government, are still in development -- they have failed in many ways already, but can still fail more. Will anyone learn anything from them? After reading these other stories, we know how we'd bet.

FBI Virtual Case File

In 2000, the FBI finally decided to get serious about automating its case management and forms processing, and in September of that year, Congress approved $379.8 million for the Information Technology Upgrade Project. What started as an attempt to upgrade the existing Automated Case Support system became, in 2001, a project to develop an entirely new system, the Virtual Case File (VCS), with a contract awarded to Science Applications International Corp.

That sounds reasonable until you read about the development time allotted (a mere 22 months), the rollout plans (a "flash cutover," in which the new system would come online and the old one would go offline over a single weekend), and the system requirements (an 800-page document specifying details down to the layout of each page).

By late 2002, the FBI needed another $123.2 million for the project. And change requests started to take a toll: According to SAIC, those totaled about 400 by the end of 2003. In April 2005, SAIC delivered 700,000 lines of code that the FBI considered so bug-ridden and useless that the agency decided to scrap the entire VCS project. A later audit blamed factors such as poorly defined design requirements, an overly ambitious schedule and the lack of an overall plan for purchases and deployment.

The FBI did use some of what it learned from the VCF disaster in its current Sentinel project. Sentinel, now scheduled for completion in 2012, should do what VCF was supposed to do using off-the-shelf, Web-based software.

Homeland Security's Virtual Fence

The U.S. Department of Homeland Security is bolstering the U.S. Border Patrol with a network of radar, satellites, sensors and communication links -- what's commonly referred to as a "virtual fence." In September 2006, a contract for this Secure Border Initiative Network (SBInet, not to be confused with Skynet) was awarded to Boeing, which was given $20 million to construct a 28-mile pilot section along the Arizona-Mexico border.

But early this year, Congress learned that the pilot project was being delayed because users had been excluded from the process and the complexity of the project had been underestimated. (Sound familiar?) In February 2008, the Government Accountability Office reported that the radar meant to detect aliens coming across the border could be set off by rain and other weather, and the cameras mean to zoom in on subjects sent back images of uselessly low resolution for objects beyond 3.1 miles. Also, the pilot's communications system interfered with local residents' WiFi networks -- not good PR.

In April, DHS announced that the surveillance towers of the pilot fence did not meet the Border Patrol's goals and were being replaced -- a story picked up by the Associated Press and widely reported in the mainstream media. But the story behind the story is less clear. The DHS and Boeing maintain the original towers were only temporary installations for demonstration purposes. Even so, the project is already experiencing delays and cost overruns, and in April, SBInet program manager Kirk Evans resigned, citing lack of a system design as just one specific concern. Not an auspicious beginning.

Census Bureau's Handheld Units

Back in 2006, the U.S. Census Bureau made a plan to use 500,000 handheld devices -- purchased from Harris Corp. under a $600 million contract -- to help automate the 2010 census. Now, though, the cost has more than doubled, and their use is going to be curtailed in 2010 -- but the Census Bureau is moving ahead with the project anyway.

During a rehearsal for the census conducted in the fall of 2007, according to the GAO, field staff found that the handheld devices froze or failed to retrieve mapping coordinates (see Hard questions needed to save projects for details). Furthermore, multiple devices had the same identification number, which meant they would overwrite one another's data.

After the rehearsal, a representative of Mitre Corp., which advises the bureau on IT matters, brought notes to a meeting with the bureau's representative that read, "It is not clear that the system will meet Census' operational needs and quality goals. The final cost is unpredictable. Immediate, significant changes are required to rescue the program. However, the risks are so large considering the available time that we recommend immediate development of contingency plans to revert to paper operations."

There you have it, a true list of IT Ig Nobels: handheld computers that don't work as well as pencil and paper, new systems that are slower and less capable than the old ones they're meant to replace. Perhaps the overarching lesson is one that project managers should have learned at their mothers' knees: Don't bite off more than you can chew.

No Prize for IT

Information technology has rarely won an Ig Nobel award in the 18 years the prizes have been doled out by the Improbable Research organization.

Should we take the snub personally?

Marc Abrahams, the editor of Improbable Research, the organization's blog, says he thinks IT's relative absence is simply because the field is younger than other disciplines. "Certainly IT offers the same level of absurdity as other areas of research," he says comfortingly.

He points out that Murphy's Law, whose three "inventors" (John Paul Stapp, Edward A. Murphy, Jr. and George Nichols,) were honored with an Ig Nobel in 2003, sprang from an IT-like project in the late 1940s. Murphy was an electrical engineer who was brought in to help the Air Force figure out why safety tests they were conducting weren't producing any results. Murphy discovered that the electronic monitoring systems had been installed "backwards and upside down," according to Abrahams, which discovery caused him to mutter the first version of the law that bears his name.

Other Ig Nobels drawn from the world of technology include:

2001: John Keogh of Hawthorn, Victoria, Australia, won in the Technology category for patenting the wheel; he shared the award with the Australian Patent Office, which granted him Innovation Patent #2001100012 (pdf) for a "circular transportation facilitation device."

2000: Chris Niswander of Tucson, Ariz., won a Computer Science Ig Nobel for his development of PawSense, software that can tell when a cat is walking across your keyboard and make a sound to scare it off.

1997: Sanford Wallace -- yes, that Sanford Wallace -- of Cyber Promotions takes the Communications Ig Nobel for being the Spam King.

The Ig Nobels, it must be remembered, aren't into value judgments.

San Francisco-based Widman is a frequent contributor to Computerworld.

Subscribe to the Daily Downloads Newsletter

Comments