Dale Sweitzer, a network administrator for Crossville Ceramics in Crossville, Tennessee, has hit a rough patch--or a series of rough patches to be exact.
Sweitzer, who handles security for 160 geographically dispersed PCs running Microsoft software, says that he spends more time and money applying software patches than he spends doing practically anything else on the job. And he's not alone.
Although vendors like Microsoft have been working to simplify software patch delivery, the problem remains critical for IT administrators, who are struggling to keep up with the patch work.
From filling security holes to upgrading features, the act of applying patches--installing a new piece of software code over an old one--has become, for many people, a full time job.
"Patching is a nightmare," said Alex Bakman, founder and CEO of patch management provider Ecora. The recent onslaught of security threats like Blaster and Slammer have only aggravated the problems administrators face, he said.
According to Bakman, applying a patch to each machine in a company's system takes an average of 30 minutes, and hundreds of patches have been released by various vendors so far this year. Companies are carefully selecting which patches they need to apply because their IT staff is already spending 2 to 3 hours a day patching systems, Bakman said.
"The current frustration level is incredibly high," Bakman said.
The Bigger Picture
Of course, to users, patching is just a symptom of a wider software problem: The seemingly endless discovery of new security vulnerabilities that must be patched and fixed throughout the software's lifetime demands an enormous investment of time and money.
"The total cost of ownership of software is incalculable," Sweitzer noted.
Vendors blame insecure code. However, eradicating patches by creating flawless software is impossible, given that imperfect humans write software code.
"Despite all our best efforts, all vendors in this industry still have vulnerabilities," Steven Adler, senior security strategist for Microsoft in Europe, the Middle East, and Africa, told an audience of IT administrators at Gartner Security Summit in London last month.
While the patch situation doesn't seem likely to improve anytime soon, vendors say they understand administrators' frustration and are working to improve the situation.
Last Resort
Oracle's chief security officer, Mary Ann Davidson, said that her company sees patching as the last phase in its security efforts.
"We try to do things right the first time, but to err is human," Davidson said.
According to Davidson, Oracle has a rigorous policy in place for testing and delivering patches, and the company notifies customers of severe problems that might be exploited.
"Otherwise, we don't want to yank their chains. People don't have time to apply a lot of patches," she said.
Patching is also a big concern for Sun Microsystems, according to Gilles Gravier, the company's managing director of operations for platform infrastructure and security in Europe, the Middle East, and Africa.
Sun combines patches with upgrades of its Solaris and Orion software, so customers can update and fix their systems on regular schedules. For more pressing security issues, however, the company releases stand-alone patches. Sometimes these are temporary patches--what Sun calls t-patches--that have not undergone a thorough testing program, Gravier said.
"People installing t-patches know that they haven't gone through full testing and that they could break something," he said. Nevertheless, the company feels it is necessary to issue patches for exploitable problems as soon as possible, Gravier said, noting that full testing can sometimes take several weeks or longer.
Despite all the efforts put into delivering timely and high-quality patches, Davidson added, all vendors think that they can do better. Users don't seem to expect a miracle, but they are looking for a lessening of their patching problems.
Positive Signs
Microsoft's recent decision to simplify the process by delivering patches once a month and combining fixes when possible may be a sign that the industry is taking the problem more seriously.
Andreas Wuchner-Bruhl, head of Global IT Security for Novartis Pharma, said that the changes were a "step in the right direction."
Wuchner-Bruhl, who is on Microsoft's security advisory board, said that the software maker is paying closer attention to customers' concerns.
Still, the company's most recent security changes will be of little help to him. Wuchner-Bruhl manages a system of more than 3000 mixed servers in a qualified environment, meaning that detailed reporting of any changes to the files and systems must be documented. Novartis is a pharmaceutical firm and must comply with detailed healthcare industry regulations.
In addition to spending 30 minutes applying each patch, his staff has to do 2 to 3 hours of paperwork to document it.
"In a qualified environment there is a lot of work behind the scenes; you don't just apply a patch," Wuchner-Bruhl said.
The company already collects patches for a monthly update, but combining multiple fixes in one patch can actually create more work in qualified systems because administrators have to document all the changes, whether they thought they needed them or not, Wuchner-Bruhl said.
Past the Patch
Even administrators working in an unqualified environment have to do more work than the phrase "applying a patch" implies. Most companies test the patches on an isolated system first to make sure it doesn't break an application--especially if that application is customized.
In fact, fear of breaking applications deters many companies from applying patches that they need, according to Ecora's Bakman. Instead, companies tend to put off patching, and certainly don't go through the process at critical times, such as before a big retail or holiday season, he said.
Similarly, "people won't apply patches for anything in the last three weeks of the fiscal year because they don't want to risk their systems going down," Oracle's Davidson said.
But patching is just a symptom of underlying problems with software, Wuchner-Bruhl noted. To address vulnerability issues, software vendors are increasingly attempting to offer more-secure products from the outset under "secure computing" initiatives.
Microsoft, for example, has said that it is rolling out technologies to protect customers from problems such as buffer overruns (which are often exploited by hackers to take over computers) and attacks on communications ports.
Looking Toward Longhorn
The Redmond, Washington, software maker has said that its next desktop operating system, code-named Longhorn, will be more secure.
Longhorn isn't due until mid-2005, however.
"That's at least two more years of patching on the desktop," Gartner analyst John Pescatore said at the Gartner Security Summit in London. "And there will still be problems."
While vendors work to deliver more-secure products, experts are advising users to establish a patch management process.
Patch Priorities
In a research note released in March, Gartner analysts advised companies to prioritize patch installations based on how critical the security vulnerability is, and to evaluate the patch installation requirements. Some patches may require that other patches be applied at the same time, Gartner said, and others may be superseded by more-current patches or service packs.
Companies should classify server and desktop configurations as standard or nonstandard so they can be patched according to their specific needs; and all patches should be tested before deployment, Gartner said. Furthermore, companies should accept only official patches, and the patch management infrastructure should be as tightly secured as the company's outward-facing Web and application servers.
All that might threaten to overwhelm an IT department, but multiple vendors have stepped forward with patch management tools that, among other things, log system configurations and automate some installation and update functions.
Though these tools offer administrators some much needed help with the symptoms of software insecurity, the underlying problem remains.
"Since software was first developed, there have been patches," Ecora's Bakman noted. "And that won't change anytime soon."
Ad Choices