Nine Real-Life IT Horror Stories
Stupid user trick No. 7: Duct tape doesn't fix everything
"This one still makes us laugh over beers," says H. Foreman, an admin at a Midwest-based organization.
"We were growing pretty well in 2004 and 2005, so we opened an office across the street," Foreman says. To connect the two offices, they decided to buy two microwave bridges.
"The setup is easy enough that we were able to do the job ourselves, though we had professional carpenters install the bridges to the walls of each building, just under the roof, pointing through double-paned office glass, so we would have no weather worries."
Success carried over into 2006, when the company decided to extend its leases.
"As part of the deal, they get permission to put up a fancy sign near the top of both buildings -- indoors but facing outward through the windows," Foreman says. "The day the sign goes up, our network goes down for about 15 minutes. We're still doing the basic set of troubleshooting diagnostics when it suddenly comes back up. Our guy shrugs, verifies everything again, and lets it go."
The next morning wasn't as forgiving. The network went down and stayed down.
"The basic software diagnostics aren't working, so we go to physical link monitoring," Foreman says. "Pretty quick, we see that one of the bridges isn't responding anymore. Upstairs we go."
Apparently the bridges had been in the way of the signs.
"The outfit that put up the signs just detached the bridges and moved them -- outside," Foreman says. "There was a balcony on the upper floor and they just moved both bridges out there and then duct-taped both of them to the railings.
"What kills us is that the network somehow recovered the first time," he says. "The duct tape across the street held, but the one on our side slipped off during the night and the bridge fell eight stories, bounced off the dumpster, and landed behind it. The sign installers apparently left a note explaining what they'd done with the receptionist across the street and she hadn't passed it on."
Naturally, Foreman and company had fun pointing the finger at the sign company in front of the CEO, who then ran out to chew out the install rep.
"But as soon as he left the room, the CIO, who is a really good ex-tech, pointed out that if we knew someone was going to be doing construction around a critical piece of network infrastructure, why the hell hadn't we gone up there to check it? Especially after the network went down during the construction process," Foreman says. "He had a point."
Solution: The basic network monitoring software this company was evidently using is as good a technology solution as you need in this instance. Without such software, however, this would have been a much nastier adventure.
Moral: What the CIO said. Construction around network infrastructure requires personal attention from your IT staff. Remote monitoring is no substitute for "eyes-on" during critical times.
Stupid user trick No. 8: Executive privilege
This one hits close to home, as some tech magazine editors epitomize the worst kind of user an IT admin can encounter: those who have read so much about IT that they simply assume hands-on expertise. What results are "special requests" of IT not unlike those we find dealing with higher-up execs.
Let me set the stage: I was working as a technical editor for an IT magazine some years back and happened to be in the executive editor's office three days in a row when this little drama went down. I can't remember whether Windows 95 or Windows 98 had just come out, but it was one of those two. The executive editor had requested the new OS on his honking Toshiba notebook -- a $6,000 box, the price of which I still can't fathom. IT had obliged and installed it. He'd happily used it for a day, taken the box home, and when he returned the next morning it was dead. Windows wouldn't boot. The conversation went something like:
IT tech: "So what did you do?"
Executive editor: "Nothing, it just didn't reboot."
IT tech: "It couldn't have just stopped for no reason. Did you install something?"
Executive editor: "No. Really. It just wouldn't reboot."
IT tech: [sigh] "OK. Fine. I'll fix it."
The next day, the tech returned the notebook, Win 95/98 fully reinstalled and working fine. The day goes well; no crashes. The next morning, the executive editor returns yet again with a $6,000 paperweight. I'm in his office for this part and had to work hard not to shoot coffee out my nose.
IT tech: "Come on, you had to have done something. Everything was working yesterday!"
Executive editor: "No, really. I didn't install a thing. I was just working and organizing."
IT tech [suspicious]: "What do you mean 'organizing'?"
Executive editor: "You know, just arranging folders so that I can find things more easily."
IT tech [still suspicious]: And which folders were you 'organizing'?"
Executive editor [annoyed]: "What does that matter?"
IT tech [equally annoyed]: "Trust me. Which ones?"
Executive editor: "My personal folder, the issue folders, the system folder --"
IT tech [squeezing his eyes shut]: "What did you do in the system folder?"
Executive editor [slowly dawning]: "Uh, well it was so messy. They had one folder for 16-bit DLLs and another for 32-bit DLLs, so I thought it'd be more efficient if they were all, you know, in a single folder."
I'm not sure who the tech wanted to kill more: the executive editor for what he did or me for sitting there, shoulders shaking, beet red, with my mouth clenched shut and tears coming out of my eyes.
Solution: Don't let your users become case studies for denying administrative access to local machines. Deny them administrative access to begin with. With senior execs, however, it still takes a social engineering degree to keep that rule enforced. That's a line of patter every IT guy needs to develop.
Moral: Even Microsoft computers don't suddenly quit for no reason. There's always a guilty user somewhere on the chain of causality. Find him early and you can avoid a large load of trouble down the line.
Stupid user trick No. 9: User populations are like bacterial ecosystems from distant planets
This particular stupid user trick hails from my days as an IT consultant, when our clients' CIO types, who had read about Shadow Copy, immediately wanted to engage on it. After all, in many cases they'd paid for it already, so they wanted it up and running right away.
Rolling out Shadow Copy was easy -- once we had Windows 2000 on every desktop and a working Active Directory domain controller. Then I used my vaunted writing skills to pen a short and sweet "Shadow Copy Advisory" memo and e-mailed it to every user. We followed that up with personal visits to all the managers in the company, explaining how the feature worked and what they needed to tell their employees about it.
The upshot was that My Docs was now being shadow-copied for every user, so all those folders they had on their desktops should be moved to My Docs to make sure everything got backed up to the network automatically.
In retrospect, I might as well have been asking them to bite off their own fingers for my amusement.
Everyone nodded excitedly, but nobody had any intention of using it.
To be fair, this was our fault as much as theirs. Assuming that users will put data exactly where they say they will is a newb mistake. But like true consulting newbs, we set up a backup policy to perform daily backups of "data" folders -- the shadow-copied stuff and the file shares users said they were going to be using -- and weekly snaps of the full server. Desktop backups relied entirely on users making My Docs their sole data dump.
Naturally, when a nasty virus hit and took out a large percentage of the desktops and simultaneously dumped two out of three servers, we found only 8 percent of users had been taking advantage of Shadow Copy. The rest were simply screwed. Worse, we found out that they had decided to build new "informal" network shares right off the server's hard disk (exactly where we hadn't expected them to), so those files were lost, too.
Solution: First, realize that you will never get away from users using their desktops as data storage. Ever. That's why it's called the "desktop." Whatever desktop backup strategy you employ, it needs to cover the desktop -- My Docs and any personal folders they've built themselves -- automatically. On the server side, you need a daily snap, so just thank God for block-level change technology.
Moral: Great ideas are fine, but you have to weigh them against every user's inherent resistance to change. User populations are like bacterial ecosystems from distant planets. You can't predict with very much precision how they'll evolve, so things like backup need to use the word "holistic" rather than "targeted."
Stupid user tricks 3: IT admin follies 
IT heroes toil away unsung in miserable conditions -- unsung, that is, until they make a colossally stupid mistake
More stupider user tricks: IT horror stories redux 
Idiot-proof your enterprise with these 10 hard-luck lessons of boneheaded IT miscues
Even dirtier IT jobs: The muck stops here 
More dirty tech deeds, done dirt cheap
The 7 dirtiest jobs in IT 
Somebody's got to do them -- and hopefully that somebody isn't you
True IT confessions 
Supergeeks fess up to some of the dumbest things they've ever done -- and the lessons they learned as a result
Stupid QA tricks: Colossal testing oversights 
The trick to nipping IT miscues is testing, testing, testing, as these hard-luck lessons in boneheaded quality assurance attest
Stupid hacker tricks, part two: The folly of youth 
Tech-savvy delinquents set the Net aflame with boneheaded exploits that earn them the wrong kind of fame
Programming IQ test: Round 2 
Acing last year's quiz was nothing. Only true hacker heroes will survive Round 2
Linux admin IQ test 
How much do you really know about the free OS?
Apple IQ test 
How much do you really know about the house that Jobs built?
Web IQ test 
Think you know all there is to know about the World Wide Web? Take our quiz to find out
Nine Real-Life IT Horror Stories