Nine Real-Life IT Horror Stories
Stupid user trick No. 3: Outsourcing Web development to the corner office
Here's a thought: Don't let the CEO design your company's customer-facing Web site just so he can save a few bucks, advises an IT consultant.
"We tried to sell a medium-sized company client on both a network install and a Web site design project," the consultant says. "We got the install contract, but the CEO figured he could design his site himself.
"When his general manager -- who was also his wife -- called us back in, she pulled the site up and it was hard not to wince. He'd used an open source editor with what looked like every freely downloadable template, fonts, and flashy widgets he could find. It looked like a teenage MySpace page."
[ More manager mishaps when meddling in IT can be found in "More stupider user tricks: IT horror stories redux " ]
Sure, the company's product information was now available on the Web, but the lack of customer-facing tools and analysis features did not bode well for the company's Web future.
"Even the Webmaster e-mail link didn't work," the consultant says. "Needless to say, the site was not attractive to customers, so Web revenue was low, and all those new and expanding Web marketing possibilities were crippled. The same CEO who built the site started spouting about how the rumors of e-commerce revenue were false."
Solution: Today, Web site design is cheap. From local outfits to eBay or Craigslist, the cost of a decently designed Web page has dropped from thousands of dollars per page to hundreds -- or less. Stop being penny-wise and pound-foolish.
Moral: Company Web sites can't be an afterthought investment, especially for small businesses. Not just an important face to your customer, your Web site is possibly the best way to analyze exactly who your customers are and how to sell to them. Treat it professionally, and you can leverage it for additional opportunities, including market research, customer analysis, and more.
Stupid user trick No. 4: Keep your enemies close, but your Linux talent closer
Going open source can save big bucks -- unless you leave your entire open source infrastructure in the hands of a single college intern, warns an admin at a small IT services firm.
"I finally find a small-business client who made the jump to Linux -- well, Linux and HP-UX due to a silo app they had to run for two big clients," the admin says. "Our new client had used his college intern to setup the basic network, but the kid had left for summer vacation a day earlier and suddenly the network was down. We were the first outfit in the phone book that didn't shy away from the phrase 'Debian on the desktop.'"
When the admin and his cohorts arrived, all the client's server lights were green, but nobody was connecting to anything and no one could log in to the system.
"We had to restore the servers from the ground up, which took about an hour. Everything was humming after that, so we took the time to sit down with the CEO and discuss plans for the network," the admin says. Stoked to locate someone unafraid to talk about open source software, the admin and his team got a little carried away shooting the bull with the CEO and stayed for more than an hour.
"As we were on the way out, the servers dumped again," the admin says. "Same story as before. Not wanting to lose our new penguin client, we rolled up our sleeves, restored the servers, and started digging for root cause."
What they found was a cron job set up off root.
"The cron 'cd'ed to a backup directory that tried to remove the files from a lengthy list of source directories, including several that didn't exist," the admin says. "Seems the kid had been changing these on the fly for some reason -- and he apparently liked doing sys admin as root. Academics."
Solution: Protect root access. Test your cron jobs. And maintain those server backup images.
Moral: Linux has definite benefits, but there's no denying that managing it requires a certain skill set. It's not something to trust entirely to an intern.
Stupid user trick No. 5: Facebook
Face it, even the most stringent social networking policies can't diffuse the ticking time bomb that is Facebook. Throw in a little Jäger, some IT naivete, and you're set for devastating corporate embarrassment.
"About a year ago, I get a call from a junior VP who's yelling at me that he's desperate and needs me to do a 'recall on Facebook,'" says one admin who wishes to remain anonymous. "I try and get a word in edgewise, but he's ranting about what crap Web technology is and why computer people can't just leave well enough alone and how everything was fine when we just used the telephone. Then he ends with, 'Is it done yet?'"
"'Is what done?'"
"'The Facebook recall, for @#$%'s sake.'"
Which gave our admin the obvious pleasure of asking, "What the @#$% is a 'Facebook recall'?"
As it turns out, the junior VP had updated his Facebook page from his phone while having a few drinks with some senior VPs and potential new clients.
"He stated that he'd dated one of the clients' wives and made some nasty comment about what she looked like naked," the admin says. "All his college buddies were on Facebook in their college group, and he knew her when they were both at school. Turns out that's where she met her husband, too, and he was on the college Facebook group as well, which the genius junior VP figured out when he got back to the table and started a conversation about Facebook with the potential clients."
As for the "Facebook recall," it appears that the junior VP thought updating his Facebook page was like sending an e-mail in Outlook.
"I told him how to lock down his page, but apparently that was a little late," the admin says. "We didn't get that account."
Solution: There really isn't one, other than trying to make sure your users have some idea of where the power of IT ends and the big, bad world begins.
Moral: The beauty of social networking is that it connects you with millions of other people. The danger of social networking is that it connects you with millions of other people.
Stupid user trick No. 6: Offshoring while under the influence of MBA
Whoever said offshoring was idiot-proof? After all, it often involves upper management -- potentially the worst IT offenders of all.
"We got a new CIO just before the bubble burst back in 2000," says D. Aubrey, who at that time worked at a Web services firm with a solid market position that it now had to defend against upstarts. "She was one of those MIS MBAs -- emphasis on MBA. All you press types started writing stories about the benefits of outsourcing around then, so she jumped on the trend, canned our Web dev team, and outsourced the whole shebang to an outfit in Mumbai that worked for $25 an hour."
The plan looked good on paper -- until you looked at the paper.
"We got a hold of the plan spreadsheet she presented to the CEO, and all she'd done was compare the cost of software tools and staff from in-house to out-of-house, so obviously the savings looked huge," Aubrey says.
"Then came the phone bill, which I think had quadrupled for that project," he adds. "And the security audit bill, since the data our Web dev guys were working with was quite a king-size waffle of personal customer data. And the hardware/services bill for moving our data out of the outsource outfit's internal datacenter -- which as far as we could tell was four servers in a closet somewhere -- and into a professional data hosting facility in Europe."
If that weren't enough, the final product -- a redesigned site -- "looked so average it might as well have been beige."
As it turned out, the new CIO had outsourced not just development, but project management and QA as well.
"There was literally nobody on our side proofing the work. They just kept showing her screenshots and she kept approving them until the day the redesign flipped," recalls Aubrey.
The volume of customer complaints about the site's new look and lack of functionality was put to a stop by the site itself, which crashed twice on the first day.
"The CEO ordered her to pull the plug and go back to the previous design," Aubrey says. "When they added it all up, she'd spent about 75 percent of the original project budget and had nothing to show for it."
"Normally, we'd have just snickered as they walked her out the door, but this was a down economy and this crap just cost us about five months of competitive advantage," he says.
The company never recovered. And though our intrepid offshorer was the first out the door, the rest of the crew followed by year's end.
"I'm not saying outsourcing doesn't work," Aubrey says. "But it takes a hell of a lot more planning than just comparing staff costs."
Solution: Go back and reread No. 3, and then realize that this submitter didn't go far enough. Web site development doesn't have to be isolated to be cheap.
Moral: If the Web site is a key revenue stream, do not entrust site development to a single exec.