The Story Behind San Francisco's Rogue Network Admin

Laying the Blame

My source doesn't appear to harbor any ill will towards Childs for this situation, and even believes that the city may be worse off with Childs out of the picture, and that some of the blame should be shouldered by Childs' superiors.

"It's a real shame. The city is losing a good network engineer -- probably the best, technically, that they've ever had. Ultimately he has no one to blame but himself, but it's too bad his superiors weren't better about establishing and enforcing policies about authentication, backups, auditing, cross-training, and separation/rotation of duties.

"You'll note the papers have referred to the new information security manager. It's only been a month or so since the City even had an information security policy, and even that is a bare, unmodified template from CCISDA that's awaiting discussion and alteration by a committee that hasn't been formed yet. (When I asked Terry if we could get a copy of the City's network security policy some months ago, he told me, "I've been trying to get them to approve one for years. I've written ones up and submitted them, but they don't want to do it, because they don't want to be held to it.")"

He also points out that by forcing the issue, the city may have significantly reduced its ability to use and control its own network.

"The one impact they haven't mentioned is that Terry was one of only two engineers assigned to special projects and to do major routing changes and perimeter firewall configuration. The service level, even after they regain control of the network, is going to be way down, until they can fill his mighty big shoes."

My source had many good things to say about Childs, but did not shy from negative comments, noting that Childs has a bad temper and can be very defensive.

"As for Terry's character, I can imagine this happening. He takes great personal and professional pride in his work -- to a fault. He can be very defensive if someone suggests there's something wrong with the way his network is set up, and that's been a problem for us (as his customer) a couple of times. Terry has a bad temper.

"He's the sort of person who, while his bile is up, won't budge an inch -- and then will call you a couple of hours later and acknowledge that maybe your suggestion was right, after all, or maybe here's an even better way to handle things."

The Inner Sanctum

Later in the e-mail, my source offered some insight into what may be at the core of the issue: Childs was so paranoid about the security of the network that he even refused to write router and switch configs to flash, which would mean that if the device was powered off, all configurations would be lost.

"At one point he was concerned about the security of the FiberWAN routers in remote offices, so he had them set up without saving the config to flash. "If they go down, I'll get alerted, and connect up to them and reload the config." Great, except we have power outages all the time in this city, some of those devices aren't on UPSs, and what happens if you're on vacation? And what about the 15 to 60 minutes it might take you to connect up and reload? He eventually conceded and (ahem) decided that disabling password recovery was sufficient security."

If Childs did this with some or all of the switches and routers comprising the FiberWAN network, then password recovery without significant network disruption becomes a bigger problem. Without first-hand knowledge of the state of those routers and switches, there's no good way to know, unfortunately.

If the details given to me in this e-mail are accurate, it would appear that this case is not nearly what it seemed originally. Perhaps it comes with the pressure and responsibility of the job, or the belief that the network they've built is simply too complex for mere mortals to comprehend, but it's not uncommon for highly skilled network administrators to become overprotective of their networks, or for networks of significant size to become an extension of the person who built them.

It certainly appears that Terry Childs believed San Francisco's FiberWAN network was his baby, and that by refusing to allow others to access the inner sanctum was in the best interests of the city, the citizens, and perhaps most importantly, himself.