Rise of the Machines: The Concept of a Mass Technological Revolt
Ever have one of those days where everything just seems to fall apart? Not in a domino fashion, but completely different technological breakdowns that just happen to coincide with other breakdowns? I'm certain I'm not the only one to experience this particular phenomenon.
Case in point: The other day, I was dealing with a particularly quixotic network problem, intermittent random packet loss across across a VPN as noted by myriad Nagios warnings that would hit suddenly and inexplicably. Testing the fiber circuit itself revealed no problems, but intratunnel traffic was showing 50 percent packet loss. Was it the tunnel endpoint having problems? The internal switching? The fiber circuit itself? As soon as I'd dig in to test one particular subsystem, the problem would cease and all would be well, maybe for 10 minutes, maybe for 10 hours. Yeah, one of those.
[ Also on InfoWorld.com: Read Paul Venezia's classic, "Nine traits of the veteran Unix admin." | Or see if you qualify for the title of certified IT ninja. | Get a weekly dose of crazy but true IT stories from InfoWorld's Off the Record newsletter. ]
That alone is a real PITA, but for the day I spent fixing it, I was accosted by all manner of other technological failures. My phone suddenly refused to make calls and would drop calls where it hadn't done so before. I was working from a remote site, and my Internet connection that had been rock-solid for many moons suddenly went flaky. I had a wireless access point spontaneously bite the dust, gone for good. The coup de grâce was when the coffee machine quit -- seriously. I was either amazingly unlucky, there was an electromagnetic pulse released overhead (but I saw no mushroom clouds), or I was being subjected to a phenomenon I call mass technological revolt (MTR).
Inside the insidious MTR malady
In essence, MTR involves an initial technological problem (in this case, the intermittent network instability) that then somehow creates a localized technological war zone. Any number of pieces of technology maybe succumb to MTR during an event, generally in arcane and obtuse ways. In layman's terms, it's as if there was a rabble-rouser that convinced other devices to say, "Hey, screw this guy," and start breaking down. Like a mob, once this idea starts to spread, it catches like wildfire (as in last week's British riots). Before you know it, your coffeemaker flips you off and throws a heating element.
Interestingly, the initial and successive problems need not be physically local to the affected person, but they must be in his or her immediate purview. An example of this might be a server outage handled by an IP KVM switch that suddenly blows a power supply and thus eliminates all remote management capabilities to that downed server.
It's important to note that these events aren't terribly common, but they can be excruciatingly difficult to deal with, especially if the case is particularly severe, such as one that occurred to a consulting colleague a few years back. In the throes of trying to restore dozens of gigabytes of email to a corporate server that hadn't been properly built to begin with, his laptop screen went dark. The backlighting had failed, so there was still an image on the screen, but it was barely visible.
As he tried to triage the situation (he was on vacation with only that laptop, naturally), he tried aiming a desk lamp in a variety of angles so that he could issue the few commands necessary to begin the restoration. After moving the lamp around a few times, the bulb exploded, which caused a short that dropped the power in his room at a bed-and-breakfast. His laptop battery was low to begin with and had only maybe 10 minutes of life left.
Lacking a phone in the room, he used his cellphone to call the front desk in the hopes they could flip the breaker back, but nobody answered -- because the phone system was tied to the same breaker. Needless to say, his laptop battery died and the restore didn't start for another few frantic hours, once he'd found an external monitor to borrow. That's a classic example of MTR with the somewhat more unique element of occurring during a vacation.
Inside the rarer but very welcome SPR recovery
As infrequent as MTR may be, there's a flip side that may be even less common: spontaneous problem resolution (SPR). This is where everything that could go right does go right, for no apparent reason. I've personally been credited with curing a BSDi server with nothing more than the laying of hands, which may have been my first significant experience of SPR.
I did have another minor event occur recently, however. I had procured eight sticks of RAM to boost the memory in an older Cisco PIX 515E for the building out a temporary network at a new building, although I needed only one stick to work. The PIX had 32MB of RAM in a single DIMM, but I needed at least 64MB, so I had grabbed a handful of compatible DIMMs just to make sure I had my bases covered.
Out of the eight sticks I brought to the remote site, all eight failed for one reason or another, understandably causing some consternation. However, instead of the situation heading in the MTR direction, I happened to gaze upon an elderly Dell OptiPlex desktop that hadn't been used in years and was currently used as a doorstop. I ripped it open, grabbed a stick of RAM, and shoved it in the PIX -- it booted just fine with 256MB of RAM.
This narrowly fits the definition of SPR because there was absolutely zero expectation that this would work, and my thought of digging into that box was spontaneous. A more straightforward example would be someone slamming a rack door in frustration and causing a barely unseated cable or internal component to settle into its slot and fix the problem.
The sad truth is that for every example of SPR, there are a dozen examples of MTR. As of now, there's no rhyme nor reason for either phenomenon, but I bet it could make for one hell of a research paper, possibly even one based on psychoanalysis of those unfortunate souls caught in the midst of an episode of MTR. Just make sure that notes are taken with a pencil and paper, since it's apparently contagious.
This story, "Rise of the machines: The concept of mass technological revolt," was originally published at InfoWorld.com. Read more of Paul Venezia's The Deep End blog at InfoWorld.com. For the latest business technology news, follow InfoWorld.com on Twitter.