Malicious code that mysteriously found its way onto an internal virtual print server took out nearly 800 computers used by the city of Norfolk, Virginia, last week.
The code apparently was activated when workers shut down their computers, said Hap Cluff, IT director for the city of Norfolk. “It was triggered by the action of logging off,” he said. “
The code nearly wiped out the C drives of the 784 affected computers and essentially deleted the Windows operating system. The contents of the system folders on those machines, normally about 1.5GB in size, shrunk to 500 MB, he said.
Cluff believes the code may have been a “time bomb,” possibly loaded a long time ago but set to activate on a specific date. “Someone could have done it who knows how long ago,” he said.
Cluff’s team noticed that computers were taking longer than normal to shut down around 4:30 p.m. on Feb. 9. Those machines could not then be restarted. After investigating, his team discovered that a virtual print server was pushing out malicious code. The team pulled the virtual server offline, scrubbed it and reverted it to a previous instance of the print server software, he said.
The code did not propagate in any other way, so once the server was offline, the code ceased to spread. “It never propagated by any other device, only that one server pushing out this code, and all it did then was destroyed Windows,” he said.
Attacks that simply destroy computer systems are rare these days, according to Andre DiMino, a co-founder of the malware tracking group Shadowserver Foundation. “Years back, [malware] used to be much more destructive: capable of wiping a hard drive and toying with the boot sector,” he said via instant message. “This hearkens back to those days.”
Ultimately, the only computers affected were those that were shut down during about an hourlong window, after which Cluff’s team noticed the problem and identified and shut down the server.
The code also affected 11 servers. Cluff believes those servers were affected when engineers who happened to be working on them the day of the attack logged off. The code was activated on those servers when the engineers logged off.
Because engineers wiped the virtual print server, they don’t know much about the code or where it might have come from. “Normally, when you see something like that, your mode is to stop it. You’re not worried about taking a picture. Now we’re going to reconsider that response,” he said. Particularly with virtual servers, it’s relatively easy to take a snapshot that can later be analyzed to learn more about the malicious code and potential vulnerabilities, he said.
Cluff’s not particularly hopeful about finding the source of the code, even though federal authorities are now involved. The Federal Bureau of Investigation and even the Naval Criminal Investigative Service are investigating the incident, he said. The city is home to Naval Station Norfolk, a major U.S. Navy facility.
Experts from Symantec also visited the site and have not been able to discover the source of the code, Cluff said. Symantec confirmed the company was there but declined to further discuss the situation.
Though the code may have been installed a long time ago, it may not have been a time bomb, said Gary Warner, director of research in computer forensics at the University of Alabama at Birmingham. Warner doesn’t know the inside details of the situation but said it sounds like the malicious code could have been domain-leaked malware. That would work by someone first loading malware onto a machine or virtual machine, such as the virtual print server. Then, “as soon as a domain administrator logs into that print server, the malware captures the password and then has access to every machine on the network,” he said.
That would only work if the city gave administrators rights over all machines. The lesson learned could be that it’s wise to have several layers of administrative control, Warner said. “So whoever is casually logging in to install a print server should not be using the account that can do whatever it wants to the entire network,” he said.
The city’s IT department has by now re-imaged all the affected computers and has restored the 11 affected servers, Cluff said. It is still doing some work reloading applications for some users. It has rebooted a number of computers that were never shut down since the beginning of the incident, and they appear to be fine, Cluff said.
Robert McMillan in San Francisco contributed to this story.