Frank Boldewin had seen a lot of malicious software in his time, but never anything like Rustock.C.
Used to infect Windows PCs and turn them into unwitting spam servers, Rustock.C is a rootkit that installs itself on the Windows operating system and then uses a variety of sophisticated techniques that make it nearly impossible to detect or even analyze.
When he first started looking at the code earlier this year, it would simply cause his computer to crash. There was driver level encryption, which had to be decrypted, and it was written in assembly language, using “spaghetti code structure” that made it extremely hard for Boldewin to figure out what the software was actually doing.
Analyzing a rootkit is typically an evening’s work for someone with Boldewin’s technical skills. With Rustock.C, however, it took him days to figure out how the software worked.
Because it is so hard to spot, Boldewin, a security researcher with German IT service provider GAD, believes that Rustock.C had been around for nearly a year before antivirus products began detecting it.
This is the story with rootkits. They’re sneaky. But are they a major threat?
In late 2005, Mark Russinovich discovered the most famous rootkit. A windows security expert, Russinovich was baffled one day when he discovered a rootkit on his PC. After some sleuthing, he eventually discovered that copy protection software used by Sony BMG Music Entertainment actually used rootkit techniques to hide itself on computers. Sony’s software wasn’t designed to do anything malicious, but it was virtually undetectable and extremely difficult to remove.
Sony’s rootkit became a major PR disaster for the company, which spent millions in legal settlements with users who were affected by the software.
Three years later, Russinovich, a technical fellow with Microsoft, still considers it the rootkit that caused the most trouble for computer users.
But the Sony rootkit presaged problems for the antivirus vendors too. The fact that none of them had even noticed this software for about a year was a serious black eye for the security industry.
Though they got their start on Unix machines years earlier, at the time of the Sony fiasco, rootkits were considered the next big threat for antivirus vendors. Security researchers explored the use of virtualization technology to hide rootkits and debated whether a completely undetectable rootkit could someday be created.
But Russinovich now says that rootkits have failed to live up to their hype. “They’re not as prevalent as everybody expected them to be,” he said in an interview.
“Malware today operates very differently from when the rootkit craze was going on,” he said. “Then… malware would throw popups all over your desktop and take over your browser. Today we’re seeing a totally different type of malware.”
Today’s malware runs quietly in the background, spamming or hosting its nasty Web sites without the victim ever noticing what ‘s going on. Ironically, though they are built to evade detection, the most sophisticated kernel-level rootkits are often so incredibly intrusive that they draw attention to themselves, security experts say.
“It’s extremely difficult to write code for your kernel that doesn’t crash your computer,” said Alfred Huger, vice president of Symantec‘s Security Response team. “Your software can step on somebody else’s pretty easily.”
Huger agrees that while rootkits are still a problem for Unix users, they’re not widespread on Windows PCs.
Rootkits make up far less than 1 percent of all the attempted infections that Symantec tracks these days. As for Rustock.C, despite all its technical sophistication, Symantec has only spotted it in the wild about 300 times.
“On the whole malware spectrum, it’s a very small piece and it’s of limited risk today,” Huger said.
Not everyone agrees with Symantec’s findings, however. Thierry Zoller, director of product security with n.runs, says that Rustock.C was widely distributed via the notorious Russian Business Network and that infections are most likely in the tens of thousands.
“Rootkits were used to hold access to a compromised target as long as possible and never had the goal to be spread widely,” he said in an interview conducted via instant message.
In the end, criminals may be avoiding rootkits for a very simple reason: They just don’t need them.
Instead of using sneaky rootkit techniques, hackers have instead developed new techniques for making it hard for antivirus vendors to tell the difference between their software and legitimate programs. For example, they make thousands of different versions of one malicious program, jumbling up the code each time so that antivirus products have a hard time spotting it.
In the second half of 2007, for example, Symantec tracked nearly half a million new types of malicious code, up 136 percent from the first half of the year. Security experts say that this situation is even worse in 2008.
“The stuff that we run across is not that complicated,” said Greg Hoglund, CEO of HBGary, a company that sells software to help customers respond to computer intrusions. “Most of the malware that’s out there nowadays… doesn’t even attempt to hide.”
For example, one of HB Gary’s customers was recently hit by a targeted attack. The bad guys knew exactly what they wanted and, after breaking into the network, swiped the information before the company’s incident response team could even get there, Hoglund said. “It was very clear that the attackers knew that they would get away with the data so quickly that they didn’t even have to hide.”