Security

How to Root Out Rootkits

Rootkits of All Evil

"The use of rootkit technologies is prevalent in the malware families our filters are picking up today," says Christoph Alme, Secure Computing's antimalware team lead. "Most commonly these tend to be spambots. Recent examples include Srizbi and Rustock."

Detected in the wild in 2007, Rustock.C spreads like a virus to infect kernel drivers, uses polymorphism (self-changing) to avoid signature detection, loads and hides beneath Microsoft's trusted system driver, and includes a back door Trojan to open and hide two-way communications channels over Port 80.

When analyzed at Rootkit.com this year, Rustock.C was called the "most powerful rootkit ever found under Windows" because of these and other advanced hiding features. The analysis went on to predict that Trojans (back doors) and rootkits will ultimately blend into one malware

By combining such hiding technologies, rootkits such as Rustock.C can easily cloak a bot's existence not only from the system, but from the network, where monitoring for suspicious machine behaviors is the last line of defense in detecting the possible presence of rootkit-infected systems.

"Companies need to keep Port 80 open so their employees can use the Internet. Some malware uses that channel to piggyback HTTP traffic," Alme says. "HTTP traffic mainly goes inbound [rather than outbound] over this port, so you need to train your filters to scan outbound HTTP traffic with your network gateway appliance."

Malicious traffic can also piggyback on accepted outbound traffic -- for example attaching to outbound DNS packets. So Alme also recommends monitoring these types of outbound channels for bursts of traffic, large files and other anomalies that might indicate remote control commands are being sent and received.

Traditionally, detecting a rootkit on a system can be even more difficult than detecting rootkit-hidden traffic on the network, because the rootkit always had as high or higher privilege than antivirus software, Dalton says.

However, VMware's recent addition of antivirus support with their new VMSafe extensions allows antivirus products to run with VMM (virtual machine monitor, aka hypervisor) protection, at higher privilege and visibility into the kernel.

"It's always been a game of cat and mouse with antivirus looking for rootkits and rootkits looking for antivirus, so the rootkit can take control of the security software and continue controlling the infected computer," Dalton says. "Now, by putting security in the Virtual Machine Manager, a kernel rootkit can't even find the security to disable it."

Subscribe to the Security Watch Newsletter

Comments