Many servers expose insecure management interfaces to the Internet through microcontrollers embedded into the motherboard that run independently of the main OS and provide monitoring and administration functions.
These Baseboard Management Controllers (BMCs) are part of the Intelligent Platform Management Interface (IPMI), a standardized interface made up of a variety of sensors and controllers that allow administrators to manage servers remotely when they’re shut down or unresponsive, but are still connected to the power supply.
BMCs are embedded systems that run inside servers and have their own firmware—usually based on Linux. They provide IPMI access through a network service accessible over UDP port 623.
Security researchers have warned in the past that most IPMI implementations suffer from architectural insecurities and other vulnerabilities that can be exploited to gain administrative access to BMCs. If attackers control the BMC they can mount attacks against the server’s OS as well as other servers from the same management group.
“For over a decade major server manufacturers have harmed their customers by shipping servers that are vulnerable by default, with a management protocol that is insecure by design, and with little to no documentation about how to make things better,” said Dan Farmer, a security researcher who has analyzed IPMI security over the past two years, in a paper published Wednesday. “These vendors have not only gone out of their way to make their offerings difficult to understand or audit but also neglected to supply any substantial defense tools or helpful security controls.”
Farmer, together with HD Moore, chief research officer at Rapid7 and lead architect of the Metasploit penetration testing framework, ran scans on the Internet in May and identified 230,000 publicly accessible BMCs. A deeper analysis revealed that 46.8 percent of them were running IPMI version 1.5, which dates back to 2001, and 53.2 percent were running IPMI version 2.0, which was released in 2004.
“BMCs running 1.5 only had a single simple problem, but it’s a whopper—nearly all server management ports had the NULL authentication option set, meaning that all accounts could be logged into without authentication,” Farmer said. “Furthermore virtually all BMCs also had the NULL user enabled, by itself a problem but not a serious one, but working in tandem with the first it means that you can login to pretty much any older IPMI system without an account or a password.”
About 90 percent of the BMCs connected to the Internet that were running IPMI 1.5 had the NULL authentication issue, Farmer said. The privileges associated with the NULL account vary from vendor to vendor, but in most cases they grant administrative access, and even when they don’t the mere ability to execute any kind of commands without authentication is a bad thing, he said.
In addition, IPMI version 1.5 doesn’t encrypt the connection between a user and a BMC so man-in-the-middle and other network attacks can be used to sniff passwords or hijack the connection. “You might think of the security of version 1.5 as something akin to using the old, reviled, unencrypted, and easily subverted telnet command for remote logins,” Farmer said.
IPMI version 2 includes cryptographic protection and supports 16 ciphers groups, but it has security issues of its own.
For example, the first cipher option, known cipher zero, provides no authentication, integrity or confidentiality protection, Farmer said. A valid user name is required for logging in, but no password is required. “The majority of servers have cipher zero enabled on their BMC by default, and HP [Hewlett-Packard], who is one of the largest, if not the largest vendor of BMCs, had apparently never allowed you to turn it off until just recently.”
The researcher found that around 60 percent of the publicly accessible BMCs running IPMI version 2 had the cipher zero vulnerability.
Another serious issue introduced by IPMI 2.0 stems from its RAKP key-exchange protocol that’s used when negotiating secure connections. The protocol allows an anonymous user to obtain password hashes associated with any accounts on the BMC, as long as the account names are known.
“This is an astonishingly bad design, because it allows an attacker to grab your password’s hash and do offline password cracking with as many resources as desired to throw at the problem,” Farmer said.
The analysis showed that 83 percent of the identified BMCs were vulnerable to this issue and a test with John the Ripper, a brute-force password guessing application, using a modest 4.7 million-word dictionary successfully cracked password hashes obtained from 30 percent of the BMCs.
“Of course numerous past studies have shown the effectiveness of what a serious attacker can do, and with orders of magnitudes faster speeds than I could muster on my consumer grade iMac,” Farmer said. “I’d say that even a well-chosen non-dictionary based password of a dozen characters or less is suspect.”
Farmer calculated that between 72.8 and 92.5 percent, depending on password cracking success rate, of BMCs running IPMI 2.0 had authentication issues and were vulnerable to unauthorized access.
“While a quarter of a million BMCs is only a tiny sliver of the total computing power in the world, it’s still an important indicator as a kind of canary in the coalmine,” because BMCs that are behind corporate firewalls share the same issues, Farmer said. “While management systems are often not directly assailable from the outside they’re often left open once the outer thin hard candy shell of an organization is breached.”
Farmer’s paper includes some recommendations for server administrators on how to mitigate some of the identified issues and better secure their BMCs, but the researcher concludes that ultimately the problem of insecure IPMI implementations will linger on for a long time.
“Many of these problems would have been easy to fix if the IPMI protocol had undergone a serious security review or if the developers of modern BMCs had spent a little more effort in hardening their products and giving their customers the tools to secure their servers,” Farmer said. “At this point, it is far too late to effect meaningful change. The sheer number of servers that include a vulnerable BMC will guarantee that IPMI vulnerabilities and insecure configurations will continue to be a problem for years to come.”