For the second time in two weeks, bad networking information spreading from China has disrupted the Internet.
On Thursday morning, bad routing data from a small Chinese ISP called IDC China Telecommunication was re-transmitted by China’s state-owned China Telecommunications, and then spread around the Internet, affecting Internet service providers such as AT&T, Level3, Deutsche Telekom, Qwest Communications and Telefonica.
“There are a large number of ISPs who accepted these routes all over the world,” said Martin A. Brown, technical lead at Internet monitoring firm Renesys.
According to Brown, the incident started just before 10 a.m. Eastern Time on Thursday and lasted about 20 minutes. During that time IDC China Telecommunication transmitted bad routing information for between 32,000 and 37,000 networks, redirecting them to IDC China Telecommunication instead of their rightful owners.
These networks included about 8,000 U.S. networks including those operated by Dell, CNN, Starbucks and Apple. More than 8,500 Chinese networks,1,100 in Australia and 230 owned by France Telecom were also affected.
The bad routes may have simply caused all Internet traffic to these networks to not get through, or they could have been used to redirect traffic to malicious computers in China.
While the incident appears to have been an accident, it underscores the weakness of the Border Gateway Protocol (BGP), a critical, but obscure, protocol used to bind the Internet together.
BGP data is used by routers to tell them how to route traffic over the Internet. Typically smaller service providers “announce” BGP routes for the networks they control, and that information is ultimately centralized and then shared between larger providers. That’s where the problems started on Thursday. For some reason, IDC China Telecommunication announced routes for tens of thousands of networks — about 10 percent of the Internet. Typically this small ISP announces about 30 routes.
That bad information was then accepted by the larger China Telecommunications, which shared the data with other major ISPs. Within minutes the bad data had spread around the globe.
ISPs may have accepted the hijacked route information, but that doesn’t necessarily mean that a lot of Web surfers got redirected. It’s common for routers to learn several BGP routes, and then route traffic to what they consider the best route. Often they choose the shortest route available. So most routers in the U.S. would have routed traffic to Apple’s servers, for example, instead of IDC China Telecommunication.
“I don’t believe there was really widespread impact, but some people must have noticed it,” said Andree Toonk, founder and lead developer of BGPmon.net, a BGP monitoring service that has been tracking the situation. “Many people probably didn’t prefer the path because they had a better path.”
There may have been more disruptions in Asia, however, where the IDC China Telecommunication route would have seemed shorter, but users were definitely affected, Brown said. “We saw routers in Belgium, Indonesia, Portugal, Thailand and the U.S. — to name a few — which were selecting these false routes,” he said.
Arbor Networks Chief Security Officer Danny McPherson believes that a large number of users were probably affected, even if only for a short time. Worse, the tens of thousands of bad routes may have just been a cover for a single targeted attack, he said.
IDC China Telecommunication could not immediately be reached for comment Thursday.
Because so many Chinese networks were also disrupted by the incident, security experts believe that it was probably unintentional.
This isn’t the first time that bad BGP routes have caused problems on the Internet.
Two weeks ago a bad BGP route from China leaked out and redirected some Chilean Internet traffic to a root DNS (Domain Name System) server in China. And two years ago, bad BGP routing information from Pakistan caused YouTube to temporarily disappear from the Internet.
Speaking about Thursday’s incident, OpenDNS CEO David Ulevitch said, “It’s not clear whether it’s deliberate, but it’s serious.”
“These things highlight just how fragile BGP is,” he added.