An Internet traffic routing error by a rival ISP is suspected of causing up to 3 million Telstra customers to be knocked offline for around 35 minutes on Thursday afternoon.
Telstra said it lost connectivity to its international data network at 1:50 p.m. Customers could not access websites with either their mobile phones or fixed services. Those customers then over-accessed domestic websites, causing more problems, according to a company statement.
“The vast majority of customers have been progressively coming back on line since 2.25 [p.m.] this afternoon,” the company said. “Telstra’s engineers are continuing to monitor the network and identify the cause.”
Simon Kong, a network engineer based in Perth, said it appeared that all of Telstra’s traffic was routed through Dodo, another ISP, by mistake. It appeared that Dodo announced new BGP (Border Gateway Protocol) routes, which are used to configure routers used by network providers to exchange traffic.
Dodo announced incorrect BGP routes, which should have been filtered by Telstra but weren’t, Kong said. All of Telstra’s traffic then flowed into Dodo’s network, which was overloaded.
Configuring BGP routes incorrectly can cause major problems. ISPs are only supposed to announce routes for blocks of IP addresses that have been assigned to them. Other ISPs are also supposed to double-check those announcements to ensure they are correct.
In 2008, Pakistan Telecom made an error with BGP after Pakistan’s government ordered that ISPs block YouTube. Pakistan Telecom misconfigured BGP, which made YouTube unreachable to users worldwide.
Send news tips and comments to jeremy_kirk@idg.com