Starting on Tuesday night, users of Verizon’s LTE (long-term evolution) network began reporting that they couldn’t connect to it with any device. The carrier said on Thursday morning that it had quickly found the cause of the crash and solved it. In between, the carrier’s most advanced mobile data network was unavailable.
The nationwide failure was a major embarrassment for Verizon, which markets itself as having a more reliable network than its main rival, AT&T. It was also a reminder that even the most sophisticated systems can fail.
Users of the HTC Thunderbolt, Verizon’s only LTE handset, were able to make phone calls or use data services via the carrier’s 3G EV-DO (Evolution-Data Optimized) or earlier 1x-RTT networks. Data devices such as the LG VL600 USB modem were supposed to switch over to 3G, but in at least one case, even that network was unavailable.
“If it happens once more I will go back to cable Internet/Wi-Fi usage in my home office, depend on hotel Internet and Wi-Fi hotspots when I’m not either at the office or home office … and cancel service on the Verizon Wireless network for laptop data usage in favor of these more reliable options,” said Verizon subscriber Bruce Ehrle, who normally works at home in Northern Virginia rather than drive to his office in Washington, D.C.
Ehrle previously used Verizon’s 3G service but was won over by Verizon’s promises of fast LTE performance. The new network was fast, but the LTE modem and software regularly have required restarts. Then, starting around 2 a.m. on Wednesday, Ehrle started to get a “Network not authorized” message when trying to start LTE, and when he tried to switch to 3G, he saw an “Unable to connect” error. He had to commute to his office and work late to have an Internet connection. Only after returning home late Wednesday night was he able to connect to the 3G network, Ehrle said in an e-mail interview.
Widespread Failure
Because the failure was so widespread, it probably was caused by a problem on the back end of Verizon’s network rather than any flaw in LTE radio technology, analysts said. The radio network is too distributed and was supplied by two different vendors, Ericsson and Alcatel-Lucent, said Phil Marshall of Tolaga Research. More likely, a configuration change or software update that was propagated throughout the network led to the failure, which could explain the wait of several hours before the error was corrected everywhere, he said.
The company was lucky that the outage occurred while the LTE network is still relatively small, analysts said. Verizon only introduced LTE in December, in 38 metropolitan areas. It plans to have the system up in 140 other areas by the end of this year and across its entire 3G footprint by the end of 2013.
“I think this is a wake-up call to Verizon,” said analyst Jack Gold of J. Gold Associates. The outage may indicate there is a single point of failure in Verizon’s LTE infrastructure, which would be a poor design, he said.
Verizon has been rolling out the new technology quickly while simultaneously integrating it with its existing 3G network, Marshall said.
“They’ve been extremely aggressive in the rate at which they’ve launched this network. And you’ve got to give them credit for that,” Marshall said. However, this does entail risk, he added. “I think we’re seeing the impact of that,” he said.
If the network suffers a similar failure soon, it might indicate there is something wrong with the underlying architecture. That would be a harder problem to fix, Marshall said.
Like failures of Research In Motion’s BlackBerry service in the past, the outage proves that no matter how highly praised a system may be, it can still fail, Gold said.
“People have gotten used to the notion of these things being up all the time … and that’s just not the case,” he said.
Stephen Lawson covers mobile, storage and networking technologies for The IDG News Service. Follow Stephen on Twitter at @sdlawsonmedia. Stephen’s e-mail address is stephen_lawson@idg.com