Heading into the heart of hurricane season 10 months after Sandy slammed the New York metropolitan area, Wall Street has had time to reassess and revamp backup plans.
Sandy’s storm surge caused the first weather-related, 48-hour closure of markets since the Great Blizzard of 1888.
“You could say Sandy forced the hand of the trading firms,” said David Weiss, an analyst with the consulting firm Aite Group.
“A confluence of trends” that lend themselves to overall system resiliency was, however, already under way, Weiss added. The commoditization of server hardware suitable for trading and back-office systems, for example, has helped give rise to third-party data centers that can help financial-sector companies reduce risk.
Meanwhile, researchers sound increasingly urgent alarms about the risks posed by climate change. What was once considered a safe increase in water temperature now appears to “commit” the area to a sea level rise “nearly twice the height of Hurricane Sandy’s peak storm surge at The Battery in New York City,” wrote Benjamin H. Strauss in a recent analysis published in the Proceedings of the National Academy of Sciences.
Data centers serving the metro area’s financial services business are not going anywhere, however. To meet the low-latency network requirements of certain types of trading, data centers need to be near the exchanges and trading firms that use their computing resources.
Even though the New York Stock Exchange maintains a trading floor at 11 Wall St. and major market participants have offices and headquarters in Manhattan, data centers are located across the Hudson River in New Jersey. Exchange data centers are in Carteret, for Nasdaq; Mahwah, for NYSE Euronext, including the Arca all-electronic exchange; Secaucus for Direct Edge; and Weehawken, for BATS.
Data center operators including Savvis, Equinix and Telx, which have co-location agreements with the exchanges and telecom companies, have facilities scattered throughout Manhattan and the New Jersey metro area.
A beacon of resiliency
Certainly major storms present risks far greater than data outages: Deaths linked to Sandy numbered more than 50 in New York and more than 30 in New Jersey. While most people in the area were concerned about the loss of lives, homes and power, there was grumbling in financial circles about the halt to trading and outages in communications systems.
Depending on who’s talking, the market closure for Sandy was a responsible, communal decision, or a sign that trading systems are not as resilient as they should be.
“People look to the New York Stock Exchange as being the symbol of American capitalism, and to see the exchange go down for two days without an adequate backup plan is very, very unfortunate,” said Arthur Levitt, former chairman of the U.S. Securities & Exchange Commission, on a Bloomberg Radio interview.
The NYSE, the other exchanges and data centers firmly reject the idea that they were not ready for the storm.
The closure was an industry decision, not one made by the NYSE only, exchange officials say, adding that the risks involved in having staff trying to get to work was of primary concern. The NYSE’s backup plan to trade off-floor on the Arca all-electronic exchange was tested and ready to be implemented, they say.
However, major trading firms reportedly voiced reservations about whether their systems were quite ready to trade on Arca with the NYSE trading floor closed. Many also felt that they would have to dispatch their own IT workers to midtown New York or Wall Street offices in order to manage the change-over, which, during this time of crisis, would not be desirable.
“There are lots of moving parts and a tremendous amount of coding that goes into these networks,” said Alex Tabb, partner and chief technology analyst at consulting firm Tabb Group. “Even if you are connected to Arca and doing most of your trading there, a closure of the trading floor could have an effect.”
Trading system algorithms are written to automatically check different exchanges to route an order to the market offering the best price, Tabb explains. If they can’t check the trading floor as they normally should, they may not execute an order, even if other exchanges are open.
Marc FerrantiGiven the possibility that flooding and trading system glitches might have prevented at least some major market participants from trading, or taken them offline in the middle of the storm, the decision made before Sandy hit to close the exchanges was understandable, Tabb notes.
“What you don’t want is a disorderly closure,” Tabb said.
In any case, had the markets been open the day after Sandy, Nasdaq would have been ready, officials say.
The company’s headquarters in downtown Manhattan were deemed uninhabitable, and the company had to failover to remote systems, but like the NYSE, Nasdaq says its data center remained up and running.
“We didn’t effectively close,” said Lou Modano, senior vice president of the exchange’s Global Infrastructure Services, via a statement sent by a spokesman. “We complied with the industries’ request and market participants. We had no issues with our facilities or flooding—we were always ready to go.”
The Wall Street neighborhood itself was hit hard. Two Citigroup buildings lost power, with the 111 Wall St. offices sustaining severe flooding. Meanwhile, diesel pumps at InterNAP’s data center at 75 Broad St., several blocks from the NYSE, were destroyed by flooding. A big Verizon switching station at 140 West St. near the World Trade Center was knocked out.
On balance, though, considering that many Wall Street firms are in or near FEMA (Federal Emergency Management Agency) Flood Zone A areasand many New Jersey data centers are in marshy lowlands, remarkably few trading-related facilities had severe operational problems during the storm.
But it’s easy to imagine that a slightly higher storm surge would have caused much more water damage. Floodwaters breached the New Jersey Turnpike in the area of the Nasdaq’s Carteret data center, and though exchange officials say there was no issue of flooding at the data center, staff at a hotel a few hundred yards away said floodwaters were so high they were trapped at work during the height of the storm.
Although Equinix officials say their data centers in Secaucus never stopped operating, they are located in neighborhoods flooded by the storm, according to data mapped out by WNYC’s Data News team.
The exchanges have backup data centers, but distance and connectivity issues pose trading-system hurdles for those facilities that are far outside the metro area. Nasdaq’s backup data center in Ashburn, Virginia, for example, is too far away to enable for the low-latency transactions needed by the high-frequency trading (HFT) and algorithm-based order routing that more local processing can offer, according to data center professionals.
For example, roundtrip latency from Telx’s new NJR3 data center in Clifton, N.J., to metro area exchanges is under 400 microseconds, “fast enough for algorithmic order routing,” according to Shawn Kaplan, general manager of financial services for Telx.
Latency time for collocated systems are even faster. One-way latency from Savvis’ NJ2 data center in Weehawken to exchanges collocated within the facility (with data speeds of 1G bps) is 64 microseconds, according to the firm’s website.
Even lower latency times, measured in nanoseconds, are required by HFT.
In contrast, a new long-distance microwave link launched by Nasdaq and the CME Group, which owns the New York Mercantile Exchange, offers one-way latency of under 4.25 milliseconds between CME’s data center in Aurora, Illinois, and Carteret. That’s more than an order of magnitude slower than latency times achieved among data centers in the metro area (one millisecond equals 1,000 microseconds, one microsecond equals 1,000 nanoseconds). The link will be used for market data, not order routing.
A tremendous amount of effort, then, goes into making sure that metro area data centers and networks are disaster-resilient.
The NYSE, for example, has proposed a rule change to the Securities and Exchange Commission that would streamline the process of using Arca as a backup when the trading floor is closed, due to a major emergency or even a minor one such as a water-main break.
The NYSE’s proposed “Print to P” plan essentially allows Arca to mark opening and closing quotes as if they came from the NYSE trading floor itself. This would be done so the quotes can be ingested by trading firms’ systems to recognize this new source of data without any reconfiguration.
With the NYSE, many firms look for the opening transactions, marked as N (for the NYSE), which kick off their daily trading algorithms. Other firms require the NYSE closing quotes, also marked as N, for calculating net access values for mutual funds. When the NYSE is down, opening and closing quotes will be marked as both N and P (which stands for an Arca exchange). Intra-day quotes on Arca will simply be marked as P.
“The Exchange believes that the proposed rule change offers a practical solution that will be compatible with most market participants’ current system coding, which will allow the proposed rule change to be quickly and efficiently implemented and avoid the costs and delays associated with system reprogramming,” according to the NYSE’s SEC filing.
Market makers such as Citigroup are working with the NYSE on the Print to P plan.
Lessons learned from the terrorist attacks of Sept. 11, 2001, have helped Wall Street prepare for storms as well. After 9/11 knocked out telecom systems, for example, NYSE Technologies created its own Secure Financial Transactional Infrastructure network.
Who has the power?
Power is also key in data center construction. The NYSE’s Mahwah data center, which opened in 2010, sits on two power grids as does Telx’s new NJR3, which opened in June. In addition, data centers all have their own generators. NJR3, for instance, has 12 3-megawatt generators.
Perhaps the biggest lesson from Sandy itself was that businesses have to have backup plans for people who lack transportation, noted Weiss, recalling gas station closures. “It doesn’t do much good to have data centers up and running if no one else can get to them,” he said. The “virtualization of not just technology, but of staffing,” with the Internet enabling remote work, is important for any backup plan, Weiss said.
“Systems including our CRM (customer relationship management) tool—how we communicate with our customers and our other facilities—that’s all Web-enabled as are all our inventory systems, our power monitoring and control systems,” said Michael Terlizzi, executive vice president for operations at Telx.
Financial sector businesses in the area also realize, however, that they sometimes also need to move people around and need to have plans in place to do that. Citigroup, for example, moved quickly to accommodate the employees from 111 Wall St. at other locations in the area, and provided transportation.
Sandy hit just 14 months after Hurricane Irene slammed the Northeast. In a sober-minded mood while inspecting the World Trade Center after Sandy, New York Gov. Andrew Cuomo said “Anyone who says there is not a change in weather patterns is denying reality.”
With climatologists predicting that global climate change will cause more frequent flooding of the metro area, few doubt that another big test of Wall Street’s storm preparedness will come, quite possibly sooner rather than later.
Updated at 12:53 p.m. PT with a video from IDG News Service.