(Reuters) – Thirty minutes into the crippling outage that hobbled the Nasdaq stock market on Thursday afternoon, stopping all trading in $5.9 trillion worth of U.S. equities, exchange officials had the problem fixed.
It was another two-and-a-half hours, however, before they were ready to flip the switch and turn the all-electronic market back on.
Most of the 191 minutes that the exchange was dark was spent in sometimes frantic conversation with scores of banks, brokers, investment companies and rival exchanges who wanted Nasdaq’s assurance that a restoration of trading would be orderly and would not lead to panic.
“We had to make sure all the exchanges were connected to us successfully and if the firms on the outside could get in,” said one Nasdaq official who spoke on condition of anonymity.
Meanwhile, banks’ trading desks were cautioning Nasdaq, operated by Nasdaq OMX Group Inc, not to rush to reopen, fearing that a restart full of technical errors would only sap more confidence from rattled markets, according to three sources at brokerages and banks who declined to be identified.
In the end, the reopening of trading did go relatively well.
Transactions first restarted at 3 p.m. EDT (1900 GMT) in a single microcap stock, Atlantic American Corp, a test case picked for its front-of-the-alphabet ticker. Twenty-five minutes later, the rest of the market opened, and, according to a Nasdaq statement, “The trading day finished in normal course.”
Shares of Nasdaq itself, which initially fell by more than 5 percent when trading resumed, recovered some of their lost ground to close the day 3.4 percent lower. The widely tracked Nasdaq Composite Index gained nearly 1.1 percent.
PERSISTENT GLITCHES
While the worst case scenarios may have been averted, the outage still stands as among the most serious in a series of recent technological failures to hit the U.S. securities business, including a software issue at the Chicago Board Options Exchange this spring that delayed the start of trading there for half a day.
It was also the latest black eye for Nasdaq, which in May agreed to pay $10 million, the largest penalty ever against a stock exchange, to settle SEC civil charges over its mishandling of Facebook’s initial public offering in 2012.
Late Thursday Nasdaq identified the problem as a “connectivity issue between an exchange participant and the SIP,” or Securities Information Processor – essentially the system that receives all traffic on quotes and orders for stocks on the exchange.
This problem “led to degradation in the ability of the SIP to disseminate consolidated quotes and trades,” Nasdaq said in a statement. “The cause of the issue has been identified and addressed.”
Whether it has been addressed to the satisfaction of regulators is another question. U.S. Securities and Exchange Commission Chair Mary Jo White called for a meeting of Wall Street leaders to help insure the “continuous and orderly” functioning of securities markets.
The incident “should reinforce our collective commitment to addressing technological vulnerabilities of exchanges and other market participants,” she said.
Thursday’s outage could well give White fodder to press ahead with new rules, proposed in March, that would hold exchanges, clearing agencies and certain “dark pool” trading venues more accountable for taking steps to prevent potential systems disruptions.
The rules, if adopted, would replace the current regulatory model in which exchanges rely on voluntary guidance known as “Automation Review Policies” to address security and stability issues with their systems.
The SEC rushed to roll out its proposal as a direct response to several high-profile software problems last year, including Nasdaq’s debacle with the Facebook IPO and a near-collapse at Knight Capital, a major Nasdaq market maker, as well as the two-day shutdown of the U.S. equity market due to Superstorm Sandy.
Exchanges, including Nasdaq and rival NYSE Euronext, parent of the New York Stock Exchange, pushed back, citing a number of concerns, including costs and an “unduly broad” requirement to disseminate information to member firms about certain incidents. “This requirement would likely have a chilling effect on communications,” they wrote.
The SEC agreed to extend the comment period on the rule, effectively delaying it, but late Thursday after this latest incident, White said she will push to get it completed.
AVOIDING THE ‘DOOMSDAY SCENARIO’
The problems surfaced at 12:14:03 p.m. (1614 GMT), when all traffic through Nasdaq stopped abruptly.
During the shutdown, trading of shares not listed on Nasdaq continued, but transactions could not be executed on the Nasdaq platform. Options trading was also halted. All rival exchanges agreed to halt trading of any Nasdaq-listed issue.
In Nasdaq’s Transaction Services division, it was quickly emergency mode as soon as the outage struck, the Nasdaq official said.
The team focused primarily on whether the exchange should reopen trading as the clock ticked toward the regular 4 p.m. close. They raised the “doomsday” scenario of reopening and not being able to execute a flood of orders.
“We asked all the what-if questions,” the official said.
As frustrated as Nasdaq customers were by the outage, they were more concerned that the exchange have all its ducks in a row before attempting to restart.
“The general feedback given to Nasdaq was, ‘Don’t rush back to fix it. It will be 10 times worse to come back online in a rush than to take time and get it right,'” said one source.
At Nasdaq, coordination was tight between the exchange’s technology staff, rival exchanges that had halted their trading of Nasdaq stocks, brokerage firm members and the firms’ major customers.
“It’s not an excuse, but anyone who understands the complexities of the trading and matching systems and the difficulties of having multiple exchanges operating and trading the same stocks can understand how difficult this was,” the Nasdaq official said. “It worked. It looks like the customers and the public did not get hurt.”
Not all will agree with that assessment, but it could be some time before the size of losses, if any, can be determined. And Nasdaq faces a reputational risk that could damage its listings business.
“If you’re advising companies to really go public, are you advising them to go public on Nasdaq?” said one source.
(Writing by Dan Burns and Jonathan Stempel; Additional reporting by Rodrigo Campos in New York and Sarah Lynch in Washington; Editing by Lisa Shumaker)