A week after the Nasdaq’s 3 hour shutdown in August 2013 the exchange published Preliminary Findings on the outage. I’ve extracted some sentences:
On August 22, the Securities Information Processor (SIP) received more than 20 connect and disconnect sequences from NYSE Arca, each of which consumed significant resources. Available capacity was further eroded as the SIP received a stream of quotes for inaccurate symbols from NYSE Arca, and generated quote rejects..
The confluence of these events vastly exceeded the SIP’s planned capacity, which caused its failure and then revealed a latent flaw in the SIP’s software code.
This latent flaw prevented the system’s built-in redundancy capabilities from failing over cleanly, and delayed the return of system messages.
My paraphrase—NYSE Arca screwed up, sent us a bunch of bogus messages, overloaded our server, and then, ahem, a bug in our code brought the whole system down.
In my 35 years working in high tech there were several times when I managed multi-million line software projects. I can empathize with Nasdaq’s problems. If you haven’t worked on large software assemblages, there is no way you can appreciate the complexities involved.
The computerization of stock exchanges has been underway since the 70’s, but in the last fifteen years the technology has moved from computers aiding humans, to the computers running the show and the humans observing. You can lament the situation all you want, but as Dorothy observed “… we’re not in Kansas anymore.”
The graph below shows the decrease in time required to process an order over time.
I’ve been thinking about the results and implications of this shift. Some of my observations:
- Defects and failures are inevitable
- With 13 interconnected USA exchanges, multitudes of dark pools, and thousands of different order types there is no way that you can have a defect free environment. Most flaws will be subtle, some of them may give High Frequency Traders an unfair advantage, and some will bring the market down.
- Testing is not a panacea
- No matter how much testing you do, you won’t eliminate all the defects—simply because the humans designing the tests can’t anticipate all possible problems. Testing is certainly worth doing, especially if it can be automated so it can be easily repeated after changes are made, but you are truly naïve if you think a system of any size can be tested for all possible eventualities.
- Get used to flash crashes—trading is unstable
- The continuous double sided auction used on our stock exchanges is inherently unstable in some market environments. When markets panic buyers step back, stop loss orders trigger, sellers become desperate, and short sellers feed the flames. When humans are directly involved the crashes take longer (the 22% decline on October 19, 1987 took all day), but don’t pretend crashes are a new thing. Flash crashes are just sped up versions of what we’ve always had
- Use orderly shutdowns (circuit breakers) on all securities
- Since crashes / system shutdowns are inevitable, it makes sense to put in automatic protective software to provide temporary halts, and if serious enough shutdown the exchanges in an orderly fashion. It’s not enough to put protection on the major indexes—we need circuit breakers on every security that is traded. The SEC approved the limit up/limit down program, currently in pilot testing, looks like a step in the right direction. Of course we should be making the exchanges as robust as possible, but we need that last line of defense. If your toaster shorts out in some horrible unanticipated way you need a circuit breaker to prevent your house from burning down.
- Indexes rule
- They used to just be metrics, but the rise of the computer has enabled indexes like the S&P 500 and the Dow Jones Industrial Average to become securities in their own right. Now index futures and index based ETFs dominate trading on the exchanges. Arbitrage opportunities between the indexes and their component parts tends to make the underlying securities move in lock step with the index—especially on down days. This increase in correlation makes it more difficult to diversify portfolios—sector rotation strategies for example have suffered. Some traders were surprised when the S&P 500 futures market continued trading during the Nasdaq August 2013 halt despite the sidelining of Apple, Google, and other high market cap stocks. But this is an old trick for them—the futures trade nearly 24 hours per day, they no longer need stocks trading to be active for them to make a market.
Skynet was the computer system in the Terminator movies that became self aware and then attempted to exterminate the human race. I don’t think the stock exchange computers are self aware yet, but I suppose it’s possible the NYSE Arca computers were having a little fun at Nasdaq’s expense…