High frequency trading

Nanex released a video showing the results of half a second of worldwide high frequency trading with Johnson and Johnson stock. I simply sped up the footage to get a better feel of what it looked like. Blow Your Mind.

This practitioner’s summary is designed to help non-high frequency investors understand the current state of high frequency trading (HFT) mainly in the U.S. It is based on the “On The Impact and Future of HFT: White Paper,” by Khaldoun Khashanah, Ionut Florescu, and Steve Yang, all of the Stevens Institute of Technology (Financial Engineering Division), and should be read in conjunction with it.

In March 2011, the Markets Committee established a Study Group to conduct a fact-finding study on high-frequency trading (HFT) in the foreign exchange (FX) market, with a view to identifying areas that may warrant further investigation by the central banking community. This initiative followed from a number of previous discussions by the Committee about factors contributing to changes in the structure of the global FX market.

The Study Group was chaired by Guy Debelle, Assistant Governor of the Reserve Bank of Australia. The Group drafted an interim report for review by the Committee in May 2011. The finalised report was presented to central bank Governors at the Global Economy Meeting in early September 2011, where it received endorsement for publication.

The subject matter of this report is clearly part of the core expertise of the Markets Committee, which has a long-standing interest in the structure and functioning of the FX market. I hope this report will serve as a timely input to the ongoing discussion about the impact of technological changes, including the rise of algorithmic trading in general and HFT in particular, on the functioning and integrity of financial markets. The FX market focus of this report should also be a valuable complement to a discussion that has so far been based mostly on developments in equity markets.

Classic asset pricing theory assumes prices will eventually adjust to and re°ect the fair value,the route and speed of transition is not speci¯ed. Market Microstructure studies how prices adjust to re°ect new information. Recent years have seen the widely available high frequency data enabled by the rapid advance in information technology. Using high frequency data,it's interesting to study the roles played by the informed traders and noise traders and how the prices are adjusted to re°ect information °ow. It's also interesting to study whether returns are more predictable in the high frequency setting and whether one could exploit limit order book dynamics in trading.

Broadly speaking, the traditional approach to statistical arbitrage is through attempting to bet on the temporal convergence and divergence of price movements of pairs and baskets of assets, using statistical methods. A more academic de¯nition of statistical arbitrage is to spread the risk among thousands to millions of trades in very short holding time, hoping to gain pro¯t in expectation through the law of large numbers. Following this line, recently, a model based approach has been proposed by Rama Cont and coauthors [1], based on a simple birth-death markov chain model. After the model is calibrated to the order book data, various types of odds can be computed. For example, if a trader could estimate the probability of mid-price uptick movement conditional on the current orderbook status and if the odds are in his/her favor, the trader could submit an order to capitalize the odds. When the trade is carefully executed with a judicious stop-loss, the trader should be able to make pro¯t in expectation.

In this project, we adopted a data-driven approach. We ¯rst built an "simulated" exchange order matching engine which allows us to reconstruct the orderbook. Therefore, in theory, we've built an exchange system which allows us to not only back-test our trading strategies but also evaluate the price impacts of trading. And we then implemented, calibrated and tested the Rama Cont model on both simulated data and real data. We also implemented, calibrated and tested an extended model. Based on these models and based on the orderbook dynamics, we explored a few high frequency trading strategies.

The issue of high-frequency traders who cancel a lot of their orders seems to have been in the news a bit recently, so let's kind of reason it out from first principles.

There are different kinds of "high-frequency traders," but many of them are in the business of what is called "electronic market making." A market maker continually offers to buy or sell stock, whichever you want. So the market maker places an order on the stock exchange to buy 100 shares of XYZ stock for $9.99,[SUP][/SUP] and places another order to sell 100 shares of XYZ for $10.01, and you can come to the stock exchange and immediately sell to the market maker for $9.99 or buy from it for $10.01. If you sell for $9.99, and then someone else comes along and buys for $10.01, the market maker collects $0.02 just for sitting in between the two of you for a little while. Whether the market maker is necessary, or worth the two cents, is a hotly and boringly debated question,[SUP][/SUP] though it should be said that market makers have existed in the stock market for a long time and that electronic market makers do the job waaaaaaaaaay cheaper than their human predecessors.[SUP][/SUP]

Okay. How does the market maker decide what orders to send to the stock market? Particularly, how does it decide on a price? Well, remember, it's a little computer program, and its little computer brain wants to balance supply and demand. If the market maker quotes $9.99 / $10.01 for a stock, and lots of people buy for $10.01 and no one sells for $9.99, then that probably means that the market maker's price is too low and it should raise the price. There's a simple dumb schematic way to do this, which is:

• Put in a bid on the stock exchange to buy 100 shares for $9.99, and an offer to sell 100 shares for $10.01.
• If someone buys 100 shares (for $10.01), raise the quote to $10.00 / $10.02. (But if someone sells 100 shares for $9.99, lower the quote to $9.98 / $10.00.)
• Repeat, moving the quote up by a penny each time someone buys, and down by a penny each time someone sells.

If the stock price is relatively stable, this is a more or less functional strategy.[SUP][/SUP] But notice how many orders the market maker cancels. It starts by putting in two orders, one to buy 100 shares for $9.99, and another to sell 100 shares for $10.01. Then (say) someone buys 100 shares for $10.01. The market maker's sell order has executed. Now it wants to move up its quote, so it cancels its $9.99 buy order and enters two more orders (a $10.00 buy and a $10.02 sell). Then (say) someone sells 100 shares for $10.00. Now the market maker's buy order has executed, it wants to move the quote back down, it cancels its sell order, and submits new buy and sell orders.

So this simplest stylized market making strategy cancels 50 percent of its orders without ever executing them.[SUP][/SUP] Is that because they are a scam, or phantom liquidity, or spoofing? No, it is because it is a market maker, and it moves its prices to respond to supply and demand.

Now let's add one important real-world complication to this stylized model. Our imaginary market maker sent its orders to "the stock exchange." But there are actually 11 stock exchanges in the U.S., plus any number of dark pools and other venues. Different investors route their orders differently to different exchanges, and if you want to do a lot of business as a market maker you have to make markets in the same stock on multiple exchanges. But the same economic principles apply: If the market maker sets a price, and people keep buying, it should raise the price. It's just that now it has to raise the price everywhere. So the model becomes:

• Put in 11 bids to buy 100 shares for $9.99, and 11 offers to sell 100 shares for $10.01, one on each of the 11 stock exchanges.
• If someone buys 100 shares on one of the exchanges, raise allthe bids to $10.00, and allthe offers to $10.02.
• Etc.

Now every time the market maker executes one order and moves its price, it cancels 21 orders: the 11 orders on the other side, and the 10 orders on the same side at different exchanges. It now has a cancellation rate of 95.5 percent. Is that because its orders are a scam, or phantom liquidity, or spoofing? I have set up the answer to be: No, it is because it is a market maker, and it moves its prices to respond to supply and demand, and our fragmented market means that it has to do that in a bunch of different places all at once.

But to be fair, a lot of people dothink that this is bad, or phantom liquidity, or even "front-running"! The problem is that the person who buys 100 shares from the market maker for $10.01 on one of the exchanges might alsowant to buy 1,000 more shares from the market maker for $10.01 on all of the other exchanges. And if the market maker is faster than the buyer, the buyer won't be able to do that: The market maker will change its quotes on the other exchanges before the buyer can get there and trade with those quotes. This is much of the plot of my Bloomberg View colleague Michael Lewis's book "Flash Boys,"[SUP][/SUP] and it is an irreconcilable conflict: People who want to buy a lot of shares don't want the price to react to their buying, while market makers do want their prices to reflect supply and demand. A fair amount of modern market-structure complexity comes out of this conflict.[SUP][/SUP]

But we will move right along because our model of the market maker is still very stylized. Our market maker just changes its prices when it trades, to balance supply and demand. But in the real world a market maker can look at other facts about the world to decide what prices it should offer.[SUP][/SUP] If the Fed raises rates, or oil prices spike, or a jobs report is weak, that will affect the prices of thousands of stocks, and the market maker should update its prices for those stocks before anyone trades with the old prices. Meaning: even more cancellations. Big events like that don't happen every second, but a high-speed market-making computer has a lot of time and computing power on its hands, and can spend the bulk of each second thinking about other, smaller events. If someone buys 100 shares of Microsoft, pushing up the price by a penny, then historical correlations might tell the market maker that it should push up its price for Google by a penny.[SUP][/SUP] And so the market maker will go cancel 22 orders on 11 different exchanges and replace them with 22 new orders at slightly higher prices, further increasing its cancellation rate above the 95.5 percent we got two paragraphs ago. Again, not because it is spoofing or front-running or whatever, but because it is a market maker, and its business is about getting the price right, and it is doing its best to get the price right.

Now, like I said, there are lots of different kinds of high-frequency traders, and different kinds cancel orders for different reasons. And sometimes people cancel orders for nefarious reasons! Navinder Sarao is accused of spoofing in the S&P 500 futures market, entering and cancelling lots of orders to create an illusion of demand, in suspicious proximity to the flash crash of 2010. He allegedly "canceled more than 99 percent of the orders that he submitted through his algorithm," which seems bad, although again remember I got to a relatively innocent 95.5 percent cancellation rate three paragraphs ago.[SUP][/SUP] And just today, the SEC settled spoofing charges with Briargate Trading for more than $1 million, alleging that Briargate sent "multiple, large, non-bona fide orders" on the New York Stock Exchange before it opened, creating the illusion of demand, and then traded on the other side of those orders on other exchanges that opened before NYSE. And then canceled the spoof orders before NYSE opened.[SUP][/SUP] Spoofing certainly happens, and cancellation rates are some indication of it, but they are not by themselves a sufficient indication. People cancel orders for lots of legitimate reasons. Some of which we've just discussed.[SUP][/SUP]

Of course, I am thinking about this today because of my Bloomberg View colleague Hillary Clinton's plan to get rid of high-frequency trading. From her financial regulation briefing today:

Impose a high-frequency trading tax and reform the rules that govern our stock markets. The growth of high-frequency trading (HFT) has unnecessarily burdened our markets and enabled unfair and abusive trading strategies that often capitalize on a “two-tiered” market structure with obsolete rules. That’s why Clinton would impose a tax targeted specifically at harmful HFT. In particular, the tax would hit HFT strategies involving excessive levels of order cancellations, which make our markets less stable and less fair.

Now I tend to assume that, like all presidential campaign proposals, this has a low probability of actually happening, and it is in any case pretty light on details. But if you do take it seriously, then the effect would be to cut back on electronic market-making and other high-frequency trading strategies that rely on updating their prices to reflect changing conditions.[SUP][/SUP] (It might cut back on spoofing, too, but spoofing is already illegal, and cancellations are the most obvious clue for where to find it, so I am not convinced that it would do much on that front.) "Mrs. Clinton’s plan to tax high-frequency trades is intended to curtail certain trading activity, not to raise tax revenue," reports the Wall Street Journal.

Do those strategies "make our markets less stable and less fair"? Obviously some people think so! Certainly it is not unreasonable to have some stability concerns about electronic trading, though I suspect that there are better ways to address those concerns than by taxing it out of existence.

As I've said before, though, the story of high-frequency trading is basically one of small smart firms undercutting big banks by being smarter and more automated and more efficient. The old, inefficient, lucrative equity trading business was in large part replaced by automated market-making at electronic trading firms. This took business away from the big banks. It was also unambiguously great for small retail investors, who can now trade stocks instantly for $10 or less, though the story is more complicated for institutional investors. In particular it is easy to find hedge fund managers who complain about high-frequency trading.

"Clinton's proposals amount to a doubling down on her bet that appeasing her party's populist base is worth more than the possibility of alienating wealthy donors," says Bloomberg, but this particular proposal fits oddly with the populist theme. Here you have an industry that has undercut the business of the big banks, irritated hedge fund managers and been great for small investors. Why would cracking down on that business be populist? Why would it alienate wealthy donors?

And yet you can see the populist appeal.[SUP][/SUP] Wall Street, to a lot of people, is Wall Street, and any attack on "Wall Street" sounds good. The political desire is to have a certain quantity of "tough on Wall Street," but what actually goes into that toughness is arbitrary and unimportant. So Clinton also wants to "reinstate the 'swaps push-out' rule for banks’ derivatives trading, which was repealed at the behest of the banking lobby in last year’s budget deal." I have long thought that swaps push-out is the purest piece of symbolic emotional identification in financial regulation, and I still think that, but for precisely that reason it resonates. No one knows what it does, and no one thinks that it matters, so it is useful as a pure abstract marker of what team you're on.

But for those of us who are more interested in finance than in politics, this just seems weird. Wall Street is nota monolith, and being "tough on Wall Street" makes no sense. Regulating the parts of Wall Street that you don't like can help out the parts of Wall Street that you do like. Lots of hedge fund managers will be thrilled by a crackdown on high-frequency trading.[SUP][/SUP] Cracking down on small automated competitors to banks might be good for banks. There are Wall Street winners and Wall Street losers to all sorts of Wall Street regulation, and a pure quantity theory of toughness elides those differences.

• There is no company with the ticker XYZ, it's just a generic stand-in, though wouldn't it have been cool if Alphabet had used XYZ? But, no, it's GOOG. (And GOOGL.)
• Of course you might prefer to sell for $10.00, and the someone else might prefer to buy for $10.00, and if the two of you come to the stock exchange at exactly the right time you should trade with each other rather than pay the market maker the two cents. It mostly comes down to the value of immediacy. (As is always true of market making!) If there were no market makers, "real" buyers and sellers could probably find each other, but would have to wait longer. In less frequently traded stocks, it might take you minutes to sell your stock, instead of milliseconds, and the market might move against you in that time. Do you care? Enough to make the high-frequency trader's intermediation worth it? I don't know.

In theory, "dark pools" are aimed at institutional investors who want to trade with each other and are willing to sacrifice the immediacy provided by high-frequency traders. In practice, many dark pools are full of high-frequency traders too.

• I mentioned this yesterday, but here is a New York Fed blog post discussing, among other things, proxied returns to market-making in equities:

The decline in high-frequency market-making returns has occurred against a backdrop of increasing competition. The expected returns to high-frequency trading (HFT) in the 1990s encouraged large investments in speed and led many new firms to enter the sector—as documented in academic studies. The sharp decline in high-frequency profits over the first ten years of our sample suggests that these profits were gradually eroded by competition as the HFT sector developed. Importantly, market-making profits have not increased since capital and liquidity regulations were tightened following the financial crisis.

So electronic market makers basically competed away the premium that banks, and earlier electronic market makers, made.
• It is more or less in Jack Treynor's "The Economics of the Dealer Function," a classic on market making.
• This is obviously very stylized -- you might not move your quote after every execution; you might get "simultaneous" buy and sell executions; etc. But you get the idea.
• This New York Times adaptation from "Flash Boys" is on this topic, what IEX "called electronic front-running — seeing an investor trying to do something in one place and racing ahead of him to the next." In the case of Brad Katsuyama, the former Royal Bank of Canada trader who co-founded IEX and is the hero of the book, his order would be executed first on BATS, and then market makers would move up their prices on other exchanges before he was able to execute against them:

Why would the market on the screens be real if you sent your order to only one exchange but prove illusory when you sent your order to all the exchanges at once? The team began to send orders into various combinations of exchanges. First the New York Stock Exchange and Nasdaq. Then N.Y.S.E. and Nasdaq and BATS. Then N.Y.S.E., Nasdaq BX, Nasdaq and BATS. And so on. What came back was a further mystery. As they increased the number of exchanges, the percentage of the order that was filled decreased; the more places they tried to buy stock from, the less stock they were actually able to buy. “There was one exception,” Katsuyama says. “No matter how many exchanges we sent an order to, we always got 100 percent of what was offered on BATS.” Park had no explanation, he says. “I just thought, BATS is a great exchange!”

Eventually they figured it out:

The reason they were always able to buy or sell 100 percent of the shares listed on BATS was that BATS was always the first stock market to receive their orders. News of their buying and selling hadn’t had time to spread throughout the marketplace. Inside BATS, high-frequency-trading firms were waiting for news that they could use to trade on the other exchanges. BATS, unsurprisingly, had been created by high-frequency traders.

• For instance, intermarket sweep orders, which we talked about the other day, are a way for investors to try to buy as many shares as they can before prices change. Similarly Brad Katsuyama's Thor router in "Flash Boys," or even IEX itself, are designed to give fundamental investors some ability to get to market makers before they can change their quotes. Market makers, meanwhile, invest in their own complexity, meaning mostly technology to make their reactions faster so that they don't get picked off by the fundamental investors.
• Or not. My understanding is that at least some market-making algorithms are pretty much as dirt-simple as the stylized one I describe. Again, there are a variety of types of high-frequency trading, with some being based more on statistical correlations and others based more on just dumb market-making price response.
• Or even: If the order book for Google changes -- if more buy orders show up on the stock exchange, though all at prices too low to execute currently -- then that might tell the market maker that there is more demand for Google, and it might choose to raise its price for Google. That is, of course, what spoofers are taking advantage of.
• That was on 11 exchanges, though. In one e-mini futures market it seems high.
• Irresistibly, here is a 2010 profile of Briargate as "The Traders Who Skip Most of the Day." Now we know why!
• Here's a terrific Bloomberg visualization of spoofing; relevant:

Of course, honest traders change their minds all the time and cancel orders as economic conditions change. That's not illegal. To demonstrate spoofing, prosecutors or regulators must show the trader entered orders he never intended to execute. That's a high burden of proof in any market.

One helpful fact is if most of a trader's (canceled) orders were on one side (say to buy) when he was mostly actually trading on the other (selling). For instance Sarao allegedly put in huge orders to sell, so that he could buy a few contracts: All his trading was on one side, but most of his orders were on the other. Then he'd switch a little while later. That seems like a bad sign.
• Or of course to just find straightforward ways around the rules:

James Angel, associate professor of finance at Georgetown University, said that too many cancellations are a burden on the stock market because of the increased messaging traffic and extra effort to process market data. At the same time, he said, market-makers need to cancel and re-enter their orders every time the price of securities change. Angel speculated that if Clinton's proposal were to become law, stock exchanges might respond by issuing new order types that let high-speed traders modify their orders without actually canceling them.

That is pretty silly, no? I mean, I'm sure he's right, but it would be silly to impose tax cancellations and not tax price changes.
• For myself, I can't see how it would alienate wealthy donors, though of course some of Clinton's other proposals might.
• Just as other hedge fund managers want a crackdown on too-big-to-fail insurers.

This column does not necessarily reflect the opinion of the editorial board or Bloomberg LP and its owners.

High-frequency trading has been in the news a lot over the past few years. The “Flash Crash” of 2010—when the Dow Jones Industrial Average experienced one of its biggest one-day point declines (of almost 1,000 points) in its history—was followed by the 2014 publication of Michael Lewis’s bestselling nonfiction book, Flash Boys. As a corollary to this story, and equally controversial, dark pools have been sought by investors who are looking to avoid interacting with aggressive liquidity, usually from high frequency trading firms. What’s the big deal?

We overview the uses of machine learning for high frequency trading and market microstructure data and problems. Machine learning is a vibrant subfield of computer science that draws on models and methods from statistics, algorithms, computational complexity, artificial intelligence, control theory, and a variety of other disciplines. Its primary focus is on computationally and informationally efficient algorithms for inferring good predictive models from large data sets, and thus is a natural candidate for application to problems arising in HFT, both for trade execution and the generation of alpha. The inference of predictive models from historical data is obviously not new in quantitative finance; ubiquitous examples include coefficient estimation for the CAPM, Fama and French factors [5], and related approaches. The special challenges for machine learning presented by HFT generally arise from the very fine granularity of the data—often microstructure data at the resolution of individual orders, (partial) executions, hidden liquidity, and cancellations — and a lack of understanding of how such low-level data relates to actionable circumstances (such as profitably buying or selling shares, optimally executing a large order, etc.). In the language of machine learning, whereas models such as CAPM and its variants already prescribe what the relevant variables or “features” are for prediction or modeling (excess returns, book-to-market ratios, etc.), in many HFT problems one may have no prior intuitions about how (say) the distribution of liquidity in the order book relates to future price movements, if at all. Thus feature selection or feature engineering becomes an important process in machine learning for HFT, and is one of our central themes.