There's been some interest in the comments about the tech of High Frequency Trading, so I figured it might be good to do a quick overview of some of the technology used in HFT.
First, a bit of terminology...
A lot of times I hear the terms "algo" or "algorithmic" trading used synonymous to HFT. While just about every HFT shop is also algo, just because a firm describes themselves as algo doesn't necessarily imply they are HFT. Algo trading just means you use algorithms to do one or more of the following tasks 1) decide when and where to execute a trade and/or 2) generate the signal that initiated the trade.
There are plenty of shops that do either 1 or 2 but not both. These are usually mid to longer term strategies.
That might sound wierd, to hear me decompose the terms "trade" and "signal" so let me explain a bit. A trade is simply the act of buying or selling something. The signal is the reason for the trade. The signal is why we are doing the trade.
So anyways, in what follows I'm really talking about algo HFT shops here where both the signals and the trade execution are almost completely done by machines. There are a number of technologies which shops tend to cluster around, and I'll go into those first. Then I'll talk about alternative technologies that some firms use to accomplish the same tasks in a different way. Finally, I'll talk about the benefits of being slightly different living and trading in a world where decisions are off-loaded to computers. The speed and scale at which you can do something stupid in HFT is mind-boggling.
There are a lot of well-trodden paths in HFT. Many, many shops host their trading software on machines running the Linux OS. The vast majority of strategies are written in some descendant of C. Many firms use FPGA technology to speed up their processing of network messages. There are good technical reasons to do this. Linux is free, open source, provides lots of customization, and is extremely extensible. C and its ilk are fast, while FPGA's offload tasks to hardware which used to be performed by software, minimizing reaction time considerably.
Likewise, quant's can follow the road more traveled and use R to analyze and manipulate their high frequency datasets. Those frustrated by R's limitations -- or those who want greater flexibility -- can plump for Python and its PyData stack. We can all use big data tech like Hadoop or Spark for big data processing.
On the other hand, some choose to be wildly different.
Some people prefer to remain on the fringe, using technologies that haven't been adopted by the broader market. Consider the prop trading firm Jane Street, which utilizes functional programming languages like oCaml to run their trading desk.
Quants who prefer to be different can use tools like Scala for data processing or HDF for big data storage (we are partial to HDF, might be partially because of the UIUC connection). Neither technology has experienced wide-spread adoption, despite active development from their backers.
In the context of High Frequency Trading, what are the benefits of walking the road less traveled?
While there are many arguments we could make, the biggest benefit comes from synchronicity, or the simultaneous occurrence of events that appear significantly related but have no discernible causal connection. This definition brings to mind events like the equity flash crash in May of 2010 or the bond flash rally in October 2014. But it's perhaps subtler than that: the technology we use shapes the way our minds think in the same way human language shapes individual thought.
Flash events like this are examples of extreme synchronicity in the markets. This sync can occur from many sources.
For example, suppose everyone individually implements a rule to shut off their strategies when the market has moved X% in Y seconds. This seemingly prudent rule on an individual scale can have big consequences on the global, market scale. Consider that if everyone implemented this kind of rule, then a move bigger than X% will synchronize everyone's strategies BY SHUTTING EVERYONE'S STRATEGIES OFF AT THE SAME TIME. Ever noticed the market just disappear while you're trading it during a big move?
Another example: do you trade the same market, using the same technology stack, with the same datafeed as your competitor? Then your strategies will share a latent synchronicity with your competitors.
A great example would be two HFT competitors using the same microwave or undersea line to connect two exchanges which have a wide geographical separation. There were days when it rained in Ohio and rendered a multi-million dollar microwave line useless. Firms which built their lines on a slightly different path could be unaffected, and thus enjoy outsized gains from increased market share.
Moreover, a quant's choice of technology has a direct impact on their strategy's implementation. This includes mathematical models. For example, a lot of statistical arbitrage traders are fading relative value relationships between different assets. All of these strategies accept a certain amount of risk, but if enough other traders are fading the same spread, the relationship can become a self-reinforcing blow up.
Consider the situation where many traders are short the 10-year and long the 30-year US treasury futures; this often occurs after large moves in a spread. Market makers become increasingly loaded up in one direction by definition: they provide liquidity in the spread at all times. As such they get selected by the market, so to speak, and at some point they have to cut risk to trade another day.
As they exit their positions it pushes the spread further out of equilibrium, further exacerbating the pain of trader's who have yet to liquidate their positions. This can snowball, creating a liquidity hole on the exit side of the trade.
In a world where automation becomes imperative to survival, technological choices have a profound influence on the correlation of overall strategies in the market. When we cluster around certain technologies we can experience a clustering of performance as well.
