As I earned $ 500K in machine learning and high-frequency trading - Part 1

From the interpreter: One of these days Habré was published topic how IT-specialists to keep and increase their money, which has caused quite a lot of interest. I have recently been interested in the financial theme, and I came across an interesting story of a guy who, using their technological skills for a year could earn half a million dollars. I think his experience may be of interest to many habrazhitelyam (even though he already had experience on the stock exchange), so I decided to translate the text (it is very voluminous, so it will be two parts). I>

In this post, I described in detail how I gained about half a million dollars on a high-frequency trading in the period from 2009 to 2010. As I worked completely independently and do not use my program, I can freely talk about it all. For the most part, I participated in the trades in futures on the DAX index and the Russell 2000.

The key to my success, in my opinion, is not the use of complex financial equations, but rather the application of general algorithmic approach that linked together a set of simple components, and include the principles of machine learning algorithm to optimize and maximize profitability. In reading this article, you do not need any knowledge of the specific terminology - when I ran my program, I rely on intuition. (Unfortunately, the beautiful course Andrew Ng'a (Andrew Ng) for machine learning was not yet available to the public - the way if click on the link, you will fall in my current project: CourseTalk, site reviews the massive open online courses / MOOC).

Firstly, I just want to show that my success was not just a happy coincidence. My program 1000-4000 of transactions conducted daily (as "long" and "short"), but the one time I have always been open only a few contracts. This means that my chances of sudden success quite quickly averaged. It also means that I have never lost more than $ 2,000 a day and never saw losses in the monthly period.



(Amendment: this performance, net of fees) i>

And here is a graph showing the variation of my earnings for the day. Note that this graph is not the last 7 months, because - as soon as the amount of stopped growing - I lost the motivation to enter them.

My experience of trading on the stock exchange h4>
Before you start working with my automated trading program, I worked for two years trader in "normal" mode. It was in 2001 - in the formative years of electronic trading, where small speculators were able to earn good money. The way I was working, it was in some ways similar to a computer, and gambling. It is successful meant to be fast, disciplined and have good intuitive ability to recognize patterns of market behavior. I earned about $ 250k, which allowed me to pay for tuition and even delayed. Profit!

Over the next five years, I have tried to raise a couple of start-ups, in parallel improving their programming skills. By trading, I could hardly go back to 2008. However, gambling seemed to me a good opportunity to improve the financial situation at the time when the money from the sale of the last start-up began to come to an end - a lesson could give me an opportunity to think about the next steps.

API for trading h4>
In 2008, I was an ordinary day trader in the futures market and used software called T4. I really wanted to get some customized combinations of "hot keys" to give commands, so learning that T4 has an open API, I bothered to study C # (the programming language used API), and then went ahead and had created the necessary " Hot keys ».

Try their hand with the API, I have decided to increase their appetites: I wanted to teach a computer to trade for me. API provides both streaming market data and easy way to send commands to the stock exchange - and all I had to do - to create a logic that would connect the two events.

Below is a screenshot of the working window T4. Especially good was the fact that by creating their own program, I was able to observe computer bids from the same interface. Make sure that both these teams come and go in the working window (there are themselves tied to my real money), and it was scary and exciting at the same time.

Design your own algorithm h4>
Initially, my goal was to create a system about which I could be reasonably assured - it makes a profit - before the start of the real gambling. In order to achieve this confidence, I needed to create a framework that simulates trades that could - as accurately as possible - to recreate the behavior of the market.

[This typically use test access to the stock exchange, it is possible, and on domestic sites. Also, many traders use a ready-made framework for developing trading robots (for Russian exchanges have several such making , mostly paid) - approx. Perevi.] i>

Since the real trade requires processing market updates, streaming borne API, simulation mode required to organize a reading market updates from the data file. To collect these data, I found the first version of my program just to connect to the API and write market update with time stamps. In order to test and train my system, it took me a total of 4 weeks to gather current information from the markets.

Even with a basic framework, I still had to figure out how to create a Trader trading system profitable. As it turned out, my algorithm as it splits into two separate components, which I will discuss separately:

Predicting price movements h4>
Perhaps an obvious component of any trading system - the ability to predict how prices will change. And my system is no exception. I have defined the current prices, as the average price between domestic demand and domestic supply, and set a goal to predict how the price will change in the next 10 seconds. So, my algorithm would need to predict the position of the price just from times throughout the day.

Creating and optimizing indicators h4>
I created several indicators by which it was possible to confirm my ability to predict price movements in the short term. Each indicator gave a value that was either positive or negative. The use of such indicators was that the most consistent positive growth, and the negative - a drop in the market.

My system allows me to quickly determine the quality of the predictions for each of the indicators, so I could experiment with a large number of them to find out which ones work most closely. Many of the indicators given by the formulas containing variables - knowing these formulas, I could find the optimal values ​​of the variables, making pairwise comparisons of the results achieved using different values.

Indicators provide maximum benefit, were relatively simple and based on the latest data from the market, where I traded, as well as information from other markets where similar traded securities.

Accurate prediction of price behavior h4>
However, having indicators predict a rise or fall in prices was not enough. I need to know exactly how each value of each indicator predicts a specific price change. I wanted a formula that would pay the value of the indicator in the exact price.

To achieve this, I distributed the predicted price hikes of 50 groups differing from the spread of indicator values. It is allowed to make a unique prediction for each group, based on the values ​​of which I could build a graph in Excel. As you can see, the probability of change in the predicted price increases with the value of the indicator.



On the basis of this schedule, I was able to create a formula that matched the curve. At first, I did this' fit under the curve "by hand, but soon wrote a subroutine that automates this process.

Incidentally, not all curves have similar indicator charts. Furthermore, groups were logarithmically distributed to each group was approximately equal to the number of source values. It should also be noted that the negative values ​​of the indicators (and the corresponding predictions of lower prices) were taken on the module and arranged with positive values ​​(my algorithm works with the raising and lowering rates exactly the same).

The layout of indicators to provide a clear prediction of the behavior of prices h4>
It is important to note here is at what point: all the indicators were not completely independent. I could not just dump a bunch of predictions indicators and choose the final value. It was important to highlight the added predictive value, which each indicator complements the predicted value. It was not too difficult to do, but it means that if I manually to fit the curve of the values ​​of several indicators at the same time, I had to be careful - a change in one indicator could cause a change in the others.

In order to fit the curve given all the indicators at the same time I set up the optimizer so that with each iteration curve prediction given me aside only 30%. This jump in the 30% prediction curves allowed to stabilize for a small number of iterations.

Due to the fact that now every indicator only increases the accuracy of the prediction of the price, I can use all the indicators at once to get a definite prognosis as to where the market will be in the next 10 seconds.

Why is not enough just to predict the price h4>
It may seem that with such a market advantage I could get rich. But it is important to remember that the market consists of supply and demand - and not only because of market prices. Success in the high-frequency trading comes to those who offer the best prices, but it is not easy.

Difficulties in creating a system that allows to conclude lucrative deals are the following: