5.1 The meaning and trap of backtesting

Summary

Backtesting is the most different place between quantitative trading and traditional trading. According to the real market data that has happened in history, the simulation strategy signal triggers and matches the transaction quickly, and the performance report and other data are obtained for a period of time. For domestic and foreign stocks, commodity futures, foreign exchange and other markets, the backtesting is the most important components of strategy development.

The meaning of backtesting

In the previous chapters, we learned the basics of the mainstream programming language, and taught you how to use these programming foundations to write some simple trading strategies. It can be said that the road has gone by more than half. However, once a strategy is written, which still has a long distance to the real market environment. It also requires constant backtesting —debugging— backtesting —debugging—and so on, until the strategy can fully implement the model content and running smooth.

From the perspective of quantitative trading logic, A trading strategy is actually based on a series of cognitions and assumptions of the market. Backtesting can determine whether these assumptions are stable or not. In historical times of instability, it could bring what kind of loss? as well as to prevent decision-making of these losses.

In addition, from the perspective of quantitative trading operations, back testing can help detect bugs in the strategy logic, such as future functions, Slipping price, multi-degree fitting, and so on. Provide reliable evidence that the strategy can be used for real market trading.

• Verify the accuracy of the trading signal.

• Verify that the trading logic and your ideas are feasible .

• Discover defects in the trading system and improve the original strategy.

So, the meaning of backtesting is preforming an as true as possible trading process, verifying the validity of historical data, avoiding costly mistakes for the wrong strategy, help us filter, improve and optimize trading strategies.

Backtesting Trap

• The trap of “Trading signal unstable”

The trading strategy is based on static historical data during the backtesting, but the data of the real trading is dynamic. For example: If the highest price is greater than yesterday’s closing price, open long position. This open position conditions in real market, if K line has not been completed, then the highest price is dynamic, trading signals is likely to flicker back and forth. During the backtesting, the backtesting engine is based on static historical data that can be simulated and synthesized.

• The trap of the “Future function”

The future function uses the future price, which means that the current conditions may be modified in the future, and the same future function can also cause the trading signal unstable, such as the “zigzag indicator”

As shown below: The zigzag indicator function indicates the turning point of the peaks and troughs. It can adjust its own value according to the latest real-time price, but if the current price changes, the result of the zigzag turning function will also change. If you use this indicator with such future function, the current order signal may be set up and placed, but this signal may not be quantified later.

• The trap of the “Stealing price”

The so-called “Stealing price” behavior refers to the use of past prices to trade. For example: If the highest price is greater than a certain fixed price to open position by using the opening price. This opening position condition is “stealing price”, because in the real market, when the highest price is greater than a certain price, the price may have been higher than the opening price of a certain distance, and then use the opening price to open position is impossible. However, in the backtesting environment, you can hypothetically open position at the opening price.

There is also another case where if the price gaps jumps over the fixed price set by the strategy, in the backtesting environment, the trading signal will occurs and can be executed; but in the real market, it obviously can’t be executed.

• The trap of the “Impossible trading price”

“Impossible trading price” can be divided into several cases:

first type: In the real market, some exchanges has the daily limit of maximum price raise and falling restriction. However, in the backtesting environment, it is possible to execute the orders.

Second: the exchange orders matching mechanism, which are price priority and time priority. Some varieties of the market order depth will often have a huge number of orders waiting for execution. In the real market, you have to wait orders that before your placing to execute, sometimes your order will never be executed. But in the backtesting environment, the pending order will be execute at any price without waiting.

Third: For arbitrage strategies, the backtesting profit usually are very high, because each time the backtesting operation has been assumed to have grabbed all those spread prices. Under real circumstances, most prices are impossible to execute, or sometimes only one direction or one trading target of trading action is executed, in general, it is almost certainly the conducive direction or trading target will first be executed, then you have to immediately try to execute the opposite direction or variety of the arbitrage strategies orders. even with such short period of delay, the spreads of this trading opportunity may has cost you more than 1 or 2 price points, while the whole spread arbitrage strategy only got very few price points of profit. This kind of situation is very hard to backtest in the simulation environment. Real profits are far away from backtesting results.

Fourth: The “Black swan” event. As shown the red circle in the following figure, the Swiss franc in foreign exchange black swan event, although on the surface there are opening price, highest price, lowest price and closing price. in fact, such extreme market situation come with a k line that its most part are vacuum, lots of stop-loss orders causes the stampede event, the liquidity are almost zero, for orders to execute is very difficult, but in the backtesting environment, it is possible to stop loss.

• The trap of the “Over-fitting”

Every time I saw the following picture, both my heart and mind are amused. this picture shows us a truthful meaning, a ridiculous model, just complex enough, is perfectly adapted to any data.

For quantitative trading, backtesting is based on historical data, but the sample of historical data is limited. If there are too many parameters of the trading strategy, or the trading logic is too complicated, the trading strategy will adapt too much to the historical data.

The modeling process of the quantitative strategy is essentially a process of finding local non-random data from a large number of seemingly random data. If you do not rely on statistical knowledge, it is easy to fall into the trap of overfitting.

So don’t deceive yourself. If you find a bad performance data outside the sample, don’t feel pity or unwilling to admit that the model does not work, and continue optimizing it until the sample data outside also performed as good as inside. it eventually will injure your real money.

• The trap of the “survivor bias”

Wall Street has a popular joke: Suppose there are 1,000 monkeys in the market who participated in the investment. In the first year, 500 monkeys were lost to the market. Halfway through the second year, there are 250 monkeys left . By the end of the third year, there are 125 monkeys left .

…

In the ninth year, the last monkey was left. Then you look at it, the more you look at it, the more you feel its face familiar. Finally, you saw the cover of a financial magazine and shouted "Hey, is that Warren Buffett?”

although this is just a joke, you can still mapping it to the real world situation. For example, if there are 1000 fund managers, after 10 years, about 10 fund managers will outperform the market for 10 consecutive years. This may caused by the random luck and decisions, these fund managers’ skill has nothing to to with it.

Like the backtest report below, most investor will choose the figure on the left. which has a very solid performance without significant retracement.

but please hold on, as shown on the right, which is the real situation. The curve on the left is only the best one among these many backtests. That is to say, there are many much worse performance than the left situation.

• The trap of the “impact cost”

In a real trading environment, prices are always fluctuating. When you are optimistic about a trading opportunity, the price may have changed at the moment of placing an order. So the slippage problem, whether in subjective trading or in quantitative trading, is inevitable.

But backtesting is based on static data, it is difficult to simulate the real trading environment. For example: the order price is 1050 to buy, but the actual trading price may be 1051. There are many situations for this phenomenon, such as: liquidity vacuum in extreme price movement, network delay, hardware and software systems delay, server response delay, etc.

Backtest without slippage

As shown above, is a backtest report without slippage, profit curve looks good, but there are differences between the backtest and the actual trading in real market. Therefore, in order to reduce this error during backtest, we added two slippage point, to increase or decrease the buying and selling price.

Backtest with slippage

As shown above, the same strategy, if added with 2 point of slippage, backtesting results with and without slippage are very different, which means that this strategy needs to be improved or abandoned. In particular, the strategy of relatively high trading frequency, adding 1~2 points slippage during backtesting can make the backtesting closer to the real trading environment.

To sum up

Some people may ask, since there may be so many problems in quantitative trading, how can I prove that my strategy is ok? The answer is simple, we must first simulation trading by this strategy some time before using the real money, if the trading price and trading simulation situation are almost the same with the strategy logic, then it at least prove the strategy logic has no problem.

In any case, for an experienced trading system developer, backtesting is a must thins. Because it can tell you whether a strategy idea can be validated in the historical data. But many times backtesting does not mean that the future will be profitable. Because there are too many pits in the backtesting, after you pay “the price”, you will understand it. And these lessons are all piled up in real money. This article is mainly for reduce the detours and traps.

After-school exercises

1. What is overfitting and how to avoid it?

2. What are the “Survivor bias” examples in real life?