One should think that most clients come up with very similar trading systems, so we could meanwhile just click them together from ready code. But it is not so. There’s apparently no limit of trading ideas. Almost any other system uses some new trading method, unusual data source, or exotic indicator. Still, the systems can be classified in several simple categories. Any of them has its specific success and failure rate.

Trading systems categorized

We classify the systems by their market and by their trading rules, both specified by the client. The 4 main markets are Forex/ CFDs, cryptocurrencies, stocks/ETFs/futures, and options. And the 4 main algorithmic trading methods are risk premia, market models, data mining, and indicator soups.  To recap:

Risk premium systems gain higher profits by accepting higher risks. In that category fall many stock portfolio rotation and options trading systems.

Market model systems exploit a particular market inefficiency by detecting anomalies in price curves. Mean reversion, market cycles, market events, or statistical arbitrage are typical model based trade methods. 

Data mining systems predict a price trend by evaluating signals with a machine learning algorithm (aka ‘artificial intelligence’). Those signals are usually derived from the order book or the price curve, but sometimes also from fundamental data or exotic data sources.

Indicator soups are the most often demanded systems. They do not target a particular market inefficiency, except maybe for going with the trend. They generate trade signals from a combination of traditional or new invented ‘technical indicators’. 

Some algo trading systems can fall in more than one category. For instance, a grid trader can be considered a risk premium system (high probability of small wins against low probability of high losses), but also a model based system (exploitation of volatility anomalies). If a system cannot be clearly assigned, it is split among several categories in the table below. We can see that clients favor some of the 16 possible combinations, while others are rare:

For determining the success or failure rate, we used 8-years backtests for unoptimized systems, and a walk forward analysis for optimized systems. A successful system had to return at least 12% CAGR for stocks, futures, or options, or 30% annual profit for Forex, CFDs, or cryptocurrencies. The R2 parameter had to be above 0.7. If clients ordered a Montecarlo analysis, the system had to pass it at 95% confidence. If one of those conditions was not fulfilled, the system was classified as failure. 

The percentages of successful systems:

 The average success rates at the end of the columns and rows are weighted by the number of systems. We can see that the overall success rate was only 66%. In 34% of cases we had to break the bad news to the client that it’s not advised to trade this system live. It produced no, or too little profit in the tests. Sometimes we could see what the problem was, and suggest ways to improve the system. But even total failures were no wasted money. When you know that your favorite manually traded system won’t work in the long run, you’ll save a lot more money than spent for programming and testing. 

And the winner is…

The statistics are spoiled by the forex and crypto systems, half of which were losers. This is at least better than most such systems from trading books or trader forums, of which 90% fail already in a simple out-of-sample test, or at least in a cluster or Montecarlo analysis. We got a surprising result in the ‘Indicator soup’ systems. You would normally expect that they all fail big time, since they are not based on a market model. But in fact almost every third indicator hodgepodge was successful, even in live trading on a test server. Maybe the clients knew more than we did. 

It is also a bit surprising that the most complex systems of all, the data mining systems that usually employ deep learning algorithms, did not fare much better. They have an acceptable success rate, but are easily surpassed by a certain sort of much simpler systems. 

Of all systems we tested so far, the big winners were the long-term trading systems for stocks, ETFs, or options. Of the option traders, the simpler systems had often better performance. It’s relatively hard to specify a losing option system, but some still managed it by using short expiration dates, complex entries, fancy rollovers, or intraday buying and selling. One of the very simple, but successful option traders was included in the Zorro scripts.

We found another trend that is not visible in the table: With a few exceptions like HFT or arbitrage, there was an almost linear correlation between time frames and performances. Systems on one, five, or ten minute bars were rarely profitable. The good systems traded mostly on 1-hour, 4-hour, or 24-hour time frames. Faster is not always better.

This does not mean that we all should now abandon forex and cryptos and trade only long-term options or ETF portfolios. Diversification is a key to success. All markets still have long periods of ineffectivity and plenty opportunities of trading profits. Maybe the statistics above help to look for them. 

Update (2025): This article was posted 5 years ago, and we have meanwhile more than 2000 systems programmed for clients. Due to technical progress especially with machine learning systems the overall success rate is now slightly better at 71%. But the success relations between the various trading methods are still the same.

Related articles

⇒ I Hired a Contract Coder

⇒ Build Better Strategies!

When you start a trading strategy, you’re almost always under water for some time. This is a normal consequence of equity curve volatility. It is the very reason why you need initial capital at all for trading (aside from covering margins and transaction costs). Here you can see the typical bumpy start of a trading system:

You can estimate from the live equity curve that this system was rather profitable (it was a grid trader exploiting the CHF price cap). It started in July 2013 and had earned about 750 pips in January 2014, 7 months later. Max drawdown was ~400 pips from September until November. So the raw return of that system was about 750/400 ~= 180%. Normally an excellent value for a trade system. But you can also see from the curve that you were down 200 pips about six weeks into trading, and thus had lost almost half of your minimum initial capital. And if you had started the system in September, you had even stayed under water for more than 3 months! This is a psychologically difficult situation. Many traders panic, pull out, and this way lose money even with highly profitable systems. Algo trading unaffected by emotions? Not true.

Not so out of sample

The basic problem: you can never fully trust your test results. No matter how out-of-sample you test it, a strategy still suffers from a certain amount of Data-Snooping Bias. The standard method of measuring bias – White’s Reality Check – works well for simple mechanically generated systems, as in the Trend Experiment. But all human decisions about algorithms, asset selection, filters, training targets, stop/takeprofit mechanisms, WFO windows, money management and so on add new bias, since they are normally affected by testing. The out-of-sample data is then not so out-of-sample anymore. While the bias by training or optimization can be measured and even eliminated with walk forward methods, the bias introduced by the mere development process is unknown. The strategy might still be profitable, or not anymore, or not at all. You can only find out by comparing live results permanently with test results.

You could do that with no risk by trading on a demo account. But if the system is really profitable, demo time is sacrificed profit and thus expensive. Often very expensive, as you must demo trade a long time for some result significancy, and many strategies have a limited lifetime anyway. So you normally demo trade a system only a few weeks for making sure that the script is bug-free, then you go live with real money.

Pull-out conditions

The simplest method of comparing live results is based on the maximum drawdown in the test. This is the pull-out inequality:

[pmath size=18]E ~<~ C + G t/y – D[/pmath]

E = Current account equity
C = Initial account capital
G = Test profit
t = Live trading period
y = Test period
D = Test maximum drawdown

This formula means simply that you should pull out when the live trading drawdown exceeds the maximum drawdown from the test. Traders often check their live results this way, but there are many problems involved with this method:

• The maximum backtest drawdown is more or less random.
• Drawdowns grow with the test period, thus longer test periods produce worse maximum drawdowns and later pull-out signals.
• The drawdown time is not considered.
• The method does not work when profits are reinvested by some money management algorithm.
• The method does not consider the unlikeliness that the maximum drawdown happens already at live trading start.

For those reasons, the above pullout inequality is often modified for taking the drawdown length and growth into account. The maximum drawdown is then assumed to grow with the square root of time, leading to this modified formula:

[pmath size=18]E ~<~ C + G t/y – D sqrt{{t+l}/y}[/pmath]

E = Current account equity
C = Initial account capital
G = Test profit
t = Live trading period
y = Test period
D = Maximum drawdown depth
l = Maximum drawdown length

This was in fact the algorithm that I often suggested to clients for supervising their live results. It puts the drawdown in relation to the test period and also considers the drawdown length, as the probability of being inside the worst drawdown right at live trading start is l/y. Still, the method does not work with a profit reinvesting system. And it is dependent on the rather random test drawdown. You could address the latter issue by taking the drawdown from a Montecarlo shuffled equity curve, but this produces new problems since trading results have often serial correlation.

After this lenghty introduction for motivation, here’s the proposed algorithm that overcomes the mentioned issues.

Keeping cold blood

For finding out if we really must immediately stop a strategy, we calculate the deviation of the current live trading situation from the strategy behavior in the test. For this we do not use the maximum drawdown, but the backtest equity or balance curve:

1. Determine a time window of length l (in days) that you want to check. It’s normally the length of the current drawdown; if your system is not in a drawdown, you’re probably in cold blood anyway. Determine the drawdown depth D,  i.e. the net loss during that time.
2. Place a time window of same size l at the start of the test balance curve.
3. Determine the balance difference G from end to start of the window. Increase a counter N when G <= D. 
4. Move the window forward by 1 day.
5. Repeat steps 3 and 4 until the window arrived at the end of the balance curve. Count the steps with a counter M.

Any window movement takes a sample out of the curve. We have N samples that are similar or worse, and M-N samples that are better than the current trading situation. The probability to not encounter such a drawdown in T out of M samples is a simple combinatorial equation:

[pmath size=18]1-P ~=~ {(M-N)!(M-T)! }/ {M!(M-N-T)!}[/pmath]

N = Number of  G <= D occurrences
M = Total samples = y-l+1
l = Window length in days
y = Test time in days
T = Samples taken = t-l+1
t = Live trading time in days

P is the cold blood index – the similarity of the live situation with the backtest. As long as P stays above 0.1 or 0.2, probably all is still fine. But if P is very low or zero, either the backtest was strongly biased or the market has significantly changed. The system can still be profitable, just less profitable as in the test. But when the current loss D is large in comparison to the gains so far, we should stop.

Often we want to calculate P soon after the begin of live trading. The window size l is then identical to our trading time t, hence T == 1. This simplifies the equation to: 

[pmath size=18]P ~=~ N/M[/pmath]

In such a situation I’d give up and pull out of a painful drawdown as soon as P drops below 5%.

The slight disadvantage of this method is that you must perform a backtest with the same capital allocation, and store its balance or equity curve in a file for later evaluation during live trading. However this should only take a few lines of code in a strategy script. 

Here’s a small example script for Zorro that calculates P (in percent) from a stored balance curve when a trading time t and drawdown of length l and depth D is given:

int TradeDays = 40;    // t, Days since live start
int DrawDownDays = 30; // l, Days since you're in drawdown
var DrawDown = 100;    // D, Current drawdown depth in $

string BalanceFile = "Log\\BalanceDaily.dbl"; // stored double array

var logsum(int n)
{
  if(n <= 1) return 0;
  return log(n)+logsum(n-1);
}

void main()
{
  int CurveLength = file_length(BalanceFile)/sizeof(var);
  var *Balances = file_content(BalanceFile);

  int M = CurveLength - DrawDownDays + 1;
  int T = TradeDays - DrawDownDays + 1;
 
  if(T < 1 || M <= T) {
    printf("Not enough samples!");
    return;
  }
 
  var GMin=0., N=0.;
  int i=0;
  for(; i < M-1; i++)
  {
    var G = Balances[i+DrawDownDays] - Balances[i];
    if(G <= -DrawDown) N += 1.;
    if(G < GMin) GMin = G;
  } 

  var P;
  if(TradeDays > DrawDownDays)
    P = 1. - exp(logsum(M-N)+logsum(M-T)-logsum(M)-logsum(M-N-T));
  else
    P = N/M;

  printf("\nTest period: %i days",CurveLength);
  printf("\nWorst test drawdown: %.f",-GMin);
  printf("\nM: %i N: %i T: %i",M,(int)N,T);
  printf("\nCold Blood Index: %.1f%%",100*P);
}

Since my computer is unfortunately not good enough for calculating the factorials of some thousand samples, I’ve summed up the logarithms instead – therefore the strange logsum function in the script.

Conclusion

• Finding out early whether a live trading drawdown is ‘normal’ or not can be essential for your wallet.
• The backtest drawdown is a late and inaccurate criteria.
• The Cold Blood Index calculates the precise probability of such a drawdown based on the backtest balance curve.

I’ve added the script above to the 2015 scripts collection. I also have suggested to the Zorro developers to implement this method for automatically analyzing drawdowns while live trading, and issue warnings when P gets dangerously low. This can also be done separately for components in a portfolio system. This feature will probably appear in a future Zorro version. 

Choice of languages

Algo trading systems are normally based on a script in some programming language. You can avoid writing scripts entirely by using a visual ‘strategy builder’, ‘code wizard’ or spreadsheet program for defining your strategy. But this is also some sort of programming, just in a different language that you have to master. And visual builders can only create rather simple ‘indicator soup’ systems that are unlikely to produce consistent trading profit. For serious algo trading sytems, real development, and real research, there’s no stepping around ‘real programming’.

You’re also not free to select the programming language with the nicest or easiest syntax. One of the best compromises of simplicity and object orientation is probably Python. It also offers libraries with useful statistics and indicator functions. Consequently, many strategy developers start with programming their systems in Python… and soon run into serious limitations. There’s another criterion that is more relevant for system development than syntax: execution speed.

Speed mostly depends on whether a computer language is compiled or interpreted. C, Pascal, and Java are compiled languages, meaning that the code runs directly on the processor (C, C++, Pascal) or on a ‘virtual machine’ (Java). Python, R, and Matlab is interpreted: The code won’t run by itself, but is executed by an interpreter software. Interpreted languages are much slower and need more CPU and memory resources than compiled languages. It won’t help much for trading strategies that they have fast C-programmed libraries. All backtests or optimization processes must still run through the bottleneck of interpreted trading logic. Theoretically the slowness can be worked around with ‘vectorized coding’ – see below – but that has little practical use.

R and Python have other advantages. They are interactive: you can enter commands directly at a console. This allows quick code or function testing. Some languages, such as C#, are inbetween: They are compiled to a machine-independent interim code that is then, dependent on implementation, either interpreted or converted to machine code. C# is about 4 times slower than C, but still 30 times faster than Python.

Here’s a benchmark table of the same two test programs written in several languages: a sudoku solver and a loop with a 1000 x 1000 matrix multiplication (in seconds):

Speed becomes important as soon as you want to develop a short-term trading system. In the development process you’re all the time testing system variants. A 10-years backtest with M1 historical data executes the strategy about 3 million times. If a C-written strategy needs 1 minute for this, the same strategy in EasyLanguage would need about 30 minutes, in Python 2 hours, and in R more than 10 hours! And that’s only a backtest, no optimization or WFO run. If I had coded the trend experiment in Python or R, I would today still wait for the results. You can see why trade platforms normally use a C variant or a proprietary compiled language for their strategies. HFT systems are anyway written in C or directly in machine language.

Even compiled languages can have large speed differences due to different implementation of trading and analysis functions. When we compare not Sudoku or a matrix multiplication, but a real trading system – the small RSI strategy from this page – we find very different speeds on different trading platforms (10 years backtest, ticks resolution):

• Zorro: ~ 0.5 seconds (compiled C)
• MT4:  ~ 110 seconds (MQL4, a C variant)
• MultiCharts: ~ 155 seconds (EasyLanguage, a C/Pascal mix)

However, the differences are not as bad as suggested by the benchmark table. In most cases the slow language speed is partically compensated by fast vector function libraries. A script that does not go step by step through historical data, but only calls library functions that process all data simultaneously, would run with comparable speed in all languages. Indeed some trading systems can be coded in this vectorized method, but unfortunately this works only with simple systems and requires entirely different scripts for backtests and live trading.

Choice of tools

Zorro is a software for financial analysis and algo-trading – a sort of Swiss Knife tool since you can use it not only for live trading, but also for all sorts of quick tests. It’s my software of choice for financial hacking because:

• It’s free (unless you’re rich).
• Scripts are in C, event driven and very fast. You can code a system or an idea in 5 minutes.
• Open architecture – you can add anything with DLL plugins.
• Minimalistic – just a frontend to a programming language.
• Can be automatized for experiments.
• Very stable – I rarely found bugs and they were fixed fast.
• Very accurate, realistic trading simulation, including HFT.
• Supports also options and futures, and portfolios of multiple assets.
• Has a library with 100s of indicators, statistics and machine learning functions, most with source code.
• Is continuously developed and supported (new versions usually come out every 2..3 months).
• Last but not least: I know it quite well, as I’ve written its tutorial…

A strategy example coded in C, the classic SMA crossover:

void run()
{
  double* Close = series(priceClose());
  double* MA30 = series(SMA(Close,30));	
  double* MA100 = series(SMA(Close,100));
	
  Stop = 4*ATR(100);
  if(crossOver(MA30,MA100))
    enterLong();
  if(crossUnder(MA30,MA100))
    enterShort();
}

More code can be found among the script examples on the Zorro website. You can see that Zorro offers a relatively easy trading implementation. But here comes the drawback of the C language: You can not as easy drop in external libraries as in Python or R. Using a C/C++ based data analysis or machine learning package involves sometimes a lengthy implementation. Fortunately, Zorro can also call R and Python functions for those purposes.

R is a script interpreter for data analysis and charting. It is not a real language with consistent syntax, but more a conglomerate of operators and data structures that has grown over 20 years. It’s harder to learn than a normal computer language, but offers some unique advantages. I’ll use it in this blog when it comes to complex analysis or machine learning tasks. It’s my tool of choice for financial hacking because:

• It’s free. (“Software is like sex: it’s better when it’s free.”)
• R scripts can be very short and effective (once you got used to the syntax).
• It’s the global standard for data analysis and machine learning.
• Open architecture – you can add modules for almost anything.
• Minimalistic – just a console with a language interpreter.
• Very stable – I found a few bugs in external libraries, but so far never in the main program.
• Has tons of “packages” for all imaginable mathematical and statistical tasks, and especially for machine learning.
• Is continuously developed and supported by the global scientific community (about 15 new packages usually come out every day).

This is the SMA crossover in R for a ‘vectorized’ backtest:

require(quantmod)
require(PerformanceAnalytics)

Data <- xts(read.zoo("EURUSD.csv", tz="UTC", format="%Y-%m-%d %H:%M", sep=",", header=TRUE))
Close <- Cl(Data)
MA30 <- SMA(Close,30)
MA100 <- SMA(Close,100)
 
Dir <- ifelse(MA30 > MA100,1,-1) # calculate trade direction
Dir.1 <- c(NA,Dir[-length(Dir)]) # shift by 1 for avoiding peeking bias
Return <- ROC(Close)*Dir.1 
charts.PerformanceSummary(na.omit(Return))

You can see that the vectorized code just consists of function calls. It runs almost as fast as the C equivalent. But it is difficult to read, it can not be used for live trading, and many parts of a trading logic – even a simple stop loss – cannot be coded for a vectorized test. Thus, so good R is for interactive data analysis, so hopeless is it for writing trade strategies – although some R packages (for instance, quantstrat) even offer rudimentary optimization and test functions. They all require an awkward coding style and do not simulate trading very realistically, but are still too slow for serious backtests.

Although R can not replace a serious backtest and trading platform, Zorro and R complement each other perfectly: Here is an example of a machine learning system build together with a deep learning package from R and the training and trading framework from Zorro.

More hacker’s tools

Aside from languages and platforms, you’ll often need auxiliary tools that may be small, simple, cheap, but all the more important since you’re using them all the time. For editing not only scripts, but even short CSV lists I use Notepad++. For interactive working with R I recommend RStudio. Very helpful for strategy development is a file comparison tool: You often have to compare trade logs of different system variants and check which variant opened which trade a little earlier or later, and which consequences that had. For this I use Beyond Compare.

Aside from Zorro and R, there’s also a relatively new system development software that I plan to examine closer at some time in the future, TSSB for generating and testing bias-free trading systems with advanced machine learning algorithms. David Aronson and Timothy Masters were involved in its development, so it certainly won’t be as useless as most other “trade system generating” software. However, there’s again a limitation: TSSB can not trade or export, so you can not really use the ingenious systems that you developed with it. Maybe I’ll find a solution to combine TSSB with Zorro.

References

TIOBE index of top programming languages

Speed comparison of programming languages

Update (November 2017). The release of new deep learning packages has made TSSB sort of obsolete. For instance, the H2O package natively supports several ways of features filtering and dimensionality reduction, as well as ensembles, both so far the strength of TSSB. H2O is supported with Zorro’s advise function. Still, the TSSB book by Davin Aronson is a valuable source of methods, approaches, and tips about machine learning for financial prediction.

Download links to the latest versions of Zorro and R are placed on the side bar. A brief tutorial to both Zorro an R is contained in the Zorro manual; a more comprehensive introduction into working with Zorro can be found in the Black Book.