Choice of languages

Algo trading systems are normally based on a script in some programming language. You can avoid writing scripts entirely by using a visual ‘strategy builder’, ‘code wizard’ or spreadsheet program for defining your strategy. But this is also some sort of programming, just in a different language that you have to master. And visual builders can only create rather simple ‘indicator soup’ systems that are unlikely to produce consistent trading profit. For serious algo trading sytems, real development, and real research, there’s no stepping around ‘real programming’.

You’re also not free to select the programming language with the nicest or easiest syntax. One of the best compromises of simplicity and object orientation is probably Python. It also offers libraries with useful statistics and indicator functions. Consequently, many strategy developers start with programming their systems in Python… and soon run into serious limitations. There’s another criterion that is more relevant for system development than syntax: execution speed.

Speed mostly depends on whether a computer language is compiled or interpreted. C, Pascal, and Java are compiled languages, meaning that the code runs directly on the processor (C, C++, Pascal) or on a ‘virtual machine’ (Java). Python, R, and Matlab is interpreted: The code won’t run by itself, but is executed by an interpreter software. Interpreted languages are much slower and need more CPU and memory resources than compiled languages. It won’t help much for trading strategies that they have fast C-programmed libraries. All backtests or optimization processes must still run through the bottleneck of interpreted trading logic. Theoretically the slowness can be worked around with ‘vectorized coding’ – see below – but that has little practical use.

R and Python have other advantages. They are interactive: you can enter commands directly at a console. This allows quick code or function testing. Some languages, such as C#, are inbetween: They are compiled to a machine-independent interim code that is then, dependent on implementation, either interpreted or converted to machine code. C# is about 4 times slower than C, but still 30 times faster than Python.

Here’s a benchmark table of the same two test programs written in several languages: a sudoku solver and a loop with a 1000 x 1000 matrix multiplication (in seconds):

Speed becomes important as soon as you want to develop a short-term trading system. In the development process you’re all the time testing system variants. A 10-years backtest with M1 historical data executes the strategy about 3 million times. If a C-written strategy needs 1 minute for this, the same strategy in EasyLanguage would need about 30 minutes, in Python 2 hours, and in R more than 10 hours! And that’s only a backtest, no optimization or WFO run. If I had coded the trend experiment in Python or R, I would today still wait for the results. You can see why trade platforms normally use a C variant or a proprietary compiled language for their strategies. HFT systems are anyway written in C or directly in machine language.

Even compiled languages can have large speed differences due to different implementation of trading and analysis functions. When we compare not Sudoku or a matrix multiplication, but a real trading system – the small RSI strategy from this page – we find very different speeds on different trading platforms (10 years backtest, ticks resolution):

• Zorro: ~ 0.5 seconds (compiled C)
• MT4:  ~ 110 seconds (MQL4, a C variant)
• MultiCharts: ~ 155 seconds (EasyLanguage, a C/Pascal mix)

However, the differences are not as bad as suggested by the benchmark table. In most cases the slow language speed is partically compensated by fast vector function libraries. A script that does not go step by step through historical data, but only calls library functions that process all data simultaneously, would run with comparable speed in all languages. Indeed some trading systems can be coded in this vectorized method, but unfortunately this works only with simple systems and requires entirely different scripts for backtests and live trading.

Choice of tools

Zorro is a software for financial analysis and algo-trading – a sort of Swiss Knife tool since you can use it not only for live trading, but also for all sorts of quick tests. It’s my software of choice for financial hacking because:

• It’s free (unless you’re rich).
• Scripts are in C, event driven and very fast. You can code a system or an idea in 5 minutes.
• Open architecture – you can add anything with DLL plugins.
• Minimalistic – just a frontend to a programming language.
• Can be automatized for experiments.
• Very stable – I rarely found bugs and they were fixed fast.
• Very accurate, realistic trading simulation, including HFT.
• Supports also options and futures, and portfolios of multiple assets.
• Has a library with 100s of indicators, statistics and machine learning functions, most with source code.
• Is continuously developed and supported (new versions usually come out every 2..3 months).
• Last but not least: I know it quite well, as I’ve written its tutorial…

A strategy example coded in C, the classic SMA crossover:

void run()
{
  double* Close = series(priceClose());
  double* MA30 = series(SMA(Close,30));	
  double* MA100 = series(SMA(Close,100));
	
  Stop = 4*ATR(100);
  if(crossOver(MA30,MA100))
    enterLong();
  if(crossUnder(MA30,MA100))
    enterShort();
}

More code can be found among the script examples on the Zorro website. You can see that Zorro offers a relatively easy trading implementation. But here comes the drawback of the C language: You can not as easy drop in external libraries as in Python or R. Using a C/C++ based data analysis or machine learning package involves sometimes a lengthy implementation. Fortunately, Zorro can also call R and Python functions for those purposes.

R is a script interpreter for data analysis and charting. It is not a real language with consistent syntax, but more a conglomerate of operators and data structures that has grown over 20 years. It’s harder to learn than a normal computer language, but offers some unique advantages. I’ll use it in this blog when it comes to complex analysis or machine learning tasks. It’s my tool of choice for financial hacking because:

• It’s free. (“Software is like sex: it’s better when it’s free.”)
• R scripts can be very short and effective (once you got used to the syntax).
• It’s the global standard for data analysis and machine learning.
• Open architecture – you can add modules for almost anything.
• Minimalistic – just a console with a language interpreter.
• Very stable – I found a few bugs in external libraries, but so far never in the main program.
• Has tons of “packages” for all imaginable mathematical and statistical tasks, and especially for machine learning.
• Is continuously developed and supported by the global scientific community (about 15 new packages usually come out every day).

This is the SMA crossover in R for a ‘vectorized’ backtest:

require(quantmod)
require(PerformanceAnalytics)

Data <- xts(read.zoo("EURUSD.csv", tz="UTC", format="%Y-%m-%d %H:%M", sep=",", header=TRUE))
Close <- Cl(Data)
MA30 <- SMA(Close,30)
MA100 <- SMA(Close,100)
 
Dir <- ifelse(MA30 > MA100,1,-1) # calculate trade direction
Dir.1 <- c(NA,Dir[-length(Dir)]) # shift by 1 for avoiding peeking bias
Return <- ROC(Close)*Dir.1 
charts.PerformanceSummary(na.omit(Return))

You can see that the vectorized code just consists of function calls. It runs almost as fast as the C equivalent. But it is difficult to read, it can not be used for live trading, and many parts of a trading logic – even a simple stop loss – cannot be coded for a vectorized test. Thus, so good R is for interactive data analysis, so hopeless is it for writing trade strategies – although some R packages (for instance, quantstrat) even offer rudimentary optimization and test functions. They all require an awkward coding style and do not simulate trading very realistically, but are still too slow for serious backtests.

Although R can not replace a serious backtest and trading platform, Zorro and R complement each other perfectly: Here is an example of a machine learning system build together with a deep learning package from R and the training and trading framework from Zorro.

More hacker’s tools

Aside from languages and platforms, you’ll often need auxiliary tools that may be small, simple, cheap, but all the more important since you’re using them all the time. For editing not only scripts, but even short CSV lists I use Notepad++. For interactive working with R I recommend RStudio. Very helpful for strategy development is a file comparison tool: You often have to compare trade logs of different system variants and check which variant opened which trade a little earlier or later, and which consequences that had. For this I use Beyond Compare.

Aside from Zorro and R, there’s also a relatively new system development software that I plan to examine closer at some time in the future, TSSB for generating and testing bias-free trading systems with advanced machine learning algorithms. David Aronson and Timothy Masters were involved in its development, so it certainly won’t be as useless as most other “trade system generating” software. However, there’s again a limitation: TSSB can not trade or export, so you can not really use the ingenious systems that you developed with it. Maybe I’ll find a solution to combine TSSB with Zorro.

References

TIOBE index of top programming languages

Speed comparison of programming languages

Update (November 2017). The release of new deep learning packages has made TSSB sort of obsolete. For instance, the H2O package natively supports several ways of features filtering and dimensionality reduction, as well as ensembles, both so far the strength of TSSB. H2O is supported with Zorro’s advise function. Still, the TSSB book by Davin Aronson is a valuable source of methods, approaches, and tips about machine learning for financial prediction.

Download links to the latest versions of Zorro and R are placed on the side bar. A brief tutorial to both Zorro an R is contained in the Zorro manual; a more comprehensive introduction into working with Zorro can be found in the Black Book.

Suppose you want to trade by moon phases. But you’re not sure if you shall buy at full moon and sell at new moon, or the other way around. So you do a series of moon phase backtests and find out that the best system, which opens positions at any first quarter moon, achieves 30% annual return. Is this finally the proof that astrology works?

A trade system based on a nonexisting effect has normally a profit expectancy of zero (minus trade costs). But you won’t get zero when you backtest variants of such a system. Due to statistical fluctuations, some of them will produce a positive and some a negative return. When you now pick the best performer, such as the first quarter moon trading system, you might get a high return and an impressive equity curve in the backtest. Sadly, its test result is not necessarily caused by clever trading. It might be just by clever selecting the random best performer from a pool of useless systems.

For finding out if the 30% return by quarter moon trading are for real or just the fool’s gold of Data Mining Bias, Halbert White (1959-2012) invented a test method in 2000.  White’s Reality Check (aka Bootstrap Reality Check) is explained in detail in the book ‘Evidence-Based Technical Analysis’ by David Aronson. It works this way:

1. Develop a strategy. During the development process, keep a record of all strategy variants that were tested and discarded because of their test results, including all abandoned algorithms, ideas, methods, and parameters. It does not matter if they were discarded by human decision or by a computer search or optimizing process.
2. Produce balance curves of all strategy variants, using detrended trade results and no trade costs. Note down the profit P of the best strategy.
3. Detrend all balance curves by subtracting the mean return per bar (not to be confused with detrending the trade results!). This way you get a series of curves with the same characteristics of the tested systems, but zero profit.
4. Randomize all curves by bootstrap with replacement. This produces new curves from the random returns of the old curves. Because the same bars can be selected multiple times, most new curves now produce losses or profits different from zero.
5. Select the best performer from the randomized curves, and note down its profit.
6.  Repeat steps 4 and 5 a couple 1000 times.
7. You now have a list of several 1000 best profits. The median M of that list is the Data Mining Bias by your strategy development process.
8. Check where the original best profit P appears in the list. The percentage of best bootstrap profits greater than P is the so-called p-Value of the best strategy. You want the p-Value to be as low as possible. If P is better than 95% of the best bootstrap profits, the best strategy has a real edge with 95% probability.
9. P minus M minus trade costs is the result to be expected in real trading the best strategy.

The method is not really intuitive, but mathematically sound. However, it suffers from a couple problems that makes WRC difficult to use  in real strategy development:

• You can see the worst problem already in step 1. During strategy development you’re permanently testing ideas, adding or removing parameters, or checking out different assets and time frames. Putting aside all discarded variants and producing balance curves of all combinations of them is a cumbersome process. It gets even more diffcult with machine learning algorithms that optimize weight factors and usually do not produce discarded variants. However, the good news are that you can easily apply the WRC when your strategy variants are produced by a transparent mechanical process with no human decisions involved. That’s fortunately the case for our trend experiment.
• WRC tends to type II errors. That means it can reject strategies although they have an edge. When more irrelevant variants – systems with random trading and zero profit expectancy – are added to the pool,  more positive results can be produced in steps 4 and 5, which reduces the probability that your selected strategy survives the test. WRC can determine rather good that a system is profitable, but can not as well determine that it is worthless.
• It gets worse when variants have a negative expectancy.  WRC can then over-estimate Data Mining Bias (see paper 2 at the end of the article). This could theoretically also happen with our trend systems, as some variants may suffer from a phase reversal due to the delay by the smoothing indicators, and thus in fact trade against the trend instead of with it.

The Experiment

First you need to collect daily return curves from all tested strategies. This requires adding a few lines to the Trend.c script from the previous article:

// some global variables
int Period;
var Daily[3000];
...
// in the run function, set all trading costs to zero
 Spread = Commission = RollLong = RollShort = Slippage = 0;
...
// store daily results in an equity curve
  Daily[Day] = Equity;
}
...
// in the objective function, save the curves in a file for later evaluation
string FileName = "Log\\TrendDaily.bin";
string Name = strf("%s_%s_%s_%i",Script,Asset,Algo,Period);
int Size = Day*sizeof(var); 
file_append(FileName,Name,strlen(Name)+1);
file_append(FileName,&Size,sizeof(int));
file_append(FileName,Daily,Size);

The second part of the above code stores the equity at the end of any day in the Daily array. The third part stores a string with the name of the strategy, the length of the curve,  and the equity values itself in a file named TrendDaily.bin in the Log folder. After running the 10 trend scripts, all 900 resulting curves are collected together in the file.

The next part of our experiment is the Bootstrap.c script that applies White’s Reality Check. I’ll write it in two parts. The first part just reads the 900 curves from the TrendDaily.bin file, stores them for later evaluation, finds the best one and displays a histogram of the profit factors. Once we got that, we did already 80% of the work for the Reality Check. This is the code:

void _plotHistogram(string Name,var Value,var Step,int Color)
{
  var Bucket = floor(Value/Step);
  plotBar(Name,Bucket,Step*Bucket,1,SUM+BARS+LBL2,Color);
}

typedef struct curve
{
  string Name;
  int Length;
  var *Values;
} curve;

curve Curve[900];
var Daily[3000];

void main()
{
  byte *Content = file_content("Log\\TrendDaily.bin");
  int i,j,N = 0;
  int MaxN = 0;
  var MaxPerf = 0.0;
	
  while(N<900 && *Content)
  {
// extract the next curve from the file
    string Name = Content;
    Content += strlen(Name)+1;
    int Size = *((int*)Content);
    int Length = Size/sizeof(var); // number of values
    Content += 4;
    var *Values = Content;
    Content += Size;

// store and plot the curve		
    Curve[N].Name = Name;
    Curve[N].Length = Length;
    Curve[N].Values = Values;
    var Performance = 1.0/ProfitFactor(Values,Length);
    printf("\n%s: %.2f",Name,Performance);
    _plotHistogram("Profit",Performance,0.005,RED);

// find the best curve		
    if(MaxPerf < Performance) {
      MaxN = N; MaxPerf = Performance;
    }
    N++;
  }
  printf("\n\nBenchmark: %s, %.2f",Curve[MaxN].Name,MaxPerf); 
}

Most of the code is just for reading and storing all the equity curves. The indicator ProfitFactor calculates the profit factor of the curve, the sum of all daily wins divided by the sum of all daily losses. However, here we need to consider the array order. Like many platforms, Zorro stores time series in reverse chronological order,  with the most recent data at the begin. However we stored the daily equity curve in straight chronological order. So the losses are actually wins and the wins actually losses, which is why we need to inverse the profit factor. The curve with the best profit factor will be our benchmark for the test.

This is the resulting histogram, the profit factors of all 900 (or rather, 705 due to the trade number minimum) equity curves:

Note that the profit factors are slightly different to the parameter charts of the previous article because they were now calculated from daily returns, not from trade results. We removed trading costs, so the histogram is centered at a profit factor 1.0, aka zero profit. Only a few systems achieved a profit factor in the 1.2 range, the two best made about 1.3. Now we’ll see what White has to say to that. This is the rest of the main function in Bootstrap.c that finally applies his Reality Check:

plotBar("Benchmark",MaxPerf/0.005,MaxPerf,50,BARS+LBL2,BLUE);	
printf("\nBootstrap - please wait");
int Worse = 0, Better = 0;
for(i=0; i<1000; i++) {
  var MaxBootstrapPerf = 0;
  for(j=0; j<N; j++) {
    randomize(BOOTSTRAP|DETREND,Daily,Curve[j].Values,Curve[j].Length);
    var Performance = 1.0/ProfitFactor(Daily,Curve[j].Length);
    MaxBootstrapPerf = max(MaxBootstrapPerf,Performance);
  }
  if(MaxPerf > MaxBootstrapPerf)
    Better++;
  else
    Worse++;
  _plotHistogram("Profit",MaxBootstrapPerf,0.005,RED);
  progress(100*i/SAMPLES,0);
}
printf("\nBenchmark beats %.0f%% of samples!",
  (var)Better*100./(Better+Worse));

This code needs about 3 minutes to run; we’re sampling the 705 curves 1000 times. The randomize function will shuffle the daily returns by bootstrap with replacement; the DETREND flag tells it to subtract the mean return from all returns before. The number of random curves that are better and worse than the benchmark is stored, for printing the percentage at the end. The progress function moves the progress bar while Zorro grinds through the 705,000 curves. And this is the result:

Hmm. We can see that the best system – the black bar – is at the right side of the histogram, indicating that it might be significant. But only with about 80% probability (the script gives a slightly different result every time due to the randomizing). 20% of the random curves achieve better profit factors than the best system from the experiment. The median of the randomized samples is about 1.26. Only the two best systems from the original distribution (first image) have profit factors above 1.26 – all the rest is at or below the bootstrap median.

So we have to conclude that this simple way of trend trading does not really work. Interestingly, one of those 900 tested systems is a system that I use for myself since 2012, although with additional filters and conditions. This system has produced good live returns and a positive result by the end of every year so far. And there’s still the fact that EUR/USD and silver in all variants produced better statistics than S&P500. This hints that some trend effect exists in their price curves, but the profit factors by the simple algorithms are not high enough to pass White’s Reality Check. We need a better approach for trend exploitation. For instance, a filter that detects if trend is there or not. This will be the topic of the next article of this series. We will see that a filter can have a surprising effect on reality checks. Since we now have the Bootstrap script for applying White’s Reality Check, we can quickly do further experiments.

The Bootstrap.c script has been added to the 2015 script collection downloadable on the sidebar.

Conclusion

• None of the 10 tested low-lag indicators, and none of the 3 tested markets shows significant positive expectancy with trend trading.
• There is evidence of a trend effect in currencies and commodities, but it is too weak or too infrequent for being effectively exploited with simple trade signals by a filtered price curve.
• We have now a useful code framework for comparing indicators and assets, and for further experiments with trade strategies.

Papers

1. Original paper by Dr. H. White:  White2000
2. WRC modification by P. Hansen: Hansen2005
3. Stepwise WRC Testing by J. Romano, M. Wolf: RomanoWolf2005
4. Technical Analysis examined with WRC, by P. Hsu, C. Kuan: HsuKuan2006
5. WRC and its Extensions by V. Corradi, N. Swanson: CorradiSwanson2011