The old-fashioned investing method is buying some low-risk stocks and then waiting a long time. Any portfolio of stocks has a certain mean return and a certain fluctuation in value; you normally want to minimize the latter and maximize the former. Since the mean return and the fluctuation changes all the time, this task requires rebalancing the portfolio in regular intervals. The optimal capital allocation among the portfolio components produces either maximum mean return for a given allowed risk, or minimum risk – respectively, minimum variance – for a given mean return. This optimal allocation is often very different to investing the same amount in all N components of the portfolio. An easy way to solve this mean / variance optimization problem was published 60 years ago by Harry Markowitz. It won him later the Nobel prize.

The unfashionable Markowitz

Unfortunately, Markowitz got largely out of fashion since then. The problem is the same as with all trading algorithms: You can only calculate the optimal capital allocation in hindsight. Optimized portfolios mysteriously failed in live trading. They were said to often return less than a simple 1/N capital distribution. But this was challenged recently in an interesting paper (1) by Keller, Butler, and Kipnis, of which I quote the first paragraph:

Mean-Variance Optimization (MVO) as introduced by Markowitz (1952) is often presented as an elegant but impractical theory. MVO is an “unstable and error-maximizing” procedure (Michaud 1989), and is “nearly always beaten by simple 1/N portfolios” (DeMiguel, 2007). And to quote Ang (2014): “Mean-variance weights perform horribly… The optimal mean-variance portfolio is a complex function of estimated means, volatilities, and correlations of asset returns. There are many parameters to estimate. Optimized mean-variance portfolios can blow up when there are tiny errors in any of these inputs…”.

The optimized portfolios of the quoted authors indeed blew up. But Markowitz is not to blame. They just did not understand what ‘optimal capital allocation’ means. Suppose you have a portfolio of very similar assets, all with almost identical mean return and variance, only one of them is a tiny bit better. The Markowitz algorithm will then tend to assign all capital to that single asset. That’s just logical, as it is the optimal capital allocation. But it’s not the optimal portfolio. You don’t want to expose all capital to a single stock. If that company goes belly up, your portfolio will too. This is the mentioned ‘stability problem’. However there is a simple and obvious solution: a per-asset weight limit.

Aside from that, the Markowitz vituperators used too long, mean-reverting time periods for sampling the returns and covariances, and they applied the MVO algorithm wrongly to mixed long/short portfolios. When correctly applied to a momentum-governed time period and long-only, well diversified portfolios with a weight limit, MVO produced out of sample results far superior to 1/N. This was proven by testing a number of example portfolios in (1) with a R MVO implementation by fellow blogger Ilya Kipnis.

However, a R implementation is not very practical for live trading. For this we have to implement MVO in a real trade platform. Then we can park our money in an optimized portfolio of stocks and ETFs, let the platform rebalance the capital allocation in regular intervals, lean back, wait, and get rich slowly.

Implementing MVO

The Zorro implementation is based on Markowitz’ 1959 publication (2). In chapter 8, he described the MVO algorithm in a clear and easy to follow way. For simple minded programmers like me, he even included a brief introduction to linear algebra! I only modified his original algorithm by adding the mentioned weight constraint. This constraint stabilizes the algorithm and keeps the portfolio diversified.

In wise anticipation of future computing machines, Markowitz also included an example portfolio for checking if you programmed his algorithm correctly. The proof:

function main()
{
	var Means[3] = { .062,.146,.128 };
	var Covariances[3][3] = {.0146,.0187,.0145,.0187,.0854,.0104,.0145,.0104,.0289};
	var Weights[3];
	
	var BestVariance = markowitz(Covariances,Means,3,0);

	markowitzReturn(Weights,1);
	printf("\nMax:  %.2f %.2f %.2f",Weights[0],Weights[1],Weights[2]);
	markowitzReturn(Weights,BestVariance);
	printf("\nBest: %.2f %.2f %.2f",Weights[0],Weights[1],Weights[2]);
	markowitzReturn(Weights,0);
	printf("\nMin:  %.2f %.2f %.2f",Weights[0],Weights[1],Weights[2]);
}

The means and covariances arrays in the script are from Markowitz’ example portfolio. The markowitz function runs the algorithm and returns the variance value associated with the best Sharpe ratio. The markowitzReturn function then calculates the capital allocation weights with the maximum mean return for a given variance. The weights for maximum, best, and minimum variance are printed. If I did it right, they should be exactly the same as in Markowitz’ publication:

Max:  0.00 1.00 0.00
Best: 0.00 0.22 0.78
Min:  0.99 0.00 0.01

Selecting the assets

For long-term portfolios you can’t use the same high-leverage Forex or CFD instruments that you preferred for your short-term strategies. Instead you normally invest in stocks, ETFs, or similar instruments. They offer several advantages for algo trading:

• No zero-sum game. In the long run, stocks and index ETFs have positive mean returns due to dividends and accumulated value, while Forex pairs and index CFDs have negative mean returns due to swap/rollover fees.
   
• Serious brokers. Stock/ETF brokers are all regulated, what can not be said of all Forex/CFD brokers.
   
• More data for your algorithms, such as volume and market depth information.
   
• Bigger choice of assets from many different market sectors.
    
• More trading methods, such as pairs trading (“stat arb”), trading risk-free assets such as T-bills, or trading volatility.

The obvious disadvantage is low leverage, like 1:2 compared with 1:100 or more for Forex instruments. Low leverage is ok for a long-term system, but not for getting rich quick. More restrictions apply to long-term portfolios. MVO obviously won’t work well with components that have no positive mean return. And it won’t work well either when the returns are strongly correlated. So when selecting assets for your long-term portfolio, you have to look not only for returns, but also for correlation. Here’s the main part of a Zorro script for that:

#define NN  30  // max number of assets

function run()
{
	BarPeriod = 1440;
	NumYears = 7;
	LookBack = 6*252; // 6 years

	string	Names[NN];
	vars	Returns[NN];
	var	Correlations[NN][NN];

	int N = 0;
	while(Names[N] = loop( 
		"TLT","LQD","SPY","GLD","VGLT","AOK"))
	{
		if(is(INITRUN))
			assetHistory(Names[N],FROM_YAHOO);
		asset(Names[N]);
		Returns[N] = series((priceClose(0)-priceClose(1))/priceClose(1));
		if(N++ >= NN) break;
	}
	if(is(EXITRUN)) {
		int i,j;
		for(i=0; i<N; i++)
		for(j=0; j<N; j++)
			Correlations[N*i+j] = 
				Correlation(Returns[i],Returns[j],LookBack);
		plotHeatmap("Correlation",Correlations,N,N);
		for(i=0; i<N; i++)
			printf("\n%i - %s: Mean %.2f%%  Variance %.2f%%",
				i+1,Names[i],
				100*annual(Moment(Returns[i],LookBack,1)),
				252*100*Moment(Returns[i],LookBack,2));
	}
}

The script first sets up some parameters, then goes into a loop over N assets. Here I’ve just entered some popular ETFs; for replacing them, websites such as etfdb.com give an overview and help searching for the optimal ETF combination.

In the initial run, the asset prices are downloaded from Yahoo. They are corrected for splits and dividends. The assetHistory function stores them as historical price data files. Then the assets are selected and their returns are calculated and stored in the Returns data series. This is repeated with all 1-day bars of a 7 years test period (obviously the period depends on since when the selected ETFs are available). In the final run the script prints the annual mean returns and variances of all assets, which are the first and second moments of the return series. The annual function and the 252 multiplication factor convert daily values to annual values. The results for the selected ETFs:

1 - TLT: Mean 10.75% Variance 2.29%
2 - LQD: Mean 6.46% Variance 0.31%
3 - SPY: Mean 13.51% Variance 2.51%
4 - GLD: Mean 3.25% Variance 3.04%
5 - VGLT: Mean 9.83% Variance 1.65%
6 - AOK: Mean 4.70% Variance 0.23%

The ideal ETF has high mean return, low variance, and low correlation to all other assets of the portfolio. The correlation can be seen in the correlation matrix that is computed from all collected returns in the above code, then plotted in a N*N heatmap:

The correlation matrix contains the correlation coefficients of every asset with every other asset. The rows and columns of the heatmap are the 6 assets. The colors go from blue for low correlation between the row and column asset, to red for high correlation. Since any asset correlates perfectly with itself, we always have a red diagonal. But you can see from the other red squares that some of my 6 popular ETFs were no good choice. Finding the perfect ETF combination, with the heatmap as blue as possible, is left as an exercise to the reader.

The efficient frontier

After selecting the assets for our portfolio, we now have to calculate the optimal capital allocation, using the MVO algorithm. However, “optimal” depends on the desired risk, i.e. volatility of the portfolio. For every risk value there’s a optimal allocation that generates the maximum return. So the optimal allocation is not a point, but a curve in the return / variance plane, named the Efficient Frontier. We can calculate and plot it with this script:

function run()
{
	... // similar to Heatmap script
 
	if(is(EXITRUN)) {
		int i,j;
		for(i=0; i<N; i++) {
			Means[i] = Moment(Returns[i],LookBack,1);
			for(j=0; j<N; j++)
				Covariances[N*i+j] =
					Covariance(Returns[i],Returns[j],LookBack);	
		}

		var BestV = markowitz(Covariances,Means,N,0);	
		var MinV = markowitzVariance(0,0);
		var MaxV = markowitzVariance(0,1);

		int Steps = 50;
		for(i=0; i<Steps; i++) {
			var V = MinV + i*(MaxV-MinV)/Steps;
			var R = markowitzReturn(0,V);
			plotBar("Frontier",i,V,100*R,LINE|LBL2,BLACK);
		}
		plotGraph("Max Sharpe",(BestV-MinV)*Steps/(MaxV-MinV),
			100*markowitzReturn(0,BestV),SQUARE,GREEN);
	}
}

I’ve omitted the first part since it’s identical to the heatmap script. Only the covariance matrix is now calculated instead of the correlation matrix. Covariances and mean returns are fed to the markowitz function that again returns the variance with the best Sharpe ratio. The subsequent calls to markowitzVariance also return the highest and the lowest variance of the efficient frontier and establish the borders of the plot. Finally the script plots 50 points of the annual mean return from the lowest to the highest variance:

At the right side we can see that the portfolio reaches a maximum annual return of about 12.9%, which is simply all capital allocated to SPY. On the left side we achieve only 5.4% return, but with less than a tenth of the daily variance. The green dot is the point on the frontier with the best Sharpe ratio (= return divided by square root of variance) at 10% annual return and 0.025 variance. This is the optimal portfolio – at least in hindsight. 

Experiments

How will a mean / variance optimized portfolio fare in an out of sample test, compared with with 1/N? Here’s a script for experiments with different portfolio compositions, lookback periods, weight constraints, and variances:

#define DAYS	252 // 1 year lookback period
#define NN	30  // max number of assets

function run()
{
	... // similar to Heatmap script

	int i,j;
	static var BestVariance = 0;
	if(tdm() == 1 && !is(LOOKBACK)) {
		for(i=0; i<N; i++) {
			Means[i] = Moment(Returns[i],LookBack,1);
			for(j=0; j<N; j++)
				Covariances[N*i+j] = Covariance(Returns[i],Returns[j],LookBack);	
		}
		BestVariance = markowitz(Covariances,Means,N,0.5);
	}
	
	var Weights[NN]; 
	static var Return, ReturnN, ReturnMax, ReturnBest, ReturnMin;
	if(is(LOOKBACK)) {
		Month = 0;
		ReturnN = ReturnMax = ReturnBest = ReturnMin = 0;
	}

	if(BestVariance > 0) {
		for(Return=0,i=0; i<N; i++) Return += (Returns[i])[0]/N; // 1/N 
		ReturnN = (ReturnN+1)*(Return+1)-1;
		
		markowitzReturn(Weights,0);	// min variance
		for(Return=0,i=0; i<N; i++) Return += Weights[i]*(Returns[i])[0];
		ReturnMin = (ReturnMin+1)*(Return+1)-1;
		
		markowitzReturn(Weights,1);	// max return
		for(Return=0,i=0; i<N; i++) Return += Weights[i]*(Returns[i])[0];
		ReturnMax = (ReturnMax+1)*(Return+1)-1;

		markowitzReturn(Weights,BestVariance); // max Sharpe
		for(Return=0,i=0; i<N; i++) Return += Weights[i]*(Returns[i])[0];
		ReturnBest = (ReturnBest+1)*(Return+1)-1;

		plot("1/N",100*ReturnN,AXIS2,BLACK);
		plot("Max Sharpe",100*ReturnBest,AXIS2,GREEN);
		plot("Max Return",100*ReturnMax,AXIS2,RED);
		plot("Min Variance",100*ReturnMin,AXIS2,BLUE);
	}
}

The script goes through 7 years of historical data, and stores the daily returns in the Returns data series. At the first trading day of every month (tdm() == 1) it computes the means and the covariance matrix of the last 252 days, then calculates the efficient frontier. This time we also apply a 0.5 weight constraint to the minimum variance point. Based on this efficient frontier, we compute the daily total return with equal weights (ReturnN), best Sharpe ratio (ReturnBest), minimum variance (ReturnMin) and maximum Return (ReturnMax). The weights remain unchanged until the next rebalancing, this way establishing an out of sample test. The four daily returns are added up to 4 different equity curves :

We can see that MVO improves the portfolio in all three variants, in spite of its bad reputation. The black line is the 1/N portfolio with equal weights for all asset. The blue line is the minimum variance portfolio – we can see that it produces slightly better profits than 1/N, but with much lower volatility. The red line is the maximum return portfolio with the best profit, but high volatility and sharp drawdowns. The green line, the maximum Sharpe portfolio, is somewhere inbetween. Different portfolio compositions can produce a different order of lines, but the blue and green lines have almost always a much better Sharpe ratio than the black line. Since the minimum variance portfolio can be traded with higher leverage due to the smaller drawdowns, it often produces the highest profits.

For checking the monthly rebalancing of the capital allocation weights, we can display the weights in a heatmap:

The horizontal axis is the month of the simulation, the vertical axis the asset number. High weights are red and low weights are blue. The weight distribution above is for the maximum Sharpe portfolio of the 6 ETFs.

The final money parking system

After all those experiments we can now code our long-term system. It shall work in the following way:

• The efficient frontier is calculated from daily returns of the last 252 trading days, i.e. one year. That’s a good time period for MVO according to (1), since most ETFs show 1-year momentum.
   
• The system rebalances the portfolio once per month. Shorter time periods, such as daily or weekly rebalancing, showed no advantage in my tests, but reduced the profit due to higher trading costs. Longer time periods, such as 3 months, let the system deteriorate.
   
•  The point on the efficient frontier can be set up with a slider between minimum variance and maximum Sharpe. This way you can control the risk of the system. 
   
• We use a 50% weight constraint at minimum variance. It’s then not anymore the optimal portfolio, but according to (1) – and my tests have confirmed this – it often improves the out of sample balance due to better diversification.

Here’s the script:

#define LEVERAGE 4	// 1:4 leverage
#define DAYS	252 	// 1 year
#define NN	30	// max number of assets

function run()
{
	BarPeriod = 1440;
	LookBack = DAYS;

	string Names[NN];
	vars	Returns[NN];
	var	Means[NN];
	var	Covariances[NN][NN];
	var	Weights[NN];

	var TotalCapital = slider(1,1000,0,10000,"Capital","Total capital to distribute");
	var VFactor = slider(2,10,0,100,"Risk","Variance factor");
	
	int N = 0;
	while(Names[N] = loop( 
		"TLT","LQD","SPY","GLD","VGLT","AOK"))
	{
		if(is(INITRUN))
			assetHistory(Names[N],FROM_YAHOO);
		asset(Names[N]);
		Returns[N] = series((priceClose(0)-priceClose(1))/priceClose(1));
		if(N++ >= NN) break;
	}

	if(is(EXITRUN)) {
		int i,j;
		for(i=0; i<N; i++) {
			Means[i] = Moment(Returns[i],LookBack,1);
			for(j=0; j<N; j++)
				Covariances[N*i+j] = Covariance(Returns[i],Returns[j],LookBack);	
		}
		var BestVariance = markowitz(Covariances,Means,N,0.5);
		var MinVariance = markowitzVariance(0,0);
		markowitzReturn(Weights,MinVariance+VFactor/100.*(BestVariance-MinVariance));

		for(i=0; i<N; i++) {
			asset(Names[i]);
			MarginCost = priceClose()/LEVERAGE;
			int Position = TotalCapital*Weights[i]/MarginCost;
			printf("\n%s:  %d Contracts at %.0f$",Names[i],Position,priceClose());
		}
	}
}

On Zorro’s panel you can set up the invested capital with a slider (TotalCapital) between 0 and 10,000$. A second slider (VFactor) is for setting up the desired risk from 0 to 100%: At 0 you’re trading with minimum variance, at 100 with maximum Sharpe ratio.

This script advises only, but does not trade: For automated trading it, you would need an API plugin to a ETF broker, such as IB. But the free Zorro version only has plugins for Forex/CFD brokers; the IB plugin is not free. However, since positions are only opened or closed once per month and price data is free from Yahoo, you do not really need an API connection for trading a MVO portfolio. Just fire up the above script once every month, and check what it prints out:

TLT:  0 Contracts at 129$
LQD:  0 Contracts at 120$
SPY:  3 Contracts at 206$
GLD:  16 Contracts at 124$
VGLT:  15 Contracts at 80$
AOK:  0 Contracts at 32$

Apparently, the optimal portfolio for this month consists of 3 contracts SPY, 16 contracts GLD, and 15 VGLT contracts. You can now manually open or close those positions in your broker’s trading platform until your portfolio matches the printed advice. Leverage is 4 by default, but you can change this to your broker’s leverage in the #define at the begin of the script. For a script that trades, simply replace the printf statement with a trade command that opens or closes the difference to the current position of the asset. This, too, is left as an exercise to the reader…

MVO vs. OptimalF

It seems natural to use MVO not only for a portfolio of many assets, but also for a portfolio of many trading systems. I’ve tested this with the Z12 system that comes with Zorro and contains about 100 different system/asset combinations. It turned out that MVO did not produce better results than Ralph Vince’s OptimalF factors that are originally used by the system. OptimalF factors do not consider correlations between components, but they do consider the drawdown depths, while MVO is only based on means and covariances. The ultimate solution for such a portfolio of many trading systems might be a combination of MVO for the capital distribution and OptimalF for weight constraints. I have not tested this yet, but it’s on my to do list.

I’ve added all scripts to the 2016 script repository. You’ll need Zorro 1.44 or above for running them. And after you made your first million with the MVO script, don’t forget to sponsor Zorro generously!

Papers

1. Momentum and Markowitz – A Golden Combination: Keller.Butler.Kipnis.2015
    
2. Harry M. Markowitz, Portfolio Selection, Wiley 1959
    
3. MVO overview at guidedchoice.com

The ideal model-based strategy development
Step 1: The model

Select one of the known market inefficiencies listed in the previous part, or discover a new one. You could eyeball through price curves and look for something suspicious that can be explained by a certain market behavior. Or the other way around,  theoretize about a behavior pattern and check if you can find it reflected in the prices. If you discover something new, feel invited to post it here! But be careful: Models of non-existing inefficiencies (such as Elliott Waves) already outnumber real inefficiencies by a large amount. It is not likely that a real inefficiency remains unknown to this day.

Once you’ve decided for a model, determine which price curve anomaly it would produce, and describe it with a quantitative formula or at least a qualitative criteria. You’ll need that for the next step. As an example we’re using the Cycles Model from the previous part:

[latex display=”true”]y_t ~=~ \hat{y} + \sum_{i}{a_i sin(2 \pi t/C_i+D_i)} + \epsilon[/latex]

(Cycles are not to be underestimated. One of the most successful funds in history – Jim Simons’ Renaissance Medallion fund – is rumored to exploit cycles in price curves by analyzing their lengths (C_i), phases (D_i) and amplitudes (a_i) with a Hidden Markov Model. Don’t worry, we’ll use a somewhat simpler approach in our example.)

Step 2: Research

Find out if the hypothetical anomaly really appears in the price curves of the assets that you want to trade. For this you first need enough historical data of the traded assets – D1, M1, or Tick data, dependent on the time frame of the anomaly. How far back? As far as possible, since you want to find out the lifetime of your anomaly and the market conditions under which it appears. Write a script to detect and display the anomaly in price data. For our Cycles Model, this would be the frequency spectrum:

Check out how the spectrum changes over the months and years. Compare with the spectrum of random data (with Zorro you can use the Detrend function for randomizing price curves). If you find no clear signs of the anomaly, or no significant difference to random data, improve your detection method. And if you then still don’t succeed, go back to step 1.

Step 3: The algorithm

Write an algorithm that generates the trade signals for buying in the direction of the anomaly. A market inefficiency has normally only a very weak effect on the price curve. So your algorithm must be really good in distinguishing it from random noise. At the same time it should be as simple as possible, and rely on as few free parameters as possible.  In our example with the Cycles Model, the script reverses the position at every valley and peak of a sine curve that runs ahead of the dominant cycle:

function run()
{
  vars Price = series(price());
  var Phase = DominantPhase(Price,10);
  vars Signal = series(sin(Phase+PI/4));

  if(valley(Signal))
    reverseLong(1); 
  else if(peak(Signal))
    reverseShort(1);
}

This is the core of the system. Now it’s time for a first backtest. The precise performance does not matter much at this point – just determine whether the algorithm has an edge or not. Can it produce a series of profitable trades at least in certain market periods or situations? If not, improve the algorithm or write a another one that exploits the same anomaly with a different method. But do not yet use any stops, trailing, or other bells and whistles. They would only distort the result, and give you the illusion of profit where none is there. Your algorithm must be able to produce positive returns either with pure reversal, or at least with a timed exit.

In this step you must also decide about the backtest data. You normally need M1 or tick data for a realistic test. Daily data won’t do. The data amount depends on the lifetime (determined in step 2) and the nature of the price anomaly. Naturally, the longer the period, the better the test – but more is not always better. Normally it makes no sense to go further back than 10 years, at least not when your system exploits some real market behavior. Markets change extremely in a decade. Outdated historical price data can produce very misleading results. Most systems that had an edge 15 years ago will fail miserably on today’s markets. But they can deceive you with a seemingly profitable backtest.

Step 4: The filter

No market inefficiency exits all the time. Any market goes through periods of random behavior. It is essential for any system to have a filter mechanism that detects if the inefficiency is present or not. The filter is at least as important as the trade signal, if not more – but it’s often forgotten in trade systems. This is our example script with a filter:

function run()
{
  vars Price = series(price());
  var Phase = DominantPhase(Price,10);
  vars Signal = series(sin(Phase+PI/4));
  vars Dominant = series(BandPass(Price,rDominantPeriod,1));
  var Threshold = 1*PIP;
  ExitTime = 10*rDominantPeriod;
	
  if(Amplitude(Dominant,100) > Threshold) {
    if(valley(Signal))
      reverseLong(1); 
    else if(peak(Signal))
      reverseShort(1);
  }
}

We apply a bandpass filter centered at the dominant cycle period to the price curve and measure its amplitude. If the amplitude is above a threshold, we conclude that the inefficiency is there, and we trade. The trade duration is now also restricted to a maximum of 10 cycles since we found in step 2 that dominant cycles appear and disappear in relatively short time.

What can go wrong in this step is falling to the temptation to add a filter just because it improves the test result. Any filter must have a rational reason in the market behavior or in the used signal algorithm. If your algorithm only works by adding irrational filters: back to step 3.

Step 5: Optimizing (but not too much!)

All parameters of a system affect the result, but only a few directly determine entry and exit points of trades dependent on the price curve. These ‘adaptable’ parameters should be identified and optimized. In the above example, trade entry is determined by the phase of the forerunning sine curve and by the filter threshold, and trade exit is determined by the exit time. Other parameters – such as the filter constants of the DominantPhase and the BandPass functions – need not be adapted since their values do not depend on the market situation.

Adaption is an optimizing procdure, and a big opportunity to fail without even noticing it. Often, genetic or brute force methods are applied for finding the “best” parameter combination at a profit peak in the parameter space. Many platforms even have “optimizers” for this purpose. Although this method indeed produces the best backtest result, it won’t help at all for the live performance of the system. In fact, a recent study (Wiecki et.al. 2016) showed that the better you optimize your parameters, the worse your system will fare in live trading! The reason of this paradoxical effect is that optimizing to maximum profit fits your system mostly to the noise in the historical price curve, since noise affects result peaks much more than market inefficiencies.

Rather than generating top backtest results, correct optimizing has other purposes:

• It can determine the susceptibility of your system to its parameters. If the system is great with a certain parameter combination, but loses its edge when their values change a tiny bit: back to step 3.
• It can identify the parameter’s sweet spots. The sweet spot is the area of highest parameter robustness, i.e. where small parameter changes have little effect on the return. They are not the peaks, but the centers of broad hills in the parameter space.
• It can adapt the system to different assets, and enable it to trade a portfolio of assets with slightly different parameters. It can also extend the lifetime of the system by adapting it to the current market situation in regular time intervals, parallel to live trading.

This is our example script with entry parameter optimization:

function run()
{
  vars Price = series(price());
  var Phase = DominantPhase(Price,10);
  vars Signal = series(sin(Phase+optimize(1,0.7,2)*PI/4));
  vars Dominant = series(BandPass(Price,rDominantPeriod,1));
  ExitTime = 10*rDominantPeriod;
  var Threshold = optimize(1,0.7,2)*PIP;
	
  if(Amplitude(Dominant,100) > Threshold) {
    if(valley(Signal))
      reverseLong(1); 
    else if(peak(Signal))
      reverseShort(1);
  }
}

The two optimize calls use a start value (1.0 in both cases) and a range (0.7..2.0) for determining the sweet spots of the two essential parameters of the system. You can identify the spots in the profit factor curves (red bars) of the two parameters that are generated by the optimization process:

In this case the optimizer would select a parameter value of about 1.3 for the sine phase and about 1.0 (not the peak at 0.9) for the amplitude threshold for the current asset (EUR/USD). The exit time is not optimized in this step, as we’ll do that later together with the other exit parameters when risk management is implemented.

Step 6: Out-of-sample analysis

Of course the parameter optimization improved the backtest performance of the strategy, since the system was now better adapted to the price curve. So the test result so far is worthless. For getting an idea of the real performance, we first need to split the data into in-sample and out-of-sample periods. The in-sample periods are used for training, the out-of-sample periods for testing. The best method for this is Walk Forward Analysis. It uses a rolling window into the historical data for separating test and training periods.

Unfortunately, WFA adds two more parameters to the system: the training time and the test time of a WFA cycle. The test time should be long enough for trades to properly open and close, and small enough for the parameters to stay valid. The training time is more critical. Too short training will not get enough price data for effective optimization, training too long will also produce bad results since the market can already undergo changes during the training period. So the training time itself is a parameter that had to be optimized.

A five cycles walk forward analysis (add “NumWFOCycles = 5;” to the above script) reduces the backtest performance from 100% annual return to a more realistic 60%. For preventing that WFA still produces too optimistic results just by a lucky selection of test and training periods, it makes also sense to perform WFA several times with slightly different starting points of the simulation. If the system has an edge, the results should be not too different. If they vary wildly: back to step 3.

Step 7: Reality Check

Even though the test is now out-of-sample, the mere development process – selecting algorithms, assets, test periods and other ingredients by their performance – has added a lot of selection bias to the results. Are they caused by a real edge of the system, or just by biased development? Determining this with some certainty is the hardest part of strategy development.

The best way to find out is White’s Reality Check. But it’s also the least practical because it requires strong discipline in parameter and algorithm selection. Other methods are not as good, but easier to apply:

• Montecarlo. Randomize the price curve by shuffling without replacement, then train and test again. Repeat this many times. Plot a distribution of the results (an example of this method can be found in chapter 6 of the Börsenhackerbuch). Randomizing removes all price anomalies, so you hope for significantly worse performance. But if the result from the real price curve lies not far east of the random distribution peak, it is probably also caused by randomness. That would mean: back to step 3.
• Variants. It’s the opposite of the Montecarlo method: Apply the trained system on variants of the price curve and hope for positive results. Variants that maintain most anomalies are oversampling, detrending, or inverting the price curve. If the system stays profitable with those variants, but not with randomized prices, you might really have found a solid system.
• Really-out-of-sample (ROOS) Test. While developing the system, ignore the last year (2015) completely. Even delete all 2015 price history from your PC. Only when the system is completely finished, download the data and run a 2015 test. Since the 2015 data can be only used once this way and is then tainted, you can not modify the system anymore if it fails in 2015. Just abandon it. Assemble all your metal strength and go back to step 1.

Step 8: Risk management

Your system has so far survived all tests. Now you can concentrate on reducing its risk and improving its performance. Do not touch anymore the entry algorithm and its parameters. You’re now optimizing the exit. Instead of the simple timed and reversal exits that we’ve used during the development phase, we can now apply various trailing stop mechanisms. For instance:

• Instead of exiting after a certain time, raise the stop loss by a certain amount per hour. This has the same effect, but will close unprofitable trades sooner and profitable trades later.
• When a trade has won a certain amount, place the stop loss at a distance above the break even point. Even when locking a profit percentage does not improve the total performance, it’s good for your health. Seeing profitable trades wander back into the losing zone can cause serious ulcers.

This is our example script with the initial timed exit replaced by a stop loss limit that rises at every bar:

function run()
{
  vars Price = series(price());
  var Phase = DominantPhase(Price,10);
  vars Signal = series(sin(Phase+optimize(1,0.7,2)*PI/4));
  vars Dominant = series(BandPass(Price,rDominantPeriod,1));
  var Threshold = optimize(1,0.7,2)*PIP;

  Stop = ATR(100);
  for(open_trades)
    TradeStopLimit -= TradeStopDiff/(10*rDominantPeriod);
	
  if(Amplitude(Dominant,100) > Threshold) {
    if(valley(Signal))
      reverseLong(1); 
    else if(peak(Signal))
      reverseShort(1);
  }
}

The for(open_trades) loop increases the stop level of all open trades by a fraction of the initial stop loss distance at the end of every bar.

Of course you now have to optimize and run a walk forward analysis again with the exit parameters. If the performance didn’t improve, think about better exit methods.

Step 9: Money management

Money management serves three purposes. First, reinvesting your profits. Second, distributing your capital among portfolio components. And third, quickly finding out if a trading book is useless. Open the “Money Management” chapter and read the author’s investment advice. If it’s “invest 1% of your capital per trade”, you know why he’s writing trading books. He probably has not yet earned money with real trading.

Suppose your trade volume at a given time t is V(t). If your system is profitable, on average your capital C will rise proportionally to V with a growth factor c:

[latex display=”true”]\frac{dC}{dt} = c V(t)~~\rightarrow~~ C(t) = C_0 + c \int_{0}^{t}{V(t) dt}[/latex]

When you follow trading book advices and always invest a fixed percentage p of your capital, so that V(t) = p C(t), your capital will grow exponentially with exponent p c:

[latex display=”true”]\frac{dC}{dt} ~=~ c p C(t) ~~\rightarrow~~ C(t) ~=~ C_0 e^{p c t}[/latex]

Unfortunately your capital will also undergo random fluctuations, named Drawdowns. Drawdowns are proportial to the trade volume V(t). On leveraged accounts with no limit to drawdowns, it can be shown from statistical considerations that the maximum drawdown depth D_max grows proportional to the square root of time t:

[latex display=”true”]{D_{max}}(t) ~=~ q V(t) \sqrt{t}[/latex]

So, with the fixed percentage investment:

[latex display=”true”]{D_{max}}(t) ~=~ q p C(t) \sqrt{t}[/latex]

and at the time T = 1/(q p)²:

[latex display=”true”]{D_{max}}(T) ~=~ q p C(T) \frac{1}{q p} ~=~ C(T)[/latex]

You can see that around the time T = 1/(q p)² a drawdown will eat up all your capital C(T), no matter how profitable your strategy is and how you’ve choosen p! That’s why the 1% rule is a bad advice. And why I advise clients not to raise the trade volume proportionally to their accumulated profit, but to its square root – at least on leveraged accounts. Then, as long as the strategy does not deteriorate, they keep a safe distance from a margin call.

Dependent on whether you trade a single asset and algorithm or a portfolio of both, you can calculate the optimal investment with several methods. There’s the OptimalF formula by Ralph Vince, the Kelly formula by Ed Thorp, or mean/variance optimization by Harry Markowitz. Usually you won’t hard code reinvesting in your strategy, but calculate the investment volume externally, since you might want to withdraw or deposit money from time to time. This requires the overall volume to be set up manually, not by an automated process. A formula for proper reinvesting and withdrawing can be found in the Black Book.

Step 10: Preparation for live trading

You can now define the user interface of your trading system. Determine which parameters you want to change in real time, and which ones only at start of the system. Provide a method to control the trade volume, and a ‘Panic Button’ for locking profit or cashing out in case of bad news. Display all trading relevant parameters in real time. Add buttons for re-training the system, and provide a method for comparing live results with backtest results, such as the Cold Blood Index. Make sure that you can supervise the system from whereever you are, for instance through an online status page. Don’t be tempted to look onto it every five minutes. But you can make a mighty impression when you pull out your mobile phone on the summit of Mt. Ararat and explain to your fellow climbers: “Just checking my trades.”

The real strategy development

So far the theory. All fine and dandy, but how do you really develop a trading system? Everyone knows that there’s a huge gap between theory and practice. This is the real development process as testified by many seasoned algo traders:

Step 1. Visit trader forums and find the thread about the new indicator with the fabulous returns.

Step 2. Get the indicator working with a test system after a long coding session. Ugh, the backtest result does not look this good. You must have made some coding mistake. Debug. Debug some more.

Step 3. Still no good result, but you have more tricks up your sleeve. Add a trailing stop. The results now look already better. Run a week analysis. Tuesday is a particular bad day for this strategy? Add a filter that prevents trading on Tuesday. Add more filters that prevent trades between 10 and 12 am, and when the price is below $14.50, and at full moon except on Fridays. Wait a long time for the simulation to finish. Wow, finally the backtest is in the green!

Step 4. Of course you’re not fooled by in-sample results. After optimizing all 23 parameters, run a walk forward analysis. Wait a long time for the simulation to finish. Ugh, the result does not look this good. Try different WFA cycles. Try different bar periods. Wait a long time for the simulation to finish. Finally, with a 19-minutes bar period and 31 cycles, you get a sensational backtest result! And this completely out of sample!

Step 5. Trade the system live.

Step 6. Ugh, the result does not look this good.

Step 7. Wait a long time for your bank account to recover. Inbetween, write a trading book.

I’ve added the example script to the 2016 script repository. In the next part of this series we’ll look into the data mining approach with machine learning systems. We will examine price pattern detection, regression, neural networks, deep learning, decision trees, and support vector machines.

⇒ Build Better Strategies – Part 4