• Trend exists, but is difficult to detect and to exploit. The experiment will find only a few profitable and a lot more unprofitable systems (with realistic trade costs).
• More sophisticated indicators, such as the Decycler or the Lowpass filter, will produce better results than the old SMA, EMA, or HMA. Modern indicators have less lag and thus react faster on the begin and end of a trend.
• White’s Reality Check will reveal a data mining bias rather close to the returns of the best systems from the experiment. If algorithmic trend trading were easy, anyone would be doing it.
• EUR/USD will return better results because according to trader’s wisdom, currencies are “more trending”.

However the results were quite a surprise.

The Trend Test System

This is the script Trend.c used for the trend experiment:

var objective() 
{ 
  return WinTotal/max(1,LossTotal); // Profit factor
}

var filter(var* Data,int Period);

void run()
{
  set(PARAMETERS|LOGFILE);
  PlotHeight1 = 200;
  PlotScale = 8;
  ColorBars[1] = BLACK; // winning trades
  ColorBars[2] = GREY; // losing trades

  StartDate = 2010;
  BarPeriod = 15;
  LookBack = 80*4*24; // 80 trading days ~ 4 months
	
  while(asset(loop("EUR/USD","SPX500","XAG/USD")))
  while(algo(loop("M15","H1","H4")))
  {
    if(Algo == "M15")
      TimeFrame = 1; // 15 minutes
    else if(Algo == "H1")
      TimeFrame = 4; // 1 hour
    else if(Algo == "H4")
      TimeFrame = 16;	// 4 hours

    int Periods[10] = { 10,20,50,100,200,500,1000,2000,5000,10000 };
    int Period = Periods[optimize(0,0,9,1)];
		
    vars Prices = series(price());
    vars Smoothed = series(filter(Prices,Period));

    if(valley(Smoothed))
      enterLong();
    else if(peak(Smoothed))
      enterShort();
  } 
}

This script is supposed to run in Zorro’s “Train” mode. We’re not really training the systems, but Train mode produces parameter histograms, and that’s what we want. The script is basically the same as described in the previous article, but two asset and algo loops are now used for cycling through three different assets (EUR/USD, S&P 500, and Silver) and three time frames (15 minutes, 1 hour, 4 hours). The optimize function selects the time period of the smoothing indicator, in 10 steps from 10 to 10,000 time frames. The meaning of the Zorro-specific functions like asset, algo, loop, optimize can be found in the Zorro manual.

6 years of price history from 2010 to 2015 are used for the simulation. The default trade costs – commission, bid/ask spread, and rollover – are taken from the average 2015 parameters of a FXCM microlot account. FXCM is not the cheapest broker, so the expectation value of a random-trading simulation is not zero, but a loss, dependent on the number of trades. For comparing results we’re using the profit factor, calculated in the objective function. The script calls a filter function that is defined as a prototype, and contained in a main script that includes Trend.c. So any of the indicators is represented by a very short main script like this (TrendLowpass.c):

#include "Strategy\Trend.c"

var filter(var* Data,int Period)
{
  return LowPass(Data,Period);
}

All the scripts can be downloaded from the scripts2015 archive at the sidebar. There’s also a batch file included that starts them all and lets them run in parallel.

The Results

When you select one of the scripts and click Zorro’s  [Train] button, it will grind its wheels about 10 minutes and then spit out a sheet of histograms like this (for the ZMA):

We got 9 histograms from the combinations of EUR/USD, S&P 500, and Silver (horizontally) with bar sizes of 15 minutes, 1 hour, and 4 hours. The red bars are the objective return – in this case, the profit factor,  total win divided by total loss. If they are above 1.0 (left scale), the system was profitable. The black and grey bars are the number of winning and losing trades (right scale). The steps on the X axis are the indicator time period; they normally don’t go all the way to 10 because systems are discarded when the time period exceeded 4 months or the system entered less than 30 trades. That’s why the experiment didn’t produce 900 systems, but only 705.

We can see that the profit generally increases, and the number of trades decreases with the time frame. As typical for trend trading systems we got far more losing trades (grey) than winning trades (black). 13 of the 82 systems in the histograms above have been profitable, 69 were losers. Most of the profitable systems traded EUR/USD. Only one S&P 500 system, and only three silver trading systems produced any profit. Here’s a statistic of all winners and losers from the 705 tested systems:

Since the results are for 2010-2015, you might get different figures when testing different time periods. The Rate column shows the percentage of successful systems. We got a mix of expected and unexpected results. Almost 40% of the EUR/USD systems are winners,  which seems to confirm that currencies tend to trend. The S&P 500 index however is not trending at all –  and this despite the fact that it shows the most significant trend in the long run, as visible in the title image. But here we’re looking for shorter term trends that can be exploited by day trading. Silver, with 20% winners, has at least some trend. Longer time frames produce dramatically better results, mostly due to less trade costs, but also partly because trend seems to prefer long cycles and low frequencies. (You can see that the difference comes not from trade costs alone when you test the systems with spread, commission, slippage, and rollover set to zero). 1-day bar periods would be even more profitable. But I haven’t included them in the test because they produce not enough trades for statistical significance.

The indicators show a surprising behavior. The old traditional indicators put up a good fight. Ehler’s smoothing, decycling, and zero-lag indicators, produce less profitable systems than their classic counterparts. However we’ll see later that the modern indicators are in fact more profitable, although within a smaller parameter range. But they all occupy the top of the profit list. The best systems, with modern indicators and long time frames, have profit factors up to 2.

Does this mean that we can put them together in a compound system and will get rich? I don’t know, because I haven’t determined yet the Data Mining Bias produced by this experiment. For this we need to store and randomize the balance curves, and apply White’s Reality Check. This will be the topic of the next article of the Trend Experiment series.

Conclusion

• Different markets show very different behavior. EUR/USD and silver are significantly more profitable with short-term trend following than S&P500.
• Low-lag smoothing indicators have a smaller profitable parameter range than traditional indicators, but produce higher profits.

• Information spreads slowly, thus producing a delayed, prolonged reaction and price movement. (There are examples where it took the majority of traders more than a year to become aware of essential, fundamental information about a certain market!)
• Events trigger other events with a similar effect on the price, causing a sort of avalanche effect.
• Traders see the price move, shout “Ah! Trend!” and jump onto the bandwagon, thus creating a self fulfilling prophecy.
• Trade algorithms detect a trend, attempt to exploit it and amplify it this way.
• All of this, but maybe on different time scales.

Whatever the cause, hackers prefer experiment over speculation, so let’s test once and for all if trend in price curves really exists and can be exploited in an algorithmic strategy. Exploiting trend obviously means buying at the begin of a trend, and selling at the end. For this, we simply assume that trend begins at the bottom of a price curve valley and ends at a peak, or vice versa. Obviously this assumption makes no difference between ‘real’ trends and random price fluctuations. Such a difference could anyway only be determined in hindsight. However, trade profits and losses by random fluctuations should cancel out each other in the long run, at least when trade costs are disregarded. When trends exist in price curves, this method should produce an overall profit from the the remaining trades that are triggered by real trends and last longer than in random walk curves. And hopefully this profit will exceed the costs of the random trades. At least – that’s the theory. So we’re just looking for peaks and valleys of the price curve. The script of the trend following experiment would look like this (all scripts here are in C for the Zorro platform; var is typedef’d to double):

void run()
{
  var *Price = series(price());
  var *Smoothed = series(Filter(Price,Period));
  
  if(valley(Smoothed))
    enterLong();
  else if(peak(Smoothed))
    enterShort();
}

This strategy is simple enough. It is basically the first strategy from the Zorro tutorial. The valley function returns true when the last price was below the current price and also below the last-but-one price; the peak function returns true when the last price was above the current and the last-but-one. Trades are thus entered at all peaks and valleys and closed by reversal only. For not getting too many random trades and too high trade costs, we’re removing the high frequencies from the price curve with some smoothing indicator, named Filter in the code above. Trend, after all, is supposed to be longer-lasting and should manifest itself in the lower frequencies. This is an example trade produced by this system: The black jagged line in the chart above is the raw price curve, the red line is the filtered curve. You can see that it lags behind the black curve a little, which is typical of smoothing indicators. It has a peak at the end of September and thus entered a short trade (tiny green dot). The red line continues going down all the way until November 23, when a valley was reached. A long trade (not shown in this chart) was then entered and the short trade was closed by reversal (other tiny green dot). The green straight line connects the entry and exit points of the trade. It was open almost 2 months and made in that time a profit of ~ 13 cents per unit, or 1300 pips.

The only remaining question is – which indicator shall we use for filtering out the high frequencies, the ripples and jaggies, from the price curve? We’re spoilt for choice.

The Smoothing Candidates

Smoothing a curve in some way is a common task of all trend strategies. Consequently, there’s a multitude of smoothing, averaging, low-lag and spectral filter indicators at our disposal. We have to test them. The following candidates were selected for the experiment, traditional indicators as well as fancier algorithms:

• SMA, the simple moving average, the sum of the last n prices divided by n.
• EMA, exponential moving average, the current price multiplied with a small factor plus the last EMA multiplied with a large factor. The sum of both factors must be 1. This is a first order lowpass filter.
• LowPass, a second order lowpass filter with the following source code:

  var LowPass(var *Data,int Period)
  {
    var* LP = series(Data[0]);
    var a = 2.0/(1+Period);
    return LP[0] = (a-0.25*a*a)*Data[0]
      + 0.5*a*a*Data[1]
      - (a-0.75*a*a)*Data[2]
      + 2*(1.-a)*LP[1]
      - (1.-a)*(1.-a)*LP[2];
  }

• HMA, the Hull Moving Average, with the following source code:

  var HMA(var *Data,int Period)
  {
    return WMA(series(2*WMA(Data,Period/2)-WMA(Data,Period),sqrt(Period))
  }

• ZMA, Ehler’s Zero-Lag Moving Average, an EMA with a correction term for removing lag. This is the source code:

  var ZMA(var *Data,int Period)
  {
    var *ZMA = series(Data[0]);
    var a = 2.0/(1+Period);
    var Ema = EMA(Data,Period);
    var Error = 1000000;
    var Gain, GainLimit=5, BestGain=0;
    for(Gain = -GainLimit; Gain < GainLimit; Gain += 0.1) {
       ZMA[0] = a*(Ema + Gain*(Data[0]-ZMA[1])) + (1-a)*ZMA[1];
       var NewError = Data[0] - ZMA[0];
       if(abs(Error) < newError) {
         Error = abs(newError);
         BestGain = Gain;
       }
    }
    return ZMA[0] = a*(Ema + BestGain*(Data[0]-ZMA[1])) + (1-a)*ZMA[1];
  }

• ALMA, Arnaud Legoux Moving Average, based on a shifted Gaussian distribution (described in this paper):

  var ALMA(var *Data, int Period)
  {
    var m = floor(0.85*(Period-1));
    var s = Period/6.0;
    var alma = 0., wSum = 0.;
    int i;
    for (i = 0; i < Period; i++) {
      var w = exp(-(i-m)*(i-m)/(2*s*s));
      alma += Data[Period-1-i] * w;
      wSum += w;
    }
    return alma / wSum;
  }

• Laguerre, a 4-element Laguerre filter:

  var Laguerre(var *Data, var alpha)
  {
    var *L = series(Data[0]);
    L[0] = alpha*Data[0] + (1-alpha)*L[1];
    L[2] = -(1-alpha)*L[0] + L[1] + (1-alpha)*L[2+1];
    L[4] = -(1-alpha)*L[2] + L[2+1] + (1-alpha)*L[4+1];
    L[6] = -(1-alpha)*L[4] + L[4+1] + (1-alpha)*L[6+1];
    return (L[0]+2*L[2]+2*L[4]+L[6])/6;
  }

• Linear regression, fits a straight line between the data points in a way that the distance between each data point and the line is minimized by the least-squares rule.
• Smooth, John Ehlers’ “Super Smoother”, a 2-pole Butterworth filter combined with a 2-bar SMA that suppresses the Nyquist frequency:

  var Smooth(var *Data,int Period)
  {
    var f = (1.414*PI) / Period;
    var a = exp(-f);
    var c2 = 2*a*cos(f);
    var c3 = -a*a;
    var c1 = 1 - c2 - c3;
    var *S = series(Data[0]);
    return S[0] = c1*(Data[0]+Data[1])*0.5 + c2*S[1] + c3*S[2];
  }

• Decycle, another low-lag indicator by John Ehlers. His decycler is simply the difference between the price and its fluctuation, retrieved with a highpass filter:

  var Decycle(var* Data,int Period)
  {
    return Data[0]-HighPass2(Data,Period);
  }

The above source codes have been taken from the indicators.c file of the Zorro platform, which contains source codes of all supplied indicators.

Comparing Step Responses

What is the best of all those indicators? For a first impression here’s a chart showing them reacting on a simulated sudden price step. You can see that some react slow, some very slow, some fast, and some overshoot: You can generate such a step response diagram with this Zorro script:

// compare the step responses of some low-lag MAs
function run()
{
  set(PLOTNOW);
  BarPeriod = 60;
  MaxBars = 1000;
  LookBack = 150;
  asset("");   // don't load an asset
  ColorUp = ColorDn = 0;  // don't plot a price curve
  PlotWidth = 800;
  PlotHeight1 = 400;

  var *Impulse = series(ifelse(Bar>200 && Bar<400,1,0)); // 0->1 impulse
  int Period = 50;
  plot("Impulse",Impulse[0],0,GREY);
  plot("SMA",SMA(Impulse,Period),0,BLACK);
  plot("EMA",EMA(Impulse,Period),0,0x808000);
  plot("ALMA",ALMA(Impulse,Period),0,0x008000);
  plot("Laguerre",Laguerre(Impulse,4.0/Period),0,0x800000);
  plot("Hull MA",HMA(Impulse,Period),0,0x00FF00);
  plot("Zero-Lag MA",ZMA(Impulse,Period),0,0x00FFFF); 
  plot("Decycle",Decycle(Impulse,Period),0,0xFF00FF);
  plot("LowPass",LowPass(Impulse,Period),0,0xFF0000);
  plot("Smooth",Smooth(Impulse,Period),0,0x0000FF);
}

We can see that from all the indicators, the ALMA seems to be the best representation of the step, while the ZMA produces the fastest response with no overshoot, the Decycler the fastest with some overshoot, and Laguerre lags a lot behind anything else. Obviously, all this has not much meaning because the time period parameters of the indicators are not really comparable. So the step response per se does not reveal if the indicator is well suited for trend detection. We have no choice: We must use them all.

The Trend Experiment

For the experiment, all smoothing indicators above will be applied to a currency, a stock index, and a commodity, and will compete in exploiting trend. Because different time frames can represent different trader groups and thus different markets, the indicators will be applied to price curves made of 15-minutes, 1-hour, and 4-hours bars. And we’ll also try different indicator time periods in 10 steps from 10 to 10000 bars.

So we’ll have to program 10 indicators * 3 assets * 3 bar sizes * 10 time periods = 900 systems. Some of the 900 results will be positive, some negative, and some zero. If there are no significant positive results, we’ll have to conclude that trend in price curves either does not exist, or can at least not be exploited with this all-too-obvious method. However we’ll likely get some winning systems, if only for statistical reasons.

The next step will be checking if their profits are for real or just caused by a statistical effect dubbed Data Mining Bias. There is a method to measure Data Mining Bias from the resulting equity curves: the notorious White’s Reality Check. We’ll apply that check to the results of our experiment. If some systems survive White’s Reality Check, we’ll have the once-and-for-all proof that markets are ineffective, trends really exist, and algorithmic trend trading works.

Next Step…