The C code of the used Hann filter, straight from Ehler’s article:

var Hann(vars Data,int Length)
{
  var Filt = 0, Coeff = 0;
  int i; for(i=1; i<=Length; i++) {
    Filt += (1-cos(2*PI*i/(Length+1)))*Data[i-1];
    Coeff += 1-cos(2*PI*i/(Length+1));
  }
  return Filt/fix0(Coeff);
}

The fix0 function in the denominator is a convenience function for fixing division by zero issues.

We will now apply this Hann filter to the undersampled SPY price curve. Undersampling means: Take only every n-th price, and throw the rest away. We use the modulo operator to detect every 5th bar:

void run()
{
 StartDate = 20220101;
 EndDate = 20221231;
 BarPeriod = 1440;
 set(PLOTNOW);
 asset("SPY");
 vars Samples = series();
 if(Init || Bar%5 == 0) // sample only every 5th bar
   Samples[0] = priceC(0);
 else
   Samples[0] = Samples[1];
 plot("Hann6",Hann(Samples,6),LINE,MAGENTA);
 plot("Hann12",Hann(Samples,12),LINE,BLUE);
 plot("Hann",Hann(seriesC(),12),0,DARKGREEN);
}

The resulting chart:

The blue line is the Hann filter of the undersampled curve with period 12, the magenta line with period 6. For comparison I’ve added the Hann filter output from the original curve (thin green line). We can see that the green line has less lag, but is also less smooth. Will using undersampled data improve a trading system? Well… I guess it depends on the system.

The Hann indicator and undersampling test script can be downloaded from the 2022 script repository.

The LR_EMA indicator can be directly converted from Apirine’s MetaStock to C:

var LREMA(int Periods,int Pds,var Mltp)
{
  var Mltp1 = 2./(Periods+1);
  var LR = LinearReg(seriesC(),Pds);
  vars Dists = series(abs(LR-priceC()));
  var ST = (Dists[0]-MinVal(Dists,Pds))/
     max(MaxVal(Dists,Pds)-MinVal(Dists,Pds),0.01);
  var Rate = min(1.,Mltp1*(1+ST*Mltp));
  return EMA(priceC(),Rate);
}

 The max and min functions prevent divisions by zero and EMA alpha parameters (Rate) above 1. This can otherwise happen at the begin of the backtest when price series are still flat.

The LREMA(20,20,5) and EMA(20) applied on SPY replicates the chart in Apirine’s article:

 We can see that the LREMA (blue line) follows the price more closely than the EMA (red line). Which is not really surprising, because LREMA period is smaller at most places. But how good can the LREMA identify trend changes in a real trading system?

For testing this, we write a simple SPY system that concurrently trades with EMA, and with LREMA. A single position is entered when the faster EMA crosses over a slower EMA, and closed when it crosses back. The LREMA is used likewise. For not comparing apples with oranges, we optimize all time periods and test with walk-forward analysis. The code of this trading system in C:

void run()
{
 BarPeriod = 1440;
 LookBack = 200;
 StartDate = 2005;
 EndDate = 2022;
 assetList("AssetsIB");
 asset("SPY");
// walk-forward optimization setup
 set(PARAMETERS,TESTNOW,PLOTNOW);
 NumWFOCycles = 8;
// run a loop over the two algos
 while(algo(loop("LR_EMA","EMA")))
 {
  vars Signals1, Signals2;
  int Color;
// optimize time periods
  int Period1 = optimize(20,10,50,10);
  int Period2 = Period1*optimize(2.5,1.5,3,0.5);
  if(Algo == "LR_EMA") {
   Signals1 = series(LREMA(Period1,Period1,5));
   Signals2 = series(LREMA(Period2,Period2,5));
   Color = BLUE;
  } else if(Algo == "EMA") {
   Signals1 = series(EMA(seriesC(),Period1));
   Signals2 = series(EMA(seriesC(),Period2));
   Color = GREY;
  }
// trade on fast/slow crossovers
  if(crossOver(Signals1,Signals2)) enterLong();
  if(crossUnder(Signals1,Signals2)) exitLong();
// plot equity curve
  plot(Algo,ProfitOpen+ProfitClosed,LINE+AXIS2,Color);
 }
}

This script trades two different algos and compares the resulting equity curves.  The fast and slow periods are optimized separately for both algorithms. The slow period is the fast period multiplied with a factor. This guarantees that the slow period is always bigger than the fast period during the optimization. The system must be first trained by clicking the [Train] button on the Zorro panel. After training, it automatically starts a backtest with walk-forward analysis.

Training and analysis take about 2 seconds. The resulting equity curve is plotted in a blue line for the LR_EMA algo, and in a grey line for the standard EMA algo:

 We can see that both algorithms yield a positive result, but the standard EMA a bit more so than the LR_EMA. Not only the profit is smaller, the drawdowns are also worse with the new invented indicator – which is probably the reason why the article, like almost any article about a new indicator, did not contain an example system or a backtest. But maybe the walk-forward period optimization was simply more effective on the fixed-period EMA, than on the variable-period LR_EMA. If so, then the LR_EMA would fare better with unoptimized systems.

The LREMA indicator and the EMA/LREMA trading system can be downloaded from the 2022 script repository.

For his loop chart, Ehlers filtered the low and high frequencies out of the two data series with a roofing filter. Its code in C for Zorro:

var Roofing(var *Data,int HPeriod,int LPeriod)
{
  var f = 1.414*PI/HPeriod;
  var hpa1 = exp(-f);
  var hpc2 = 2*hpa1*cos(f/2);
  var hpc3 = -hpa1*hpa1;
  var hpc1 = (1 + hpc2 - hpc3) / 4;
  f = 1.414*PI/LPeriod;
  var ssa1 = exp(-f);
  var ssc2 = 2*ssa1*cos(f/2);
  var ssc3 = -ssa1*ssa1;
  var ssc1 = 1 - ssc2 - ssc3;

  vars HP = series(0,3);
  HP[0] = hpc1*(Data[0] - 2*Data[1] + Data[2]) + hpc2*HP[1] + hpc3*HP[2];
  vars SS = series(HP[0],3);
  SS[0] = ssc1*(HP[0] + HP[1])/2 + ssc2*SS[1] + ssc3*SS[2];
  var Scaled = EMA(SS[0]*SS[0],.0242);
  return SS[0]/sqrt(Scaled);
}

For demonstration purposes we apply that filter to the FDX price and volume series. The filtered data points serve as XY coordinates for our curve – the “Ehlers Loop”. The following script reads 3 months stock data from an online source and displays it in a scatter plot where the XY points are connected with splines:

function run()
{
  StartDate = ymd(wdate(NOW)-90); // 90 days before today
  BarPeriod = 1440;
  asset("FDX");
  var PriceRMS = Roofing(seriesC(),125,20);
  var VolRMS = Roofing(series(marketVol()),125,20);

  if(is(LOOKBACK)) return; // don't plot the lookback period
  plotGraph("Loop",VolRMS,PriceRMS,DOT|GRAPH,BLUE);
  plotGraph("Loops",VolRMS,PriceRMS,SPLINE|GRAPH,TRANSP|GREY);
  if(is(EXITRUN))
    plotGraph("Last",VolRMS,PriceRMS,SQUARE|GRAPH,RED);
}

The plotGraph function is used to display each coordinate with a blue dot. The last day is marked with a red square.

Ehlers intended his loops for discretionary trading, but it can of course also be automated. For instance, the last N coordinates could be used as inputs for Zorro’s neural net, which can then be trained to predict tomorrow’s price. Or even simpler, the slope of a polynomial regression through the last points could trigger a buy order when positive, or a sell order when negative. You can use Zorro’s polyfit function and filter with the regression error. I leave this trading system as an exercise to the reader.

The same method can be used to compare a stock with an index for a pairs rotation strategy. We’re using RTX and SPY:

function run()
{
  BarPeriod = 1440;
  LookBack = 300;
  StartDate = 20210801;
  EndDate = 20211130;

  assetAdd("RTX","YAHOO:*");
  asset("RTX");
  var Y = Roofing(seriesC(),125,20);
  assetAdd("SPY","YAHOO:*");
  asset("SPY");
  var X = Roofing(seriesC(),125,20);

  if(is(LOOKBACK)) return;
  plotGraph("Loop",X,Y,DOT|GRAPH,BLUE);
  plotGraph("Loops",X,Y,SPLINE|GRAPH,TRANSP|GREY);
  if(is(EXITRUN))
    plotGraph("Last",X,Y,SQUARE|GRAPH,RED);
}

The resulting chart for the period from August to November 2021:

I found that Ehlers’ roofing filter is sensitive to the lookback period. It needs some time to ‘swing in’. A too short lookback period, like Zorro’s default 80 bars, produces a visibly different loop. With 300 bars we’re on the safe side.

What can you now do with this loop chart? Ehlers proposed to switch between stock and index depending on the loop rotation and angle quadrant. Our example above, with the SPY index on the Y axis and the stock on the X axis, has a clockwise rotating loop. On clockwise rotations, buy the stock and sell the index from 5 to 10 on the clock face. Sell the stock and buy the index from 11 to 4. Be out of the market between 4 and 5, and have both positions between 10 and 11.

On a counter clockwise rotation, do just the opposite. This was intended for discretionary trading, but can of course be automated with a script. Which I also gracefully leave to the reader.

The Roofing indicator and the Ehlers Loop scripts can be downloaded from the 2022 script repository. The Zorro software can be downloaded from https://zorro-project.com.

The trading algorithm “Sell in August and buy back in October” could have been realized with the Zorro software in just 5 lines of C:

asset("SPY");
if(month() == 8 && tdm() == 1) // sell 1st trading day of August
  exitLong();
else if(month() == 10 && tdm() == 1) // buy back 1st trading day of October
  enterLong();

Alas, this seasonality trading system was apparently way too simplistic for Markos Katsanos. His version is a veritable monster with many, many trade entry and exit conditions. The trading script below is a 1:1 translation from his AmiBroker code to C for the Zorro platform. I only moved his VFI indicator – a variant of the On Balance Volume, but with a lot more signals – in a separate indicator function because I didn’t like trading logic cluttered with indicator code.

var priceAvg(int Offset) {
  return (priceC(Offset)+priceH(Offset)+priceL(Offset))/3;
}

var VFI(var Period,var Coef, var VCoef)
{
  vars Inters = series(log(priceAvg(0))-log(priceAvg(1)));
  var Vinter = StdDev(Inters,30);
  var Cutoff = Coef * Vinter * priceC();
  vars Volumes = series(marketVol());
  var Vave = SMA(Volumes+1,Period);
  var Vmax = Vave * VCoef;
  var VC = min(Volumes[0],Vmax);
  var MF = priceAvg(0)-priceAvg(1);
  vars VCPs = series(ifelse(MF > Cutoff,VC,ifelse(MF < -Cutoff,-VC,0)));
  var VFI1 = Sum(VCPs,Period)/Vave;
  return EMA(VFI1,3);
}

function run()
{
  StartDate = 2006;
  EndDate = 2022;
  BarPeriod = 1440; // 1 day
  LookBack = 150;

  assetList("AssetsIB");
  MaxLong = 1;
  Capital = 100000;
  Margin = Equity; // invest all you have
  Leverage = 1;
  BarZone = EST;
  Fill = 3; // enter/exit at next day open
  set(PARAMETERS,TESTNOW,PLOTNOW);

  asset("VIX");
  var VIXdn = (priceC(0)/HH(25,0)-1)*100;
  var VIXup = (priceC(0)/LL(25,0)-1)*100;

  asset("SPY");
  int SellMonth = optimize(8,5,8,1);
  var VIXupMax = optimize(60,50,60,10);
  var Crit = -optimize(20,15,20,5); //VFI SELL
  var K = optimize(1.5,1.3,1.7,.2); // ATR/VIX RATIO
  vars ATRs = series(ATR(15));
  var ATRDn = (ATRs[0]/MaxVal(ATRs,25)-1)*100;
  var ATRUp = (ATRs[0]/MinVal(ATRs,25)-1)*100;
  vars VFIs = series(VFI(130,0.2,2.5));
  vars SMAVFIs = series(SMA(VFIs,10));
  bool VolCondition = (VIXup < VIXupMax || ATRUp < K*VIXupMax ) && VFIs[0] > Crit;
  bool Buy = (month() >= 10 || month() < SellMonth) && ref(VolCondition,1) != 0;
  bool SellSeasonal = month() == SellMonth ; //SEASONAL
  bool SellVolatility = VIXup > 2*VIXupMax ; //VOLATILITY EXIT
  bool SellMF = crossUnder(VFIs,Crit) && SMAVFIs[0] < SMAVFIs[1] ;
  bool Sell = SellSeasonal || ref(SellVolatility,1) != 0 || ref(SellMF,1) != 0;

  if(Sell)
    exitLong();
  else if(Buy)
    enterLong();
}

Phew. And who said that you need at least 30 historical trades per optimized parameter? We’re optimizing the heck out of sell month, threshold, ratio, and other parameters, and won’t care that this strategy trades only once per year. All is in-sample of course, since the low number of trades prevents walk-forward optimization. As can be expected, our reward is a splendid backtest:

We can see that the system managed to avoid the 2008 market drop, since it doubtlessly knew it in advance due to the in-sample optimization. So take the result with a grain – or better, a bag – of salt. For the audacious experimenter, the VFI indicator and the seasonality trading system can be downloaded from the 2022 script repository. You need Zorro S 2.47 or above for supporting volume and for the ref macro. And since the question recently came up: you can find the script repositories under “Links & Download” on the right side of this page.

Suppose you’re developing an algorithmic trading strategy, following all rules of proper system development. But you are not aware that your trading algorithm has no statistical edge. The strategy is worthless, the trading rules equivalent to random trading, the profit expectancy – aside from transaction costs – is zero. The problem: you will rarely get a zero result in a backtest. A random trading strategy will in 50% of cases produce a negative backtest result, in 50% a positive result. But if the result is negative, you’re normally tempted to tweak the code or select assets and time frames until you finally got a profitable backtest. Which will happen relatively soon even when applying random modifications to the system. That’s why there are so many unprofitable strategies around, with nevertheless great backtest performances.

Does this mean that backtests are worthless? Not at all. But it is essential to know whether you can trust the test, or not.

The test-the-backtest experiment

There are several methods for verifying a backtest. None of them is perfect, but all give insights from different viewpoints. We’ll use the Zorro algo trading software, and run our experiments with the following test system that is optimized and backtested with walk-forward analysis:

function run()
{
  set(PARAMETERS,TESTNOW,PLOTNOW,LOGFILE);
  BarPeriod = 1440;
  LookBack = 100;
  StartDate = 2012;
  NumWFOCycles = 10;

  assetList("AssetsIB");
  asset("SPY");

  vars Signals = series(LowPass(seriesC(),optimize(10,2,20,2)));
  vars MMIFast = series(MMI(seriesC(),optimize(50,40,60,5)));
  vars MMISlow = series(LowPass(MMIFast,100));

  MaxLong = 1;
  if(falling(MMISlow)) {
    if(valley(Signals))
      enterLong();
    else if(peak(Signals))
      exitLong();
  }
}

This is a classic trend following algorithm. It uses a lowpass filter for trading at the peaks and valleys of the smoothed price curve, and a MMI filter (Market Meanness Index) for distinguishing trending from non-trending market periods. It only trades when the market has switched to rend regime, which is essential for profitable trend following systems. It opens only long positions. Lowpass and MMI filter periods are optimized, and the backtest is a walk-forward analysis with 10 cycles.

The placebo trading system

It is standard for experiments to compare the real stuff with a placebo. For this we’re using a trading system that has obviously no edge, but was tweaked with the evil intention to appear profitable in a walk-forward analysis. This is our placebo system:

void run()
{
  set(PARAMETERS,TESTNOW,PLOTNOW,LOGFILE);
  BarPeriod = 1440;
  StartDate = 2012;
  setf(TrainMode,BRUTE);
  NumWFOCycles = 9;

  assetList("AssetsIB");
  asset("SPY");

  int Pause = optimize(5,1,15,1);
  LifeTime = optimize(5,1,15,1);

// trade after a pause...
  static int NextEntry;
  if(Init) NextEntry = 0;
  if(NextEntry-- <= 0) {
    NextEntry = LifeTime+Pause;
    enterLong();
  }
}

This system opens a position, keeps it a while, then closes it and pauses for a while. The trade and pause durations are walk-forward optimized between 1 day and 3 weeks. LifeTime is a predefined variable that closes the position after the given time. If you don’t believe in lucky trade patterns, you can rightfully assume that this system is equivalent to random trading. Let’s see how it fares in comparison to the trend trading system.

Trend trading vs. placebo trading

This is the equity curve with the trend trading system from a walk forward analysis from 2012 up to 3/2022:

The plot begins 2015 because the preceding 3 years are used for the training and lookback periods. SPY follows the S&P500 index and rises in the long term, so we could expect anyway some profit with a long-only system. But this system, with profit factor 3 and R2 coefficient 0.65 appears a lot better than random trading. Let’s compare it with the placebo system:

The placebo system produced profit factor 2 and R2 coefficient 0.77. Slightly less than the real system, but in the same performance range. And this result was also from a walk-forward analysis, although with 9 cycles – therefore the later start of the test period. Aside from that, it seems impossible to determine solely from the equity curve and performance data which system is for real, and which is a placebo.

Checking the reality

Methods to verify backtest results are named ‘reality check’. They are specific to the asset and algorithm; in a multi-asset, multi-algo portfolio, you need to enable only the component you want to test. Let’s first see how the WFO split affects the backtest. In this way we can find out whether our backtest result was just due to lucky trading in a particular WFO cycle. We’re going to plot a WFO profile that displays the effect of the number of walk-forward cycles on the result. For this we outcomment the NumWFOCycles = … line in the code, and run it in training mode with the WFOProfile.c script:

#define run strategy
#include "trend.c" // <= your script
#undef run
#define CYCLES 20 // max WFO cycles

function run()
{
  set(TESTNOW);
  NumTotalCycles = CYCLES-1;
  NumWFOCycles = TotalCycle+1;
  strategy();
}

function evaluate()
{
  var Perf = ifelse(LossTotal > 0,WinTotal/LossTotal,10);
  if(Perf > 1)
    plotBar("WFO+",NumWFOCycles,NumWFOCycles,Perf,BARS,BLACK);
  else
    plotBar("WFO-",NumWFOCycles,NumWFOCycles,Perf,BARS,RED);
}

We’re redefining the run function to a different name. This allows us to just include the tested script and train it with WFO cycles from 2 up to the number defined by CYCLES. A backtest is executed after training. If an evaluate function is present, Zorro runs it automatically after any backtest. It plots a histogram bar of the profit factor (y axis) from each number of WFO cycles. First, the WFO profile of the trend trading system:

We can see that the performance rises with the number of cycles. This is typical for a system that adapts to the market. All results are positive with a profit factor > 1. Our arbitrary choice of 10 cycles produced a less than average result. So we can at least be sure that this backtest result was not caused by a particularly lucky number of WFO cycles.

The WFO profile of the placebo system:

This time the number of WFO cycles had a strong random effect on the performance. And it is now obvious why I used 9 WFO cycles for that system. For the same reason I used brute force optimization, since it increases WFO variance and thus the chance to get lucky WFO cycle numbers. That’s the opposite of what we normally do when developing algorithmic trading strategies.

WFO profiles give insight into WFO cycle dependency, but not into randomness or overfitting by other means. For this, more in-depth tests are required. Zorro supports two methods, the Montecarlo Reality Check (MRC) with randomized price curves, and White’s Reality Check (WRC) with detrended and bootstrapped equity curves of strategy variants. Both methods have their advantages and disadvantages. But since strategy variants from optimizing can only be created without walk-forward analysis, we’re using the MRC here.

The Montecarlo Reality Check

First we test both systems with random price curves. Randomizing removes short-term price correlations and market inefficiencies, but keeps the long-term trend. Then we compare our original backtest result with the randomized results. This yields a p-value, a metric of the probability that our test result was caused by randomness. The lower the p-Value, the more confidence we can have in the backtest result. In statistics we normally consider a result significant when its p-Value is below 5%.

The basic algorithm of the Montecarlo Reality Check (MRC):

1. Train your system and run a backtest. Store the profit factor (or any other performance metric that you want to compare).
2. Randomize the price curve by randomly swapping price changes (shuffle without replacement).
3. Train your system again with the randomized data and run a backtest. Store the performance metric.
4. Repeat steps 2 and 3 1000 times.
5. Determine the number N of randomized tests that have a better result than the original test. The p-Value is N/1000.

If our backtest result was affected by an overall upwards trending price curve, which is certainly the case for this SPY system, the randomized tests will be likewise affected. The MRC code:

#define run strategy
#include "trend.c" // <= your script
#undef run
#define CYCLES 1000

function run()
{
  set(PRELOAD,TESTNOW);
  NumTotalCycles = CYCLES;
  if(TotalCycle == 1) // first cycle = original
    seed(12345); // always same random sequence
  else
    Detrend = SHUFFLE;
  strategy();
  set(LOGFILE|OFF); // don't export files
}

function evaluate()
{
  static var OriginalProfit, Probability;
  var PF = ifelse(LossTotal > 0,WinTotal/LossTotal,10);
  if(TotalCycle == 1) {
    OriginalProfit = PF;
    Probability = 0;
  } else {
    if(PF < 2*OriginalProfit) // clip image at double range
      plotHistogram("Random",PF,OriginalProfit/50,1,RED);
    if(PF > OriginalProfit)
      Probability += 100./NumTotalCycles;
  }
  if(TotalCycle == NumTotalCycles) { // last cycle
    plotHistogram("Original",
     OriginalProfit,OriginalProfit/50,sqrt(NumTotalCycles),BLACK);
    printf("\n-------------------------------------------");
    printf("\nP-Value %.1f%%",Probability);
    printf("\nResult is ");
    if(Probability <= 1)
      printf("highly significant") ;
    else if(Probability <= 5)
      printf("significant");
    else if(Probability <= 15)
      printf("maybe significant");
    else
      printf("statistically insignificant");
    printf("\n-------------------------------------------");
  }
}

This code sets up the Zorro platform to train and test the system 1000 times. The seed setting ensures that you get the same result on any MRC run. From the second cycle on, the historical data is shuffled without replacement. For calculating the p-value and plotting a histogram of the MRC, we use the evaluate function again. It calculates the p-value by counting the backtests resulting in higher profit factors than the original system. Depending on the system, training and testing the strategy a thousand times will take several minutes with Zorro. The resulting MRC histogram of the trend following system:

The height of a red bar represents the number of shuffled backtests that ended at the profit factor shown on the x axis. The black bar on the right (height is irrelevant, only the x axis position matters) is the profit factor with the original price curve. We can see that most shuffled tests came out positive, due to the long-term upwards trend of the SPY price. But our test system came out even more positive. The p-Value is below 1%, meaning a high significance of our backtest. This gives us some confidence that the simple trend follower can achieve a similar result in real trading.

This cannot be said from the MRC histogram of the placebo system:

The backtest profit factors now extend over a wider range, and many were more profitable than the original system. The backtest with the real price curve is indistinguishable from the randomized tests, with a p-value in the 40% area. The original backtest result of the placebo system, even though achieved with walk-forward analysis, is therefore meaningless.

It should be mentioned that the MRC cannot detect all invalid backtests. A system that was explicitly fitted to a particular price curve, for instance by knowing in advance its peaks and valleys, would get a low p-value by the MRC. No reality check could distinguish such a system from a system with a real edge. Therefore, neither MRC nor WRC can give absolute guarantee that a system works when it passes the check. But when it does not pass, you’re advised to better not trade it with real money.

I have uploaded the strategies to the 2022 script repository. The MRC and WFOProfile scripts are included in Zorro version 2.47.4 and above. You will need Zorro S for the brute force optimization of the placebo system.

The algorithm from Vitali’s article, slightly reformulated for clarity:

RSEMA[0] = Alpha * Price + (1-Alpha) * RSEMA[1]

This is a classical EMA, but with a volatility dependent alpha factor:

Alpha = 2/(TimePeriod+1) * (1 + 10*abs(EMAUp-EMADn)/(EMAUp+EMADn))

EMAUp and EMADn are the VIX EMA of days with positive returns and the VIX EMA of days with negative returns. The separation into positive and negative days resembles the RSI. And we have a double EMA in this formula, which will hopefully render the result twice as good. The factor 10 is inserted for getting an even better result.

VIX historical data is not available via API from free online sources as far as I know, but it can be downloaded from most brokers with the Zorro Download script. The resulting RSEMA code in C for Zorro:

var RSEMA(int Periods, int PdsEMA, var Mltp)
{
  asset("VIX");
  var VolatC = priceC();
  asset(AssetPrev); // re-select asset
  var VolatUp = series(ifelse(RET(1)>0,VolatC,0)),
    VolatDn = series(ifelse(RET(1)<0,VolatC,0));
  var EMAUp = EMA(VolatUp,PdsEMA), EMADn = EMA(VolatDn,PdsEMA);
  var RS = abs(EMAUp-EMADn)/(EMAUp+EMADn+0.00001)*Mltp;
  var Rate = 2./(Periods+1)*(RS+1);
  return EMA(priceC(),Rate);
}

AssetPrev is the asset that was selected when the RSEMA function is called. Adding 0.00001 to the divisor prevents a crash by division by zero. Note that the often used EMA is available in two versions, to be called with a data series or with a single variable. I’m using the variable version here.

The RSEMA(10,10,10) applied on a SPX500 chart replicates the chart in the TASC article:

We can see that the RSEMA (blue) has noticeably less lag than the EMA (red), which is not really surprising because the higher alpha rate is equivalent to a smaller EMA time period. The author believed that EMA crossovers of this indicator can be used as entry points for long trades. But with the caveat that they “produce lots of whipsaws […] and should be used in conjunction with price analysis”.

He’s certainly right. A long-only SPY trading system with RSEMA-EMA crossovers at default parameter values produced a positive end result, as seen in the chart below. But with large drawdowns and, sadly, far less profitable than buy-and-hold. So, be sure to use it only in conjunction with price analysis. Whatever that means. And better not on a real money account.

For your own experiments, the RSEMA indicator and the SPY trading system can be downloaded from the 2022 script repository.

The MyRSI indicator sums up positive and negative changes in a data series, and returns their normalized difference. The C version for Zorro:

var MyRSI(vars Data,int Period)
{
 var CU = SumUp(Data,Period+1);
 var CD = -SumDn(Data,Period+1);
 return ifelse(CU+CD != 0,(CU-CD)/(CU+CD),0);
}

For eliminating noise, the function below (NET = Noise Elimination Technology) calculates the Kendall Correlation of a data series with a rising slope. This is supposed to remove all components that are not following the main trend of the indicator output. The code:

var NET(vars Data,int Period)
{
 int i,k;
 var Num = 0;
 for(i=1; i<Period; i++)
   for(k=0; k<i; k++)
     Num -= sign(Data[i]-Data[k]);
 var Denom = .5*Period*(Period-1);
 return Num/Denom;
}

For making the effect of the denoising visible, we produce a SPY chart from 2019-2020. The red line is the original MyRSI, the blue line the de-noised version:

The technology appears to work well in this example for removing the noise. But the NET function is not meant as a replacement of a lowpass or smoothing filter: its output is always in the -1 .. +1 range, so it can be used for de-noising oscillators, but not, for instance, to generate a smoothed version of the price curve.

One might be tempted to use the blue line in the above chart for trade signals. RSI-based signals usually trigger a long trade when the indicator crosses below a lower threshold, and a short trade when it crosses above an upper threshold. But alas, neither is strategy development this easy nor is the de-noised MyRSI the holy grail of indicators. I tested several methods to use it for generating SPY trade signals, but neither was a strong improvement over a simple lowpass filter strategy or even to buy-and-hold. Maybe a reader can find a more convincing solution?

The MyRSI and NET functions and the script for generating the chart can be downloaded from the 2020 script repository. You need Zorro 2.33 or above.

Theoretically, many trend trading systems would fall in the Turk category since they are based on following the herd. But here we’re not looking into trader’s opinions of the current market situation, but into their expectations of the future. There are several methods. The most usual is evaluating the Commitment Of Traders report, for which many platforms, also Zorro, provide easy-to-use indicators. But some publications (1) found that using the COT report for predicting the markets produces mixed results at best.

There’s a more precise way to get the market’s expectations. It’s using options premiums. If an option expires in 6 weeks, its current premium reflects what option buyers or sellers think about the underlying price in 6 weeks.

The price probability distribution

For deriving the expected underlying price at expiration, we have to take all strike prices in consideration. Here’s the algorithm in short (I haven’t invented it, a longer description can be found in (2)). Assume SPY is currently trading at $200. For getting the probability that it will rise to between 210 and 220 in 6 weeks, we’re looking into the 210 call and the 220 call. Suppose they trade at $14 and $10. So we can buy the 210 call and sell the 220 call and pay $4 difference. If at expiration SPY is under 210, both contracts have no worth and we lose the 4 dollars. If it is above 220, our gain is the $10 strike difference minus $4 premium difference, so we win 6 dollars. If it is between 210 and 220, our win or loss is also inbetween, with an average of (-4+6)/2 = 1 dollar.

When options prices are “fair”, i.e. solely determined by probabilities where the price will end up at expiration, summing up over all possible outcomes yields zero profit or loss. So

-$4×L + $1×M + $6×H = $0

where L is the probability for SPY to end up below 210, M is the probability that it will be between 210 and 220, and H is the probability that it will be above 220. Since one of these three alternatives will always happen, the sum of all 3 probabilities must be 1:

L + M + H = 1

Now let’s assume that we know L already. Then we have two equations with two unknowns, which are easily solved:

5L – 5H = 1    =>  H = L – 0.2
10L + 5M = 6    =>  M = 1.2 -2L 

Assuming L = 50% yields H = 30% and M = 20%.

How do we now get the value L? We simply take the lowest strike offered at the market, and assume that the probability of SPY ending up even below that is zero. So our first L is 0.  We can now use the method above to calculate the M belonging to the interval between the lowest and the second-lowest strike. That M is then added to L since it’s the probability of SPY ending up at or below the second-lowest strike. We continue that process with the next interval, getting a specific L and M for any interval until we arrived at the highest strike. If the traders have been consistent with their assumptions, the final L – the sum over all Ms – should be now at or close to 1.

Here’s a small script that displays the 6-weeks L distribution of SPY:

void main() 
{
	StartDate = 20170601;
	LookBack = 0;

	assetList("AssetsIB");
	asset("SPY"); 

// load today's contract chain
	contractUpdate(0,0,CALL|PUT);
	printf("\n%i contracts",NumContracts);
	if(!NumContracts) return;

// get underlying price
	var Price,Current = priceClose(0);
	printf("\nCurrent price %.2f",Current);

// plot CPD histogram	
	contractCPD(45);	// 6 weeks
	int N = 0;
	for(Price = 0.75*Current; Price < 1.25*Current; N++, Price += 0.01*Current)
		plotBar("CPD",N,floor(Price),cpd(Price),BARS|LBL2,RED);
	printf("\nExpected price %.2f",cpdv(50));

// compare with real future price
	set(PEEK);
	N = timeOffset(UTC,-45,0,0);
	printf("\nFuture price %.2f",priceClose(N));
}

The result is a histogram of expected probabilities, in percent, that the SPY price will be at or below the price at the x-axis six weeks in the future. I’ve marked the current price with a black bar.

The contractCPD function generates the distribution with the above described algorithm, cpd returns the accumulated probability L (in percent) at a given price, and cpdv returns the price at a given L. Therefore, cpdv(50) is the median of market expectations. In our case, at June 1 2017, a modest price increase to $245 was expected for mid-July (in fact the price ended up at $245.51, but don’t get too excited – often trader’s expectations fall short by several dollars). We can also see an unusal step at the begin of the histogram: about 10% of traders expected a strong drop to below $200. Maybe Trump twittered something that morning.

The strategy

We will now exploit trader’s expectations for a strategy, and this way check if they have any merit. This is our mechanical Turk:

#define Sentiment AssetVar[0]

void run() 
{
	StartDate = 20120102;
	EndDate = 20171231;
	BarPeriod = 1440;

	assetList("AssetsIB");
	asset("SPY");
	MaxLong = MaxShort = 1;

// load today's contract chain
	contractUpdate(0,0,CALL|PUT);
	int N = contractCPD(45);

// increase/decrease market sentiment 
	if(N) {
		var Expect = cpdv(50) - priceClose();
		if(Expect < 0)
			Sentiment = min(Expect,Sentiment+Expect);
		else
			Sentiment = max(Expect,Sentiment+Expect);
	}
	if(Sentiment > 5)
		enterLong();
	else if(Sentiment < -5)
		enterShort();
	
	plot("Sentiment",Sentiment,NEW,RED);
}

We’re checking the market expectation every day and add it to a previously defined Sentiment asset variable. If the expectation changes sign, so does Sentiment. If its amount has accumulated above 5, we buy a long or short position dependent on its sign. This way we’re only considering the market expectation when it’s either relatively strong or had the same direction for several days.

The result:

The system produces about 20% annual return with profit factor 3. The red line in the lower chart is the Sentiment variable – we can see that it often steadily increases, but sometimes also decreases. Since SPY normally rises, shorting it with this strategy produces less profit than going long, but is still slightly profitable with an 1.16 profit factor.

Predicting the price in 6 weeks

What are traders currently thinking of the SPY price in 6 weeks? For this, the first script above needs just be slightly modified so that it does not use historical options data, but connects to IB and downloads the current options chain with all prices:

void main() 
{
	StartDate = NOW;
	LookBack = 0;

	assetList("AssetsIB");
	asset("SPY"); 

// load today's contract chain
	contractUpdate(0,0,CALL|PUT);
	printf("\n%i contracts",NumContracts);
	if(!NumContracts) return;

// get underlying price
	var Price,Current = priceClose(0);
	printf("\nCurrent price %.2f",Current);

// plot CPD histogram
	printf("\nWait time approx %i minutes",1+NumContracts/200);
	contractCPD(45);
	int N = 0;
	for(Price = 0.75*Current; Price < 1.25*Current; N++, Price += 0.01*Current)
		plotBar("CPD",N,floor(Price),cpd(Price),BARS|LBL2,RED);
	printf("\nExpected price %.2f",cpdv(50));
}

This script must be started in Zorro’s Trade mode with the IB plugin selected. Mind the displayed “wait time”. IB sometimes needs up to ten seconds for returning the price of an option contract, so downloading the prices of all 6-weeks contracts can take half an hour or more.

I ran that script today (Dec 11 2018) and the option traders expect the SPY price to rise to $269 in six weeks. So I’ll check the price by the end of January and post here if they have been right.

I’ve added the scripts to the 218 repository. You’ll need Zorro 1.99 or above, and SPY options EOD history for the first 2 scripts. You’ll really have to buy it this time, the free artificial options history won’t do for market sentiment.

Conclusions

• A Turk can beat a thinking machine.
• Option traders tend to underestimate future price changes.
• But they are often right in the price direction.

Literature

(1) Sanders, Irwin, Merrin: Smart Money? The Forecasting Ability of CFTC Large Traders (2007)

(2) Pat Neal, Option Prices Imply a Probability Distribution

To clarify: I’ve selected the system not because of high profit expectancy or clever algorithm, but because it is quite simple and does not need any of the additional data normally used for option systems, such as earnings reports, open interest, implied volatility, or greeks. Which means that you don’t need to call R functions for options math, and you don’t need to pay for iVolatility options data, Zacks earnings, or any other historical data for backtesting the system. The free Zorro version is sufficient.

The book cover praises the system inside:  To reduce your investment risk to nearly zero – Achieve consistent high annual returns in excess of 30% – It does not require you to learn fundamental and technical analyzes, deltas, thetas, gamas, vegas or other Greek goblethegooks of stocks or options  – It does not require the ability to predict market direction – It does not require stock picking skills – It does not require close monitoring.

All statements with which I, of course, highly sympathize. After all, why would we need Greek goblethegooks when we get annual 30% without them! And here are the (simplified) rules of our strategy:

1. Sell a 6 weeks call and a 6 weeks put of an index ETF. Choose strike prices so that the premiums are in the $1..$2 range.
2. If the underlying price touches one of our strike prices, thus threatening an in-the-money expiration, buy back that option and immediately sell a new option of the same type, but to a further expiration date, and a premium that covers the loss.
3. Wait until all options are expired, then go back to 1.

If you have a bit experience with options, you’ll notice that rule 1 describes a strangle combo. And you’ll next notice something strange with rule 2. Right, such a system can never lose, since any loss would apparently be compensated by the premium from the new trade. Have we finally found the Holy Grail, an ever-winning system? 

Strangle profit

For getting an impression of the profit and risk, let’s first check the gain/loss diagram of the 6-week $2 premium strangle. This is the definition of a strangle in the curve plotting script from the last article:

// Strangle
void combo()
{
	optionAdd(1,SELL|CALL,6);
	optionAdd(1,SELL|PUT,-6);
}

The $6 strike-spot distances have been chosen for $2 premium from a hypothetical index ETF with $250 price, multiplier 100, and 15% annual volatility. This is the profit/loss diagram:

Our potential gain is about $400 per combo trade, as expected (2 * 100 * $2 premium). But the price of our index ETF should better not move more than $10 in any direction until expiration. Otherwise the loss can quickly reach the thousand dollar zone. This does not really look like “reduce your investment risk to nearly zero”. But wait, we have rule 2, which will certainly save the day! Let’s put that to the backtest. 

The system

// Quite simple options trading system 
#include <contract.c>

#define PREMIUM	2.00
#define WEEKS	6    // expiration

int i;
var Price;

CONTRACT* findCall(int Expiry,var Premium)
{
	for(i=0; i<50; i++) {
		if(!contract(CALL,Expiry,Price+0.5*i)) return 0;
		if(between(ContractBid,0.1,Premium)) return ThisContract;
	}
	return 0;
}

CONTRACT* findPut(int Expiry,var Premium)
{
	for(i=0; i<50; i++) {
		if(!contract(PUT,Expiry,Price-0.5*i)) return 0;
		if(between(ContractBid,0.1,Premium)) return ThisContract;
	}
	return 0;
}

void run() 
{
	StartDate = 20110101;
	EndDate = 20161231;
	BarPeriod = 1440;
	BarZone = ET;
	BarOffset = 15*60+20; // trade at 15:20 ET
	LookBack = 1;

	assetList("AssetsIB");
	asset("SPY"); // unadjusted!
	Multiplier = 100;

// load today's contract chain
	contractUpdate("SPY",0,CALL|PUT);
	Price = priceClose(); 

// check for in-the-money and roll 	
	for(open_trades) {
		var Loss = -TradeProfit/Multiplier;
		if(TradeIsCall && Price >= TradeStrike) {
			exitTrade(ThisTrade);
			printf("#\nRoll %.1f at %.2f Loss %.2f",
				TradeStrike,Price,TradeProfit);
			CONTRACT* C = findCall(NWEEKS*7,Loss*1.1);
			if(C) {
				MarginCost = 0.15*Price - (C->fStrike-Price);
				enterShort();
			}
		} else if(TradeIsPut && Price <= TradeStrike) {
			exitTrade(ThisTrade);
			printf("#\nRoll %.1f at %.2f Loss %.2f",
				TradeStrike,Price,TradeProfit);
			CONTRACT* C = findPut(NWEEKS*7,Loss*1.1);
			if(C) { 
				MarginCost = 0.15*Price - (Price-C->fStrike);
				enterShort();
			}
		}
	}
	
// all expired? enter new options
	if(!NumOpenShort) { 
		CONTRACT *Call = findCall(NWEEKS*7,PREMIUM); 
		CONTRACT *Put = findPut(NWEEKS*7,PREMIUM); 		
		if(Call && Put) {
			MarginCost = 0.5*(0.15*Price-
				min(Call->fStrike-Price,Price-Put->fStrike));
			contract(Call); enterShort();
			contract(Put); enterShort();
		}
	}
}

A brief discussion of the code (a more detailed intro in system coding can be found in the Black Book). The findCall function gets an expiration time and a premium, and looks through the current option chain for a call contract that matches these two parameters. For this it increases the strike price in 50 steps. If then still no contract is found at or below the desired premium, it returns 0. Otherwise it returns a pointer to the found contract. The findPut function does the same for a put contract.

The run function sets up the backtest time and other parameters for the backtest as well as for live trading. It’s a daily script, and the function runs every day at 3:20 pm Eastern Time. It uses two historical data files for the backtest. The asset function loads a file with the unadjusted SPY prices (why unadjusted? Because determining the strikes-price distances would not work with dividend adjusted prices). The contractUpdate function loads the SPY options chain of that day, either from the broker, or from a file.  Those two files must be present, plus the asset list AssetsIB.csv that contains commission, margin, and other parameters for simulating the broker or exchange where we trade.

The next part of the code implements the miraculous rule 2. It calculates the current loss, closes any position that is at or in the money, and immediately opens a new position, with a premium slightly above our loss (Loss*1.1). This way we’re punishing the market for going against us. The printf function just stores that event in the log, so that we can go through it and better see the fate of those trades.

The last part of the code is the strangle. Note the MarginCost calculation. Margin affects the required capital and thus the backtest performance, so it should reflect your broker’s margin requirement. By default, the margin of a sold option is the premium plus some fixed percentage of the underlying that’s set up in the asset list. But brokers often apply a more complex margin formula for option combos. Here we assume that the margin of a sold strangle is the premium (which is automatically added) plus 15% of the underlying price minus the minimum of the two strike differences. We multiply that by half because we have 2 positions, but the margin formula is for the whole strangle.

The backtest from 2011-2016 needs only about 2 seconds. This is the result (assuming we always open 1 contract):

Monte Carlo Analysis... Median AR 12%
Win 3699$  MI 51.38$  DD 935$  Capital 5108$
Trades 93  Win 59.1%  Avg +39.8p  Bars 24
AR 12%  PF 1.84  SR 1.08  UI 5%  R2 0.89

We have won about 60% of all trades, and made 12% annual return based on Montecarlo analysis.  Not too exciting. What about the “consistent high annual returns in excess of 30%”? And how can we get a $935 drawdown when we always compensate our loss with a new trade?

Is rolling over irrational?

Let’s try the same strategy without the rule 2. This simplifies the script a bit:

// Even simpler options trading system 
#include <contract.c>

#define PREMIUM	2.00
#define WEEKS	6 // expiration

int i;
var Price;

CONTRACT* findCall(int Expiry,var Premium)
{
	for(i=0; i<50; i++) {
		if(!contract(CALL,Expiry,Price+0.5*i)) return 0;
		if(between(ContractBid,0.1,Premium)) return ThisContract;
	}
	return 0;
}

CONTRACT* findPut(int Expiry,var Premium)
{
	for(i=0; i<50; i++) {
		if(!contract(PUT,Expiry,Price-0.5*i)) return 0;
		if(between(ContractBid,0.1,Premium)) return ThisContract;
	}
	return 0;
}

void run() 
{
	StartDate = 20110101;
	EndDate = 20161231;
	BarPeriod = 1440;
	BarZone = ET;
	BarOffset = 15*60+20; // trade at 15:20 ET
	LookBack = 1;
	set(PLOTNOW);
	set(PRELOAD|LOGFILE);

	assetList("AssetsIB");
	asset("SPY"); // unadjusted!
	Multiplier = 100;

// load today's contract chain
	Price = priceClose();
	contractUpdate("SPY",0,CALL|PUT);

// all expired? enter new options
	if(!NumOpenShort) { 
		CONTRACT *Call = findCall(WEEKS*7,PREMIUM); 
		CONTRACT *Put = findPut(WEEKS*7,PREMIUM); 		
		if(Call && Put) {
			MarginCost = 0.5*(0.15*Price-min(Call->fStrike-Price,Price-Put->fStrike));
			contract(Call); enterShort();
			contract(Put); enterShort();
		}
	}
}

Simply removing the rolling over improved the system remarkably:

Monte Carlo Analysis... Median AR 25%
Win 5576$  MI 77.46$  DD 785$  Capital 3388$
Trades 78  Win 80.8%  Avg +71.5p  Bars 35
AR 27%  PF 2.00  SR 0.92  UI 5%  R2 0.92

The equity curve with no rolling:

Now the 25% annual return are somewhat closer to the promised profit. Of course at cost of higher risk, since no limiting mechanism is in place. We could now test other option combos instead of the strangle, for instance a condor for limiting the risk. We can run an optimization for finding out how the profit is affected by different premiums and expirations. I leave that to the reader. The interesting question is why rolling over options, not only with this, but with many option trading systems that we have coded so far, reduces the performance remarkably. Often to the client’s great surprise.

Rolling over with loss compensation establishes in fact a Martingale system. And such a system fares no better in option trading than in the casino. In fact, even worse. In the casino you have at least the same chance with every play. In trading, a losing option combo hints that the market starts trending – and the trend is likely to continue with the rolled over contract. Quite soon you cannot anymore compensate your losses with higher premiums, since you’ll find no contracts at that value. Ok, you could then start increasing the contract volume. If you really did that, you can calculate under the link above how long your account will survive. Rolling over a losing contract is typical irrational human behavior –  but the markets tend to punish irrationality.

Artificial options data

Since the system does not rely on goblethegooks, we can check whether the artificial options data that we created in the first part of this mini series can be used for testing this system. The backtest results above were with real options data. Here’s the result with the synthetic data:

Monte Carlo Analysis... Median AR 31%
Win 7162$  MI 99.49$  DD 1188$  Capital 3866$
Trades 88  Win 81.8%  Avg +81.4p  Bars 30
AR 31%  PF 2.36  SR 1.12  UI 4%  R2 0.88

It’s similar, but not quite identical to the real data. Artificial data represents a more efficient market situation, since its option premiums are identical to their theoretical values, and fundamentals such as earnings reports play no role. You can use it for confirming the real data backtest. Or for saving money, by backtesting a non-goblethegooks system (yes, I like this word) first with artifical data, and only if it looks good, purchasing real data for the final test.

I’ve added the full script to the 2017 repository. You’ll need Zorro version 1.73 or above. You can find the unadjusted SPY data in the History folder of the archive (alternatively, download it with the Zorro command assetHistory( “SPY.US”, FROM_STOOQ | UNADJUSTED)). If you don’t want to create the artificial 2011-2016 options history yourself, you can download it from the historical data archives here. 

Conclusions

• Mind the margin cost in backtests.
• Do not roll over losing contracts.
• If your system has no goblethegooks, try artificial data.

Literature

(1) is the book from which I pulled the system. The book is ok – not better or worse than most other options books, but at only $10, getting it is no mistake. 
(2) is a really good introduction into the options trading matter. Even though its author shamelessly plagiarized the title of my blog, and this even years before I started writing it!

(1) Daniel Mollat, $tock option$, BN Publishing 2011
(2) Philip Z Maymin, Financial Hacking, Wspc 2012

The HFT advantage is receiving price quotes earlier and getting orders filled faster than the majority of market participants. Its profit depends on the system’s latency, the delay between price quote and subsequent order execution at the exchange. Latency is the most relevant factor of a HFT system. It can be optimized in two ways: by minimizing the distance to the exchange, and by maximizing the speed of the trading system. The former is more important than the latter.

The location

Ideally, a HFT server is located directly at the exchange. And indeed most exchanges are dear selling server space in their cellars – the closer to the main network hub, the dearer the space. Electrical signals in a shielded wire travel with about 0.7 .. 0.9 times the speed of light (300 km/ms). 1 m closer to the signal source means a whopping 8 nanoseconds round-trip time advantage. How many trade opportunities disappear after 8 ns? I don’t know, but people are willing to pay for any saved nanosecond.

Unfortunately (or fortunately, from a cost perspective), the HFT system that we’ll examine here can not draw advantage from colocating at the exchange. For reasons that we’ll see soon, it must trade and receive quotes from the NYSE and the CME simultaneously. There’s a high speed cable and also a microwave radio link running between both cities. The theoretically ideal location of such a HFT system is Warren, Ohio, exactly halfway between New York City and Chicago. I don’t know whether there’s a high speed node in Warren, but if it is, the distance of 357 miles to both exchanges would be equivalent to about 4 ms round-trip latency.

Without doubt, a server in this pleasant town would come at lower cost than a server in the cellar of the New York Stock Exchange. My today’s free tip for getting rich: Buy some garage space in Warren, tap into the high speed New York-Chicago cable or radio link, and rent out server racks!

The software

When you’ve already invested money for the optimal location and connection of the HFT system, you’ll also want an algo trading software that matches the required speed. Commercial trading platforms – even Zorro – are normally not fast enough. And you don’t know what they do behind your back. HFT systems are therefore normally not based on a universal trading platform, but directly coded. Not in R or Python, but in a fast language, usually one of the following:

• C or C++. Good combination of high level and high speed. C is easy to read, but very effective and the fastest language of all – almost as fast as machine code.
• Pentium Assembler. Write your algorithm in machine instructions. Faster than even C when the algorithm mostly runs in loops that can be manually optimized. There are specialists for optimizing assembler code. Disadvantage: Any programmer has a hard time to understand assembler programs written by another programmer.
• CUDA, HLSL, or GPU assembler. Run your algorithm on the shader pipeline of the PC’s video card. This makes sense when it is heavily based on vector and matrix operations.
• VHDL. If any software would be too slow and trade success really depends on nanoseconds, the ultimate solution is coding the system in hardware. In VHDL you can define arithmetic units, digital filters, and sequencers on a FPGA (Field Programmable Gate Array) chip with a clock rate up to several 100 MHz. The chip can be directly connected to the network interface.

With exception of VHDL, especially programmers of 3D computer games are normally familar with those high-speed languages. But the standard language of HFT systems is plain C/C++. It is also used in this article.

The trading algorithm

Many HFT systems prey on traders by using fore-running methods. They catch your quote, buy the same asset some microseconds earlier, and sell it to you with a profit. Some exchanges prevent this in the interest of fair play, while other exchanges encourage this in the interest of earning more fees. For this article we won’t use fore-running methods, but simply exploit arbitrage between ES and SPY. We’re assuming that our server is located in Warren, Ohio, and has a high speed connection to Chicago and to New York.

ES is a Chicago traded S&P500 future, exposed to supply and demand. SPY is a New York traded ETF, issued by State Street in Boston and following the S&P500 index (minus State Street’s fees). 1 ES Point is equivalent to 10 SPY cents, so the ES price is about ten times the SPY price. Since both assets are based on the same index, we can expect that their prices are highly correlated. There have been some publications (1) that claim this correlation will break down at low time frames, causing one asset trailing the other. We’ll check if this is true. Any shortlived ES-SPY price difference that exceeds the bid-ask spreads and trading costs constitutes an arbitrage opportunity. Our arbitrage algorithm would work this way:

• Determine the SPY-ES difference.
• Determine its deviation from the mean. 
• If the deviation exceeds the ask-bid spreads plus a threshold, open positions in ES and SPY in opposite directions.
• If the deviation reverses its sign and exceeds the ask-bid spreads plus a smaller threshold, close the positions.

This is the barebone HFT algorithm in C. If you’ve never seen HFT code before, it might look a bit strange:

#define THRESHOLD  0.4  // Entry/Exit threshold 

// HFT arbitrage algorithm
// returns 0 for closing all positions
// returns 1 for opening ES long, SPY short
// returns 2 for opening ES short, SPY long
// returns -1 otherwise
int tradeHFT(double AskSPY,double BidSPY,double AskES,double BidES)
{
	double SpreadSPY = AskSPY-BidSPY, SpreadES = AskES-BidES;
	double Arbitrage = 0.5*(AskSPY+BidSPY-AskES-BidES);

	static double ArbMean = Arbitrage;	
	ArbMean = 0.999*ArbMean + 0.001*Arbitrage;
	static double Deviation = 0;
	Deviation = 0.75*Deviation + 0.25*(Arbitrage - ArbMean);

	static int Position = 0;	
	if(Position == 0) {
		if(Deviation > SpreadSPY+THRESHOLD)
			return Position = 1;
		if(-Deviation > SpreadES+THRESHOLD)
			return Position = 2;
	} else {
		if(Position == 1 && -Deviation > SpreadES+THRESHOLD/2)
			return Position = 0;
		if(Position == 2 && Deviation > SpreadSPY+THRESHOLD/2)
			return Position = 0;
	}
	return -1;	
}

The tradeHFT function is called from some framework – not shown here – that gets the price quotes and sends the trade orders. Parameters are the current best ask and bid prices of ES and SPY from the top of the order book (we assume here that the SPY price is multiplied with 10 so that both assets are on the same scale). The function returns a code that tells the framework whether to open or close opposite positions or to do nothing. The Arbitrage variable is the mid price difference of SPY and ES. Its mean (ArbMean) is filtered by a slow EMA, and the Deviation from the mean is also slightly filtered by a fast EMA to prevent reactions on outlier quotes. The Position variable constitutes a tiny state machine with the 3 states long, short, and nothing. The entry/exit Threshold is here set to 40 cents, equivalent to a bad SPY spread multiplied with 10. This is the only adjustable parameter of the system. If we wanted to really trade it, we had to optimize the threshold using several months’ ES and SPY data.

This minimalistic system would be quite easy to convert to Pentium assembler or even to a FPGA chip. But it is not really necessary: Even compiled with Zorro’s lite-C compiler, the tradeHFT function executes in just about 750 nanoseconds. A strongly optimizing C compiler, like Microsoft VC++,  gets the execution time down to 650 nanoseconds. Since the time span between two ES quotes is normally 10 microseconds or more, the speed of C is largely sufficient.

Our HFT experiment has to answer two questions. First, are those price differences big enough for arbitrage profit? And second, at which maximum latency will the system still work? 

The data

For backtesting a HFT system, normal price data that you can freely download from brokers won’t do. You have to buy high resolution order book or BBO data (Best Bid and Offer) that includes the exchange time stamps. Without knowing the exact time when the price quote was received at the exchange, it is not possible to determine the maximum latency.

Some companies are recording all quotes from all exchanges and are selling this data. Any one has its specific data format, so the first challenge is to convert this to lean data files that we then evaluate with our simulation software. We’re using this very simple target format for price quotes:

typedef struct T1    // single tick
{
	double time; // time stamp, OLE DATE format
	float fVal;  // positive = ask price, negative = bid price	
} T1;

The company watching the Chicago Mercantile Exchange delivers its data in a specific CSV format with many additional fields, of which most we don’t need here (f.i. the trade volume or the quote arrival time). Every day’s quotes are stored in one CSV file. This is the Zorro script for pulling the ES December 2016 contract out of it and converting it to a dataset of T1 price quotes:

//////////////////////////////////////////////////////
// Convert price history from Nanotick BBO to .t1
//////////////////////////////////////////////////////

#define STARTDAY 20161004
#define ENDDAY   20161014

string InName = "History\\CME.%08d-%08d.E.BBO-C.310.ES.csv";  // name of a day file
string OutName = "History\\ES_201610.t1";
string Code = "ESZ";	// December contract symbol

string Format = "2,,%Y%m%d,%H:%M:%S,,,s,,,s,i,,";  // Nanotick csv format
void main()
{
	int N,Row,Record,Records;
	for(N = STARTDAY; N <= ENDDAY; N++)
	{
		string FileName = strf(InName,N,N+1);
		if(!file_date(FileName)) continue;
		Records = dataParse(1,Format,FileName);  // read BBO data
		printf("\n%d rows read",Records);
		dataNew(2,Records,2);  // create T1 dataset
		for(Record = 0,Row = 0; Record < Records; Record++)
		{
			if(!strstr(Code,dataStr(1,Record,1))) continue; // select only records with correct symbol
			T1* t1 = dataStr(2,Row,0);  // store record in T1 format
			float Price = 0.01 * dataInt(1,Record,3);  // price in cents
			if(Price < 1000) continue;  // no valid price
			string AskBid = dataStr(1,Record,2);
			if(AskBid[0] == 'B')  // negative price for Bid
				Price = -Price;
			t1->fVal = Price;
			t1->time = dataVar(1,Record,0) + 1./24.;  // add 1 hour Chicago-NY time difference
			Row++;
		}
		printf(", %d stored",Row);
		dataAppend(3,2,0,Row);  // append dataset
		if(!wait(0)) return;
	}
	dataSave(3,OutName);  // store complete dataset
}

We’re converting here 10 days in October 2016 for our backtest. The script first parses the CSV file into an intermediary binary dataset, which is then converted to the T1 target format. Since the time stamps are in Chicago local time, we have to add one hour to convert them to NY time.

The company watching the New York Stock exchange delivers data in a highly compressed specific format named “NxCore Tape”. We’re using the Zorro NxCore plugin for converting this to another T1 list:

//////////////////////////////////////////////////////
// Convert price history from Nanex .nx2 to .t1
//////////////////////////////////////////////////////

#define STARTDAY 20161004
#define ENDDAY 	 20161014
#define BUFFER	 10000

string InName = "History\\%8d.GS.nx2";  // name of a single day tape
string OutName = "History\\SPY_201610.t1";
string Code = "eSPY";

int Row,Rows;

typedef struct QUOTE {
	char	Name[24];
	var	Time,Price,Size;
} QUOTE;

int callback(QUOTE *Quote)
{
	if(!strstr(Quote->Name,Code)) return 1;
	T1* t1 = dataStr(1,Row,0);  // store record in T1 format
	t1->time = Quote->Time;
	t1->fVal = Quote->Price;
	Row++; Rows++;
	if(Row >= BUFFER)	{   // dataset full?
		Row = 0;
		dataAppend(2,1);    // append to dataset 2
	}
	return 1;
}


void main()
{
	dataNew(1,BUFFER,2); // create a small dataset
	login(1);            // open the NxCore plugin
	
	int N;
	for(N = STARTDAY; N <= ENDDAY; N++) {
		string FileName = strf(InName,N);
		if(!file_date(FileName)) continue;
		printf("\n%s..",FileName);
		Row = Rows = 0;  // initialize global variables
		brokerCommand(SET_HISTORY,FileName); // parse the tape
		dataAppend(2,1,0,Row);  // append the rest to dataset 2
		printf("\n%d rows stored",Rows);
		if(!wait(0)) return;  // abort when [Stop] was hit
	}
	dataSave(2,OutName); // store complete dataset
}

The callback function is called by any quote in the tape file. We don’t need most of the data, so we filter out only the SPY quotes (“eSPY”).

Verifying the inefficiency

With data from both sources, we can now compare the ES and SPY prices in high resolution. Here’s a typical 10 seconds sample from the price curves:

The resolution is 1 ms. ES is plotted in $ units, SPY in 10 cents units, with a small offset so that the curves lie over each other. The prices shown are ask prices. We can see that ES moves in steps of 25 cents, SPY in steps of 1 cent. The prices are still well correlated even on a millisecond scale. ES appears to trail a tiny bit.

We can also see an arbitrage opportunity at the steep price step in the center at about 10:01:30. ES reacted a bit slower, but stronger on some event, probably a moderate price jump of one of the stocks of the S&P 500. This event also triggered a fast sequence of oscillating SPY quotes, most likely from other HFT systems, until the situation calmed down again a few 100 ms later (the scale is not linear since time periods with no quotes are skipped). For several milliseconds the ES-SPY difference exceeded the ask-bid spread of both assets (usually 25 cents for ES and 1..4 cents for SPY). We would here ideally sell ES and buy SPY immediately after the ES price step. So we have verified that the theoretized inefficiency does really exist, at least in this sample. 

The script for displaying high resolution charts:

#define ES_HISTORY	"ES_201610.t1"
#define SPY_HISTORY	"SPY_201610.t1"
#define TIMEFORMAT	"%Y%m%d %H:%M:%S"
#define FACTOR		10
#define OFFSET		3.575

void main()
{
	var StartTime = wdatef(TIMEFORMAT,"20161005 10:01:25"),
		EndTime = wdatef(TIMEFORMAT,"20161005 10:01:35");
	MaxBars = 10000;
	BarPeriod = 0.001/60.;	// 1 ms plot resolution
	Outlier = 1.002;  // filter out 0.2% outliers

	assetList("HFT.csv");
	dataLoad(1,ES_HISTORY,2);
	dataLoad(2,SPY_HISTORY,2);
	int RowES=0, RowSPY=0;
	
	while(Bar < MaxBars)
	{
		var TimeES = dataVar(1,RowES,0), 
			PriceES = dataVar(1,RowES,1), 
			TimeSPY = dataVar(2,RowSPY,0), 
			PriceSPY = dataVar(2,RowSPY,1);

		if(TimeES < TimeSPY) RowES++;
		else RowSPY++;

		if(min(TimeES,TimeSPY) < StartTime) continue;
		if(max(TimeES,TimeSPY) > EndTime) break;

		if(TimeES < TimeSPY) {
			asset("ES");
			priceQuote(TimeES,PriceES);
		} else {
			asset("SPY");
			priceQuote(TimeSPY,PriceSPY);
		}
		
		asset("ES");
		if(AssetBar > 0) plot("ES",AskPrice+OFFSET,LINE,RED);
		asset("SPY");
		if(AssetBar > 0) plot("SPY",FACTOR*AskPrice,LINE,BLACK);
	}
}

The script first reads the two historical data files that we’ve created above, and then parses them row by row. For keeping the ES and SPY quotes in sync, we always read the quote with the smaller timestamp from the datasets (the quotes are stored in ascending timestamp order). The priceQuote function checks the prices for outliers, stores the ask price in the AskPrice variable and the ask-bid difference in Spread, and increases the Bar count for plotting the price curves. A bar of the curve is equivalent to 1 ms. The AssetBar variable is the last bar with a price quote of that asset, and is used here to prevent plotting before the first quote arrived. 

Testing the system

For backtesting our HFT system, we only need to modify the script above a bit, and call the tradeHFT function in the loop:

#define LATENCY		4.0	// milliseconds

function main()
{
	var StartTime = wdatef(TIMEFORMAT,"20161005 09:30:00"),
		EndTime = wdatef(TIMEFORMAT,"20161005 15:30:00");
	MaxBars = 200000;
	BarPeriod = 0.1/60.;	// 100 ms bars 
	Outlier = 1.002;

	assetList("HFT.csv");
	dataLoad(1,ES_HISTORY,2);
	dataLoad(2,SPY_HISTORY,2);
	int RowES=0, RowSPY=0;
	
	EntryDelay = LATENCY/1000.;
	Hedge = 2;
	Fill = 8; // HFT fill mode;
	Slippage = 0;
	Lots = 100;
		
	while(Bar < MaxBars)
	{
		var TimeES = dataVar(1,RowES,0), 
			PriceES = dataVar(1,RowES,1),
			TimeSPY = dataVar(2,RowSPY,0),
			PriceSPY = dataVar(2,RowSPY,1);

		if(TimeES < TimeSPY) RowES++;
		else RowSPY++;

		if(min(TimeES,TimeSPY) < StartTime) continue;
		if(max(TimeES,TimeSPY) > EndTime) break;

		if(TimeES < TimeSPY) {
			asset("ES");
			priceQuote(TimeES,PriceES);
		} else {
			asset("SPY");
			priceQuote(TimeSPY,FACTOR*PriceSPY);
		}
		
		asset("ES");
		if(!AssetBar) continue;
		var AskES = AskPrice, BidES = AskPrice-Spread;
		asset("SPY");
		if(!AssetBar) continue;
		var AskSPY = AskPrice, BidSPY = AskPrice-Spread;

		int Order = tradeHFT(AskSPY,BidSPY,AskES,BidES);	
		switch(Order) {
			case 1: 
			asset("ES"); enterLong();
			asset("SPY"); enterShort();
			break;
		
			case 2: 
			asset("ES"); enterShort();
			asset("SPY"); enterLong();
			break;
		
			case 0:
			asset("ES"); exitLong(); exitShort();
			asset("SPY"); exitLong(); exitShort();
			break;
		}
	}
	printf("\nProfit %.2f at NY Time %s",
		Equity,strdate(TIMEFORMAT,dataVar(1,RowES,0)));
}

This script runs a HFT backtest of one trading day, from 9:30 until 15:30 New York time. We can see that it just calls the HFT function with the ES and SPY prices, then executes the returned code in the switch-case statement. It opens 100 units of each asset (equivalent to 2 ES and 1000 SPY contracts). The round-trip latency is set up with the EntryDelay variable. In HFT mode (Fill = 8), a trade is filled at the most recent price quote after the given delay. This ensures a realistic latency simulation when price quotes are entered with their exchange time stamps.

Here’s the HFT profit at the end of the day with different round-trip latency values:

The ES-SPY HFT arbitrage system makes about $800 each day with an unrealistic small latency of 500 microseconds. Unfortunately, due to the 700 miles between the NYSE and the CME, you’d need a time machine for that (or some faster-than-light method of quantum teleportation). A HFT server in Warren, Ohio, at 4 ms latency would generate about $300 per day. A server slightly off the direct NY-Chicago line can still grind out $200 daily. But the system deteriorates quickly when located further away, as with a server in Nashville, Tennessee, at around 10 ms latency. This is a strong hint that some high-speed systems in the proximity of both exchanges are already busy with exploiting ES-SPY arbitrage.

Still, $300 per day result in a decent $75,000 annual income. However, what needed you to invest for that, aside from hardware and software? With SPY at $250, the 100 units translate to 100*$2500 + 100*10*$250 = half a million dollars trade volume. So you would get only 15% annual return on your investment. But you can improve this by adding more pairs of NY ETFs and their equivalent CME futures to the arbitrage strategy. And you can improve it further when you find a broker or similar service that can receive your orders directly at the exchanges. Due to the ETF/future hedging the position is almost without risk. So you could probably negotiate a large leverage. And also a flat monthly fee, since broker commissions were not included in the test.

Conclusion 

• When systems react fast enough, profits can be achieved with very primitive methods, such as arbitrage between highly correlated assets.
• Location has a large impact on HFT profits.
• ES-SPY arbitrage cannot be traded by everyone from everywhere. You’re competing with people that are doing this already. Possibly in Warren, Ohio.

I’ve added the scripts and the asset list to the 2017 script archive in the “HFT” and “History” folders. Unfortunately I could not add the ES / SPY price history files, since I do not own the data. For reproducing the results, get BBO history from Nanex or Nanotick – their data can be read with the scripts above. You’ll also need Zorro S version 1.60 or above, which supports HFT fill mode.

References

(1) When Correlations Break Down (Jared Bernstein 2014)

(2) Financial Programming Greatly Accelerated (Cousin/Weston, Automated Trader 42/2017)

Addendum. I have been asked how the profit would be affected when the server is located in New York, with 0.25 ms signal delay to the NYSE and 4.8 ms signal delay to the CME. For simulating this, modify the script:

var TimeES = dataVar(1,RowES,0)+4.8/(1000*60*60*24),
  TimeSPY = dataVar(2,RowSPY,0)+0.25/(1000*60*60*24),
...
switch(Order) {
	case 1: 
		EntryDelay = 4.8/1000;
		asset("ES"); enterLong();
		EntryDelay = 0.25/1000;
		asset("SPY"); enterShort();
		break;
		
	case 2: 
		EntryDelay = 4.8/1000;
		asset("ES"); enterShort();
		EntryDelay = 0.25/1000;
		asset("SPY"); enterLong();
		break;
		
	case 0:
		EntryDelay = 4.8/1000;
		asset("ES"); exitLong(); exitShort();
		EntryDelay = 0.25/1000;
		asset("SPY"); exitLong(); exitShort();
		break;
}

The first line simulates the price quote arrivals with 4.8 and 0.25 ms delay, the other lines establish the different order delays for SPY and ES. Alternatively, you could artificially delay the incoming SPY quotes by another 4.55 ms, so that their time stamps again match the ES quotes. In both cases the system returns about $240 per day, almost as much as in Warren. However a similar system located in Aurora, close to Chicago (swap the delays in the script), would produce zero profit. The asymmetry is caused by the relatively long constant periods of ES, making the SPY latency more relevant for the money at the end of the day.