Our example will be a research project – a machine learning experiment for answering two questions. Does a more complex algorithm – such as, more neurons and deeper learning – produce a better prediction? And are short-term price moves predictable by short-term price history? The last question came up due to my scepticism about price action trading in the previous part of this series. I got several emails asking about the “trading system generators” or similar price action tools that are praised on some websites. There is no hard evidence that such tools ever produced any profit (except for their vendors) – but does this mean that they all are garbage? We’ll see.

Our experiment is simple: We collect information from the last candles of a price curve, feed it in a deep learning neural net, and use it to predict the next candles. My hypothesis is that a few candles don’t contain any useful predictive information. Of course, a nonpredictive outcome of the experiment won’t mean that I’m right, since I could have used wrong parameters or prepared the data badly. But a predictive outcome would be a hint that I’m wrong and price action trading can indeed be profitable.

Machine learning strategy development
Step 1: The target variable

To recap the previous part: a supervised learning algorithm is trained with a set of features in order to predict a target variable. So the first thing to determine is what this target variable shall be. A popular target, used in most papers, is the sign of the price return at the next bar. Better suited for prediction, since less susceptible to randomness, is the price difference to a more distant prediction horizon, like 3 bars from now, or same day next week. Like almost anything in trading systems, the prediction horizon is a compromise between the effects of randomness (less bars are worse) and predictability (less bars are better).

Sometimes you’re not interested in directly predicting price, but in predicting some other parameter – such as the current leg of a Zigzag indicator – that could otherwise only be determined in hindsight. Or you want to know if a certain market inefficiency will be present in the next time, especially when you’re using machine learning not directly for trading, but for filtering trades in a model-based system. Or you want to predict something entirely different, for instance the probability of a market crash tomorrow. All this is often easier to predict than the popular tomorrow’s return.

In our price action experiment we’ll use the return of a short-term price action trade as target variable. Once the target is determined, next step is selecting the features.

Step 2: The features

A price curve is the worst case for any machine learning algorithm. Not only does it carry little signal and mostly noise, it is also nonstationary and the signal/noise ratio changes all the time. The exact ratio of signal and noise depends on what is meant with “signal”, but it is normally too low for any known machine learning algorithm to produce anything useful. So we must derive features from the price curve that contain more signal and less noise. Signal, in that context, is any information that can be used to predict the target, whatever it is. All the rest is noise.

Thus, selecting the features is critical for success – even more critical than deciding which machine learning algorithm you’re going to use. There are two approaches for selecting features. The first and most common is extracting as much information from the price curve as possible. Since you do not know where the information is hidden, you just generate a wild collection of indicators with a wide range of parameters, and hope that at least a few of them will contain the information that the algorithm needs. This is the approach that you normally find in the literature. The problem of this method: Any machine learning algorithm is easily confused by nonpredictive predictors. So it won’t do to just throw 150 indicators at it. You need some preselection algorithm that determines which of them carry useful information and which can be omitted. Without reducing the features this way to maybe eight or ten, even the deepest learning algorithm won’t produce anything useful.

The other approach, normally for experiments and research, is using only limited information from the price curve. This is the case here: Since we want to examine price action trading, we only use the last few prices as inputs, and must discard all the rest of the curve. This has the advantage that we don’t need any preselection algorithm since the number of features is limited anyway. Here are the two simple predictor functions that we use in our experiment (in C):

var change(int n)
{
	return scale((priceClose(0) - priceClose(n))/priceClose(0),100)/100;
}

var range(int n)
{
	return scale((HH(n) - LL(n))/priceClose(0),100)/100;
}

The two functions are supposed to carry the necessary information for price action: per-bar movement and volatility. The change function is the difference of the current price to the price of n bars before, divided by the current price. The range function is the total high-low distance of the last n candles, also in divided by the current price. And the scale function centers and compresses the values to the +/-100 range, so we divide them by 100 for getting them normalized to +/-1. We remember that normalizing is needed for machine learning algorithms.

Step 3: Preselecting predictors

When you have selected a large number of indicators or other signals as features for your algorithm, you must determine which of them is useful and which not. There are many methods for reducing the number of features, for instance:

• Determine the correlations between the signals. Remove those with a strong correlation to other signals, since they do not contribute to the information.
• Compare the information content of signals directly, with algorithms like information entropy or decision trees.
• Determine the information content indirectly by comparing the signals with randomized signals; there are some software libraries for this, such as the R Boruta package.
• Use an algorithm like Principal Components Analysis (PCA) for generating a new signal set with reduced dimensionality.
• Use genetic optimization for determining the most important signals just by the most profitable results from the prediction process. Great for curve fitting if you want to publish impressive results in a research paper.

Reducing the number of features is important for most machine learning algorithms, including shallow neural nets. For deep learning it’s less important, since deep nets  with many neurons are normally able to process huge feature sets and discard redundant features. For our experiment we do not preselect or preprocess the features, but you can find useful information about this in articles (1), (2), and (3) listed at the end of the page.

Step 4: Select the machine learning algorithm

R offers many different ML packages, and any of them offers many different algorithms with many different parameters. Even if you already decided about the method – here, deep learning – you have still the choice among different approaches and different R packages. Most are quite new, and you can find not many empirical information that helps your decision. You have to try them all and gain experience with different methods. For our experiment we’ve choosen the Deepnet package, which is probably the simplest and easiest to use deep learning library. This keeps our code short. We’re using its Stacked Autoencoder (SAE) algorithm for pre-training the network. Deepnet also offers a Restricted Boltzmann Machine (RBM) for pre-training, but I could not get good results from it. There are other and more complex deep learning packages for R, so you can spend a lot of time checking out all of them.

How pre-training works is easily explained, but why it works is a different matter. As to my knowledge, no one has yet come up with a solid mathematical proof that it works at all. Anyway, imagine a large neural net with many hidden layers:

Training the net means setting up the connection weights between the neurons. The usual method is error backpropagation. But it turns out that the more hidden layers you have, the worse it works. The backpropagated error terms get smaller and smaller from layer to layer, causing the first layers of the net to learn almost nothing. Which means that the predicted result becomes more and more dependent of the random initial state of the weights. This severely limited the complexity of layer-based neural nets and therefore the tasks that they can solve. At least until 10 years ago.

In 2006 scientists in Toronto first published the idea to pre-train the weights with an unsupervised learning algorithm, a restricted Boltzmann machine. This turned out a revolutionary concept. It boosted the development of artificial intelligence and allowed all sorts of new applications from Go-playing machines to self-driving cars. Meanwhile, several new improvements and algorithms for deep learning have been found. A stacked autoencoder works this way:

1. Select the hidden layer to train; begin with the first hidden layer. Connect its outputs to a temporary output layer that has the same structure as the network’s input layer.
2. Feed the network with the training samples, but without the targets. Train it so that the first hidden layer reproduces the input signal – the features – at its outputs as exactly as possible. The rest of the network is ignored. During training, apply a ‘weight penalty term’ so that as few connection weights as possible are used for reproducing the signal.
3. Now feed the outputs of the trained hidden layer to the inputs of the next untrained hidden layer, and repeat the training process so that the input signal is now reproduced at the outputs of the next layer.
4. Repeat this process until all hidden layers are trained. We have now a ‘sparse network’ with very few layer connections that can reproduce the input signals.
5. Now train the network with backpropagation for learning the target variable, using the pre-trained weights of the hidden layers as a starting point.

The hope is that the unsupervised pre-training process produces an internal noise-reduced abstraction of the input signals that can then be used for easier learning the target. And this indeed appears to work. No one really knows why, but several theories – see paper (4) below – try to explain that phenomenon.

Step 5: Generate a test data set

We first need to produce a data set with features and targets so that we can test our prediction process and try out parameters. The features must be based on the same price data as in live trading, and for the target we must simulate a short-term trade. So it makes sense to generate the data not with R, but with our trading platform, which is anyway a lot faster. Here’s a small Zorro script for this, DeepSignals.c:

function run()
{
	StartDate = 20140601; // start two years ago
	BarPeriod = 60; // use 1-hour bars
	LookBack = 100; // needed for scale()

	set(RULES);   // generate signals
	LifeTime = 3; // prediction horizon
	Spread = RollLong = RollShort = Commission = Slippage = 0;
	
	adviseLong(SIGNALS+BALANCED,0,
		change(1),change(2),change(3),change(4),
		range(1),range(2),range(3),range(4));
	enterLong(); 
}

We’re generating 2 years of data with features calculated by our above defined change and range functions. Our target is the result of a trade with 3 bars life time. Trading costs are set to zero, so in this case the result is equivalent to the sign of the price difference at 3 bars in the future. The adviseLong function is described in the Zorro manual; it is a mighty function that automatically handles training and predicting and allows to use any R-based machine learning algorithm just as if it were a simple indicator.

In our code, the function uses the next trade return as target, and the price changes and ranges of the last 4 bars as features. The SIGNALS flag tells it not to train the data, but to export it to a .csv file. The BALANCED flag makes sure that we get as many positive as negative returns; this is important for most machine learning algorithms. Run the script in [Train] mode with our usual test asset EUR/USD selected. It generates a spreadsheet file named DeepSignalsEURUSD_L.csv that contains the features in the first 8 columns, and the trade return in the last column.

Step 6: Calibrate the algorithm

Complex machine learning algorithms have many parameters to adjust. Some of them offer great opportunities to curve-fit the algorithm for publications. Still, we must calibrate parameters since the algorithm rarely works well with its default settings. For this, here’s an R script that reads the previously created data set and processes it with the deep learning algorithm (DeepSignal.r): 

library('deepnet', quietly = T) 
library('caret', quietly = T)

neural.train = function(model,XY) 
{
  XY <- as.matrix(XY)
  X <- XY[,-ncol(XY)]
  Y <- XY[,ncol(XY)]
  Y <- ifelse(Y > 0,1,0)
  Models[[model]] <<- sae.dnn.train(X,Y,
      hidden = c(50,100,50), 
      activationfun = "tanh", 
      learningrate = 0.5, 
      momentum = 0.5, 
      learningrate_scale = 1.0, 
      output = "sigm", 
      sae_output = "linear", 
      numepochs = 100, 
      batchsize = 100,
      hidden_dropout = 0, 
      visible_dropout = 0)
}

neural.predict = function(model,X) 
{
  if(is.vector(X)) X <- t(X)
  return(nn.predict(Models[[model]],X))
}

neural.init = function()
{
  set.seed(365)
  Models <<- vector("list")
}

TestOOS = function() 
{
  neural.init()
  XY <<- read.csv('C:/Zorro/Data/DeepSignalsEURUSD_L.csv',header = F)
  splits <- nrow(XY)*0.8
  XY.tr <<- head(XY,splits);
  XY.ts <<- tail(XY,-splits)
  neural.train(1,XY.tr)
  X <<- XY.ts[,-ncol(XY.ts)]
  Y <<- XY.ts[,ncol(XY.ts)]
  Y.ob <<- ifelse(Y > 0,1,0)
  Y <<- neural.predict(1,X)
  Y.pr <<- ifelse(Y > 0.5,1,0)
  confusionMatrix(Y.pr,Y.ob)
}

We’ve defined three functions neural.train, neural.predict, and neural.init for training, predicting, and initializing the neural net. The function names are not arbitrary, but follow the convention used by Zorro’s advise(NEURAL,..) function. It doesn’t matter now, but will matter later when we use the same R script for training and trading the deep learning strategy. A fourth function, TestOOS, is used for out-of-sample testing our setup.

The function neural.init seeds the R random generator with a fixed value (365 is my personal lucky number). Otherwise we would get a slightly different result any time, since the neural net is initialized with random weights. It also creates a global R list named “Models”. Most R variable types don’t need to be created beforehand, some do (don’t ask me why). The ‘<<-‘ operator is for accessing a global variable from within a function.

The function neural.train takes as input a model number and the data set to be trained. The model number identifies the trained model in the “Models” list. A list is not really needed for this test, but we’ll need it for more complex strategies that train more than one model. The matrix containing the features and target is passed to the function as second parameter. If the XY data is not a proper matrix, which frequently happens in R depending on how you generated it, it is converted to one. Then it is split into the features (X) and the target (Y), and finally the target is converted to 1 for a positive trade outcome and 0 for a negative outcome. 

The network parameters are then set up. Some are obvious, others are free to play around with:

• The network structure is given by the hidden vector:  c(50,100,50) defines 3 hidden layers, the first with 50, second with 100, and third with 50 neurons. That’s the parameter that we’ll later modify for determining whether deeper is better.
• The activation function converts the sum of neuron input values to the neuron output; most often used are sigmoid that saturates to 0 or 1, or tanh that saturates to -1 or +1.

We use tanh here since our signals are also in the +/-1 range. The output of the network is a sigmoid function since we want a prediction in the 0..1 range. But the SAE output must be “linear” so that the Stacked Autoencoder can reproduce the analog input signals on the outputs. Recently in fashion came RLUs, Rectified Linear Units, as activation functions for internal layers. RLUs are faster and partially overcome the above mentioned backpropagation problem, but are not supported by deepnet.

• The learning rate controls the step size for the gradient descent in training; a lower rate means finer steps and possibly more precise prediction, but longer training time.
• Momentum adds a fraction of the previous step to the current one. It prevents the gradient descent from getting stuck at a tiny local minimum or saddle point.
• The learning rate scale is a multiplication factor for changing the learning rate after each iteration (I am not sure for what this is good, but there may be tasks where a lower learning rate on higher epochs improves the training).
• An epoch is a training iteration over the entire data set. Training will stop once the number of epochs is reached. More epochs mean better prediction, but longer training.
• The batch size is a number of random samples – a mini batch – taken out of the data set for a single training run. Splitting the data into mini batches speeds up training since the weight gradient is then calculated from fewer samples. The higher the batch size, the better is the training, but the more time it will take.
• The dropout is a number of randomly selected neurons that are disabled during a mini batch. This way the net learns only with a part of its neurons. This seems a strange idea, but can effectively reduce overfitting.

All these parameters are common for neural networks. Play around with them and check their effect on the result and the training time. Properly calibrating a neural net is not trivial and might be the topic of another article. The parameters are stored in the model together with the matrix of trained connection weights. So they need not to be given again in the prediction function, neural.predict. It takes the model and a vector X of features, runs it through the layers, and returns the network output, the predicted target Y. Compared with training, prediction is pretty fast since it only needs a couple thousand multiplications. If X was a row vector, it is transposed and this way converted to a column vector, otherwise the nn.predict function won’t accept it.

Use RStudio or some similar environment for conveniently working with R. Edit the path to the .csv data in the file above, source it, install the required R packages (deepnet, e1071, and caret), then call the TestOOS function from the command line. If everything works, it should print something like that:

> TestOOS()
begin to train sae ......
training layer 1 autoencoder ...
####loss on step 10000 is : 0.000079
training layer 2 autoencoder ...
####loss on step 10000 is : 0.000085
training layer 3 autoencoder ...
####loss on step 10000 is : 0.000113
sae has been trained.
begin to train deep nn ......
####loss on step 10000 is : 0.123806
deep nn has been trained.
Confusion Matrix and Statistics

          Reference
Prediction    0    1
         0 1231  808
         1  512  934
                                          
               Accuracy : 0.6212          
                 95% CI : (0.6049, 0.6374)
    No Information Rate : 0.5001          
    P-Value [Acc > NIR] : < 2.2e-16       
                                          
                  Kappa : 0.2424          
 Mcnemar's Test P-Value : 4.677e-16       
                                          
            Sensitivity : 0.7063          
            Specificity : 0.5362          
         Pos Pred Value : 0.6037          
         Neg Pred Value : 0.6459          
             Prevalence : 0.5001          
         Detection Rate : 0.3532          
   Detection Prevalence : 0.5851          
      Balanced Accuracy : 0.6212          
                                          
       'Positive' Class : 0               
                                          
> 

TestOOS reads first our data set from Zorro’s Data folder. It splits the data in 80% for training (XY.tr) and 20% for out-of-sample testing (XY.ts). The training set is trained and the result stored in the Models list at index 1. The test set is further split in features (X) and targets (Y). Y is converted to binary 0 or 1 and stored in Y.ob, our vector of observed targets. We then predict the targets from the test set, convert them again to binary 0 or 1 and store them in Y.pr. For comparing the observation with the prediction, we use the confusionMatrix function from the caret package.

A confusion matrix of a binary classifier is simply a 2×2 matrix that tells how many 0’s and how many 1’s had been predicted wrongly and correctly. A lot of metrics are derived from the matrix and printed in the lines above. The most important at the moment is the 62% prediction accuracy. This may hint that I bashed price action trading a little prematurely. But of course the 62% might have been just luck. We’ll see that later when we run a WFO test.

A final advice: R packages are occasionally updated, with the possible consequence that previous R code suddenly might work differently, or not at all. This really happens, so test carefully after any update.

Step 7: The strategy

Now that we’ve tested our algorithm and got some prediction accuracy above 50% with a test data set, we can finally code our machine learning strategy. In fact we’ve already coded most of it, we just must add a few lines to the above Zorro script that exported the data set. This is the final script for training, testing, and (theoretically) trading the system (DeepLearn.c):

#include <r.h>

function run()
{
	StartDate = 20140601;
	BarPeriod = 60;	// 1 hour
	LookBack = 100;

	WFOPeriod = 252*24; // 1 year
	DataSplit = 90;
	NumCores = -1;  // use all CPU cores but one

	set(RULES);
	Spread = RollLong = RollShort = Commission = Slippage = 0;
	LifeTime = 3;
	if(Train) Hedge = 2;
	
	if(adviseLong(NEURAL+BALANCED,0,
		change(1),change(2),change(3),change(4),
		range(1),range(2),range(3),range(4)) > 0.5) 
		enterLong();
	if(adviseShort() > 0.5) 
		enterShort();
}

We’re using a WFO cycle of one year, split in a 90% training and a 10% out-of-sample test period. You might ask why I have earlier used two year’s data and a different split, 80/20, for calibrating the network in step 5. This is for using differently composed data for calibrating and for walk forward testing. If we used exactly the same data, the calibration might overfit it and compromise the test. 

The selected WFO parameters mean that the system is trained with about 225 days data, followed by a 25 days test or trade period. Thus, in live trading the system would retrain every 25 days, using the prices from the previous 225 days. In the literature you’ll sometimes find the recommendation to retrain a machine learning system after any trade, or at least any day. But this does not make much sense to me. When you used almost 1 year’s data for training a system, it can obviously not deteriorate after a single day. Or if it did, and only produced positive test results with daily retraining, I would strongly suspect that the results are artifacts by some coding mistake.

Training a deep network takes really a long time, in our case about 10 minutes for a network with 3 hidden layers and 200 neurons. In live trading this would be done by a second Zorro process that is automatically started by the trading Zorro. In the backtest, the system trains at any WFO cycle. Therefore using multiple cores is recommended for training many cycles in parallel. The NumCores variable at -1 activates all CPU cores but one. Multiple cores are only available in Zorro S, so a complete walk forward test with all WFO cycles can take several hours with the free version.

In the script we now train both long and short trades. For this we have to allow hedging in Training mode, since long and short positions are open at the same time. Entering a position is now dependent on the return value from the advise function, which in turn calls either the neural.train or the neural.predict function from the R script. So we’re here entering positions when the neural net predicts a result above 0.5. 

The R script is now controlled by the Zorro script (for this it must have the same name, DeepLearn.r, only with different extension). It is identical to our R script above since we’re using the same network parameters. Only one additional function is needed for supporting a WFO test:

neural.save = function(name)
{
  save(Models,file=name)  
}

The neural.save function stores the Models list – it now contains 2 models for long and for short trades – after every training run in Zorro’s Data folder. Since the models are stored for later use, we do not need to train them again for repeated test runs.

This is the WFO equity curve generated with the script above (EUR/USD, without trading costs):

Although not all WFO cycles get a positive result, it seems that there is some predictive effect. The curve is equivalent to an annual return of 89%, achieved with a 50-100-50 hidden layer structure. We’ll check in the next step how different network structures affect the result.

Since the neural.init, neural.train, neural.predict, and neural.save functions are automatically called by Zorro’s adviseLong/adviseShort functions, there are no R functions directly called in the Zorro script. Thus the script can remain unchanged when using a different machine learning method. Only the DeepLearn.r script must be modified and the neural net, for instance, replaced by a support vector machine. For trading such a machine learning system live on a VPS, make sure that R is also installed on the VPS, the needed R packages are installed, and the path to the R terminal set up in Zorro’s ini file. Otherwise you’ll get an error message when starting the strategy.

Step 8: The experiment

If our goal had been developing a strategy, the next steps would be the reality check, risk and money management, and preparing for live trading just as described under model-based strategy development. But for our experiment we’ll now run a series of tests, with the number of neurons per layer increased from 10 to 100 in 3 steps, and 1, 2, or 3 hidden layers (deepnet does not support more than 3). So we’re looking into the following 9 network structures: c(10), c(10,10), c(10,10,10), c(30), c(30,30), c(30,30,30), c(100), c(100,100), c(100,100,100). For this experiment you need an afternoon even with a fast PC and in multiple core mode. Here are the results (SR = Sharpe ratio, R2 = slope linearity): 

	* 10 neurons	* 30 neurons	* 100 neurons
1
2
3

We see that a simple net with only 10 neurons in a single hidden layer won’t work well for short-term prediction. Network complexity clearly improves the performance, however only up to a certain point. A good result for our system is already achieved with 3 layers x 30 neurons. Even more neurons won’t help much and sometimes even produce a worse result. This is no real surprise, since for processing only 8 inputs, 300 neurons can likely not do a better job than 100.  

Conclusion

Our goal was determining if a few candles can have predictive power and how the results are affected by the complexity of the algorithm. The results seem to suggest that short-term price movements can indeed be predicted sometimes by analyzing the changes and ranges of the last 4 candles. The prediction is not very accurate – it’s in the 58%..60% range, and most systems of the test series become unprofitable when trading costs are included. Still, I have to reconsider my opinion about price action trading. The fact that the prediction improves with network complexity is an especially convincing argument for short-term price predictability.

It would be interesting to look into the long-term stability of predictive price patterns. For this we had to run another series of experiments and modify the training period (WFOPeriod in the script above) and the 90% IS/OOS split. This takes longer time since we must use more historical data. I have done a few tests and found so far that a year seems to be indeed a good training period. The system deteriorates with periods longer than a few years. Predictive price patterns, at least of EUR/USD, have a limited lifetime.

Where can we go from here? There’s a plethora of possibilities, for instance:

• Use inputs from more candles and process them with far bigger networks with thousands of neurons.
• Use oversampling for expanding the training data. Prediction always improves with more training samples.
• Compress time series f.i. with spectal analysis and analyze not the candles, but their frequency representation with machine learning methods.
• Use inputs from many candles – such as, 100 – and pre-process adjacent candles with one-dimensional convolutional network layers.
• Use recurrent networks. Especially LSTM could be very interesting for analyzing time series – and as to my knowledge, they have been rarely used for financial prediction so far.
• Use an ensemble of neural networks for prediction, such as Aronson’s “oracles” and “comitees”.

Papers / Articles

(1) A.S.Sisodiya, Reducing Dimensionality of Data 
(2) K.Longmore, Machine Learning for Financial Prediction 
(3) V.Perervenko, Selection of Variables for Machine Learning
(4) D.Erhan et al, Why Does Pre-training Help Deep Learning?

I’ve added the C and R scripts to the 2016 script repository. You need both in Zorro’s Strategy folder. Zorro version 1.474, and R version 3.2.5 (64 bit) was used for the experiment, but it should also work with other versions. 

Binary options, in their most common form, are very different to real options. They are a bet that the price of an asset will rise or fall within a given time frame. If you win the bet, the broker pays your stake multiplied with a win payout factor in the 75%..95% range. If you lose, you pay the stake minus a possible loss payout. You’re trading not against the market, but against the broker. The broker needs you to lose, otherwise they would not make any profit. Even if they really pay out your wins, and even if they do not manipulate the price curve, they can still control your profit with their payout factors. So it seems that even if you had a winning system, the broker would just reduce the payout for making sure that you lose in the long run.

However this conclusion is a fallacy. It can in fact be of advantage for the broker to offer a payout that allows you to win, as long as most other traders still lose. A broker has not the freedom of arbitrarily reducing the payout. He’s competing with other brokers. But why would you want to trade binary options anyway, when you also can trade serious instruments instead? If you wanted a binary outcome, you can also achieve this by opening a Put or Call Spread with real options – and this with a serious broker, much higher payout factors (even > 100% in some cases) and the possibility to sell the options prematurely.

But aside from tax advantages in some countries, there is one single compelling reason that might make a binary options trading experiment worthwhile. Profit and trading cost of a binary option are independent of the time frame. So you can trade on very short time frames, which would be difficult, if not impossible with real options or other financial instruments. You can find a discussion of this problem in the Scalping article.

Binary scalping math

The required minimum win rate for binary trading can be calculated from the broker’s win and loss payout:

[pmath size=16]W~=~ {1-Pl}/{1+Pw-Pl}[/pmath]

W = required win rate for break even
Pl = Loss payout
Pw = Win payout

With 85% win payout and no loss payout, you need a win rate of

[pmath size=16]W ~=~ 1/{1.85} ~=~ 54%[/pmath]

54% win rate seem to be manageable on short time frames. The transaction costs of a non-binary, conventional broker would require a much higher win rate, as in the following graph from the Scalping article:

You had to win almost 80% of five-minutes trades – impossible for a trading system under normal conditions unless you enforce that win rate with some tricks, which however won’t help getting in the profit zone.

So, smaller trading costs on low time frames are the obvious benefit of trading binary options. With all the side benefits of low time frames, such as more data for backtests, and shorter drawdown periods in live trading. But how can we take advantage of that? There are three problems to solve.

Three steps to potential binary profit

1. Find a strategy with a win rate that is better than the W determined with the above payout formula. But be aware that prices on small time frames are strongly feed dependent. Normally you won’t know your binary broker’s price source (if he has any at all). For being on the safe side, test with different historical price data from different serious brokers (f.i. Oanda or FXCM) and stay some percent points above the minimum W.
    
2. Find a way to trade automated. Binary brokers often do not want you to do that. Consequently most do not offer a platform or API for automated trading. But they all have a web interface. So you need either a software tool (such as Zorro) that can send key strokes and mouse clicks to a website, or some other means to get your trades to the broker.
     
3. Find a broker that’s halfway honest. At least one that allows you to really collect your gains. All binary brokers make it easy to deposit, but some follow the philosophy: “If you gave it to me, it’s mine.” Make test withdrawals before you deposit large amounts. Keep the account balance small. Check the broker’s opportunity to manipulate the price curve: the more customers they have and the more bets they handle, the more difficult is it to manipulate without attracting unwanted attention. Retrieve as many information as possible about your broker: Where are they really located? For some reason, fraudsters seem to be concentrated in Tel Aviv. Check what customers say about that broker – but be aware: positive statements on trader forums are often planted by the broker himself.

All those issues make trading binary options sort of “messy”. However it’s the messy methods that sometimes offer the best opportunities. Ed Thorp made his first millions not with ‘serious trading’, but with a Blackjack strategy and with a method to estimate the value of warrants, both also considered messy and hard to calculate at that time.

Step 1: The system

A price curve is no random walk. At least not all of the time. Long time frames are often dominated by trend, short time frames by mean reversion. When transaction costs do not matter, it’s not very hard to find a system with > 54% win rate on 5-minutes bars. Here’s a simple example that exploits the mean reversion tendency of short time frames (script for Zorro):

var objective()
{
	return ((var)(NumWinLong+NumWinShort))/(NumLossLong+NumLossShort);
}


function run()
{
	BarPeriod = 5;
	LookBack = 100;
	NumWFOCycles = 20;
	NumCores = -1;
	
	set(BINARY);
	WinPayout = 85;
	LossPayout = 0;

	set(PARAMETERS);
	int TimePeriod = optimize(20,10,100);
	var Threshold = 0.01*(HH(TimePeriod)-LL(TimePeriod));

	if(NumOpenLong+NumOpenShort == 0) 
	{
		LifeTime = 1;
		if(HH(TimePeriod) - priceClose() < Threshold)
			enterShort();
		else if(priceClose() - LL(TimePeriod) < Threshold)
			enterLong();
	}
}

In the C code above we defined an individual objective() function that optimizes the system for binary trading. It measures the system performance as the number of winning trades divided by the number of losing trades. Otherwise the optimizer would hunt for the most robust profit factor, which makes no sense for binary trading.

The setup establishes a 5 minutes bar period, which is the time frame of our bets. We use 20 WFO cycles and let the optimizer use all CPU cores but one. This way the training run takes about 5-10 minutes for 5 years data. The BINARY flag activates binary trades, and we’re simulating a broker with 85% win payout and no loss payout.

We have a mean reverting system that trades whenever the current price is closer than a threshold – here, 1% of recent volatility – to its previous High or Low. The time period for determining the High and Low is the only system parameter that we optimize. You could improve the system in many ways, for instance by optimizing also the threshold, by modifying the objective() function so that it prefers systems with more trades, and by applying a filter that prevents trading in non mean-reverting market regimes. Since we bet on the price in 5 minutes, we’ve set the LifeTime of a trade to one bar. Here’s the equity curve from a 5 years walk forward test with EUR/USD:

The system has about 56% win rate and a notable, although not spectacular positive return. Which is not achieved by the crude mean reversion mechanism, but mostly by amplifying the small entry-exit price differences through binary trading, even though the payout is only 85%. You won’t get a similar result with conventional trades. The same system not trading binary options, but leveraged forex positions produces a very different equity curve (for testing, comment out the BINARY flag and the Payout settings in the code):

With the same trades we have now only 40% win rate and an overall loss, since all the trade profit is eaten up by spread and commission. 

Step 2: Automatizing

How do you let your script automatically enter a bet at the right moment? This is a technical issue unrelated to trading, but it comes up whenever you have a broker with a web based platform and no proper connection for automatizing. Here’s a code snippet for detecting the positions of [Buy] and [Sell] buttons on a website, and automated clicking them:

void main()
{
	int BuyX,BuyY,SellX,SellY; // button coordinates

// open the browser	
	printf("\nOpening the broker's website...");
	exec("http://financial-hacker.com/shark.htm",0,1);

// get the position of the Buy button
	printf("\nRight click on [Buy]!");
	while(wait(50)) {
		int button = mouse(&BuyX,&BuyY,window(""));
		if(button & 2) break;
	}
// wait until right mouse key released	
	while(wait(50)) {
		int x,y,button = mouse(&x,&y,0);
		if(!(button & 2)) break;	
	}

// get the position of the Sell button	
	printf("\nRight click on [Sell]!");
	while(wait(50)) {
		int button = mouse(&SellX,&SellY,window(""));
		if(button & 2) break;
	}
// wait until right mouse key released	
	while(wait(50)) {
		int x,y,button = mouse(&x,&y,0);
		if(!(button & 2)) break;	
	}

// send mouse clicks to Buy and Sell	
	printf("\nI will now click on [Buy]!");
	keys("[click %d,%d]",BuyX,BuyY);
	printf("\nI will now click on [Sell]!");
	keys("[click %d,%d]",SellX,SellY);
	printf("\nDone!");
}

Start the script, and wait until the broker’s website pops up in your browser. Then follow the instructions in Zorro’s message window. Manoever the mouse onto the “Buy” button and hit the right mouse key. Then do the same with the “Sell” button. The script will store the button positions and then use the keys function to send test clicks to both positions of the active window. For testing purposes I’ve imitated a typical binary broker’s trading platform.

You now only need to glue together your trading script with the button clicking script, and adapt the latter to the website of your broker. This is left as an exercise to the reader. And better use improved versions – the scripts here are kept simple for demonstration purposes. As long as the script trades, make sure that the browser window stays in the foreground, or else it can not click on the buttons. For the position size, either enter a fixed size for all positions, or let your script click into the size field and send key strokes to set individual sizes.

Step 3: The broker

Of course I don’t want to recommend a particular binary options broker. In the end, they’re all crooks – but some are crookier than others. Finding a suited broker is, also, left as an exercise to the reader. Binary broker comparison websites are often – surprise, surprise – installed and paid by binary brokers. US citizens are normally not allowed to trade binary options with brokers that are not regulated in the US. Some brokers will accept your deposit nevertheless, but use that as pretext to refuse payout. If you’re a citizen of Israel, you might not be accepted by many binary brokers since they’re not allowed to fraud compatriots. 

Conclusion

It’s often the “messy” and despised trade instruments that can still provide opportunities when they are correctly understood. I’ve uploaded the two scripts to the 2016 repository. You’ll need Zorro 1.52 or above for running them. When you now make huge profits with binary options, don’t forget where the money comes from: Not from the broker, but from his less fortunate customers that maybe just haven’t read the right blog.

Addendum: From all articles on this blog, this one attracted by far the most spam comments. From them it appears that a new lucrative business has established in the orbit of binary brokers: recovery fraud. As soon as you’ve lost your money, you’ll receive offers by “hackers” or “law firms” to recover it, for a fee of course. Where did they get your email from? Naturally from the very broker that bagged your money…

There is no real answer, as the inventors of those methods usually forgot to mention – aside from some vague financial verbiage – how and why they are supposed to work, and which market pattern or inefficiency they are supposed to exploit. Often the methods are merely recipes to be followed meticulously, like the spells in ancient conjuring books. Superstition and esotericism in financial trading are approved by seemingly serious organizations such as the Market Technicians Association, and even trained in their certification programs! Here’s a certainly non-complete list of ways of trading that I still fail to understand.

1. Staring at Price Curves. Opening and closing positions manually without a trading system can work. When you have some information about the particular asset that the other traders don’t have. Or when you got a stroke of luck. Otherwise you can rightfully expect to lose your money at the rate of the transaction costs. In reality it’s even worse – private traders lose on average 13 pips per trade, according to FXCM statistics. The human mind can do a lot of things, but it can not identify a market inefficiency by looking at a price curve. Studies have shown that ‘expert traders’ can not even distinguish real price curves from meaningless random numbers – something even a simple computer algorithm has no problem with. And as to luck, it indeed endows about 35% of private traders every year with some profit or at least no loss, but it has one problem: it can end anytime.

   

2. Technical Analysis. Do traditional ‘technical indicators’, such as MACD, Stochastic, Ichimoku etc., really work? A study by David Aronson, described in his “Evidence-Based TA” book, suggests otherwise. None of the studied basic indicators was any better than throwing a coin. However, the study only applied the indicators to the S&P500 index. The trend experiment suggests that this index is the least predictable of the tested asset types, at least for short-term trend following. Also, Aronson only tested the basic indicators, but not complex combinations of them. So it’s still unknown whether traditional indicators do sometimes work, or not at all. And as long as this is the case, it’s not clear to me why they are so widely and naively used.

 

3. Elliott Waves. Ralph Nelson Elliott claimed in his 1938 published book that the prices of financial assets always move up and down in a fractal pattern of five waves. He gave many examples. And indeed you can see all sorts of waves when you stare at curves long enough. Elliott explained his waves with “mass psychology”, but was not interested in going into details. Although cycles in price curves are real, they have no reason to appear in series of five, or in any series whatsoever. Lacking any rational background, one should think that there was at least some statistical evidence of Elliott Waves – but no. No serious research has ever found any sign of them in real price curves, nor of the countless wave variants invented by Elliott’s many imitators.

 

4. Gann Magic. In the early 20th century, the trader William Delbert Gann was desperate, as he could not support his family with his trading.

   

   But suddenly he discovered the way to success: Planting anecdotes that promoted himself as a genius trader, and selling esoteric trade systems and books. It seems that Gann was the ancestor of all scammers in the trading scene. He did not die rich, though, as in his late years he apparently began to believe in his own methods. He lost at the stock market almost all wealth he accumulated by marketing his methods. I know of no tests that found any value in his magic squares, lines, cycles, pyramids, or angles. But even today, many traders still believe in them, to the great joy of their brokers.

 

5. Astrology. It’s widely accepted as a normal trade method and discussed in many trading books. Even Perry Kaufman, in his standard reference of trading systems, published code to calculate the Jupiter-Saturn cycle! Indeed, trading would be a breeze if you could calculate tomorrow’s prices just from the positions of sun, moon, and planets. But sorry, Perry: Celestial bodies still stubbornly refuse to predict earthly events. No test ever confirmed a correlation of the full moon and the EUR/USD price. Neither was Saturn ever observed dragging down the S&P500 index. And contrary to popular belief, even the sun is not responsible for the change of seasons (it’s the tilt of the earth axis). As long as we got no interstellar stock exchange, trading still happens on earth.

 

6. Rice Candle Patterns. With names like “Three Stars in the South” or “Concealing Baby Swallow”, they bring at least some poetry into trading. Candle patterns had been developed in the 18th century by Japanese traders for predicting the local rice markets. And indeed they might have had some value back then. But even today many traders are still squinting at price charts, hoping for a lucky trade when a candle formation matches a bullish pattern in their “Get Rich with Candle Patterns” book. The TA-Lib fortunately contains indicators of all those patterns, so I could run quick tests of them with several assets, similar to the trend indicator test. You can imagine the result. – By the way, new patterns invented by trade book authors – “Lizard’s Day”, “Gilligan’s Island” etc. – didn’t fare any better.

 

7. Fibonacci Numbers. This simple number series – 1, 2, 3, 5, 8, 13, etc. – can be found in some patterns of geometric growth. But there’s no reason to expect it in the price levels or time periods of financial assets. And indeed, it isn’t there. As far as I know, no one has ever discovered any price series property related to Fibonacci Numbers, or to Golden Ratios, Golden Squares, or Golden Whatevers derived from them. Nevertheless traders seem to like the word “Fibonacci”, maybe because it lets them imagine that they apply serious math. When a system uses Fibonacci Numbers for trade signals, you can safely assume that it would also work, and most likely better, with any other numbers.

 

8. Harmonic Trading Patterns. By connecting pivot points on the price curve, you can produce funny polygonal figures such as diamonds, butterflies, crabs, or bats. Their shapes predict profitable trade entry points. Or do they? I don’t know – I admit I have not yet tested a system based on trade predictions by polygonal figures. I’m only sure that harmonic trading is profitable for the tool and seminar vendors who promote it.

 

9. Your Trading Style. In trading books you’ll often read advices like ‘place the stop loss at a distance that suits your trading style‘. This makes you wonder what Your Trading Style might be. Do you trade fast, slow, risky, greedy, or in the style of the Kamikaze? And should this style affect the distance of the stop loss? I think not. Since any parameter in a trade system has an optimal or most robust value, any other value will therefore produce a less than optimal result. If it suits your style to win rather than lose, better select your trade methods and parameters not by style, but by performance.

 

10. Your Trading Plan. You trade for increasing your fortune in three years from $5000 to $500,000. So you’ve created a detailed plan by which percentage it has to grow every week. Fortunately, it’s just 3%. Unfortunately, the markets could not care less about Your Trading Plan. And if you’re following some system, your wins and losses will often have serial correlation. So when you continue trading after a loss until you reached your weekly goal, usually only your loss will grow. And the initial $5000 will become zero dollars – not in 3 years, but much faster.

 

11. The Holy Grail. On any trader forum you’ll find some lengthy thread about a system with miraculous profits. The thread starter has discovered the ultimate trade method. He feeds the thread periodically with reports of his impressive trade results – such as, doubling his account every month – and vague hints about his miracle algorithm and its complex math. His devotees eagerly absorb any droplet of information in their attempts to replicate the trade method – but alas, some essential ingredient is always missing. Miracles have the nasty habit of disappearing at closer look. The thread will eventually dry out when either the miraculist got broke or when the last follower has realized that he was following a Fata Morgana. If you found a miracle method where this did not happen, please let me know!

 

12. Robots. They are offered by anonymous vendors on countless websites. And supported by likewise anonymous users that all claim on trader forums that they have earned millions with that particular robot. Theoretically, a correctly developed and tested robot could indeed work, if algorithmic trading works at all. But apparently none is offered for sale. The simplest way to sell robots is offering a free trial period: When trades are entered at random, about 55% of users will lose during that period and 45% will win. Those will then order the robot – at least that’s the vendor’s hope.More sophisticated robot vendors provide faked trade histories or even faked live trading equity curves, like this one:

    

    Can you see from the above curve when this robot started selling? Hint: It’s close to the end. How to program a scam robot with an impressive faked equity curve on a verified MyFXBook account is explained in this book (of course for educational purposes only!). But if you ever encounter a real person who really made money by buying such a robot, please notify me…

 

13. Book systems. There are countless books describing countless trade systems – and certainly not all of them are garbage. But amazingly, many of the described systems survive not even a simple backtest. Their authors often suspect this already. That’s why some – for instance, best selling trade book writer Thomas Carr – warn strongly against testing their systems because “backtests are useless anyway”. This is correct – insofar as a positive backtest does not prove that the system works. But a negative backtest means that the money you paid for the book will be the least of your worries when you really trade one of the praised systems. Almost 90% of all systems from books, forums or websites that I’ve tested were clear losers – and this could often be determined in 5 minutes.

 

14. Trade Copy Services. The best-known is Zulutrade®, but many competitors have meanwhile entered this lucrative business. The principle is always the same: Let others place your trades. Follow successful traders and copy their trading. A brilliant idea. If those successful traders really existed. And indeed, you seem to have plenty of choice on the service’s website. Select some of the Top Traders with 500% profit and impressive equity curve, place an investment, and wait for the money to roll in. After a while you will inevitably find that it rolls in the other direction. One after the other of your selected Top Traders will encounter a nasty drawdown, just after you started following them. And their equity curves now all look like the robot curve above. Damn bad luck! – Or is anything else behind it?
    Behind it is no scam, but simple statistics. Thousands of traders compete for followers on copy services. In the Top Traders list you’ll only find those who trade risky but had a stroke of luck so far. Because luck does not last, the Top list is changing permanently. A trader’s survival time in that list is a few weeks, maybe months – then his fortune takes a dip, and takes all his followers with it. The trader won’t mind. He just opens a new account under a new name. He has earned considerably more follower commission than he lost with his trading. Tip: Look not into the Top Traders, but into the Top Followers list (if the service dares to publish one). That might tell you something about the chance of keeping your money when copying trades.
    

 

15. Martingale or d’Alembert Methods. They are used by many trading robots, signal providers, and beginners in roulette: you bet on red or black and double your stake after every loss. Alternatively, you open two new positions for every lost trade. The theory is that a loss increases your win chance the next time. Unfortunately, it won’t. On the contrary, market inefficiencies can be autocorrelated, so after losing a trade there’s a good chance that you’ll also lose the next one. Although martingale systems at first seem to make steady profits – even when trades are entered at random – they will encounter a long loss streak sooner or later and wipe out the account.
    How long will it take until this happens? Assume that you invest 1% of your capital per trade. After a loss you double your investment: 2%, 4%, 8%, 16%, 32%… and the sixth loss will empty your account and cause a margin call. With 50% win rate and uncorrelated returns, the probability of 6 consecutive losses is 0.5 ⁶ = 0.015625. The probability of this event not happening in n trades is (1-0.015625) ⁿ, so the number of trades until the margin call probability exceeds 50% is log(0.5)/log(1-0.015625) = 44. With one trade per day, your account will last about 2 months. A higher win rate, like 90%, won’t help – the average loss is then normally also 9 times higher, so the account lifetime is still the same 2 months. Investing only 0.1% instead – $10 per $10,000 capital – would extend the average account survival time to about 17 months.

 

16. Deadly Accuracy. A 99% win rate is easy – just use a 5 pips profit target and a 500 pips stop loss distance. You’ll then probably win the next 99 trades, regardless of your strategy, and will be worshipped as a god on your trader forum. Unfortunately the 100th trade will hit the stop and eat up all profit. Even worse, that fatal trade can happen anytime, even right at the beginning. You can easily identify scammers that use a high win ratio for selling their systems: their published profit curve looks like a straight upwards slope, with any trade winning about the same small amount. They usually disappear when hit by the 100th trade.
    

 

17. Grid trading. Such a system is a special case of using a high win ratio, up to 100%. It opens many trades at a fixed price grid, and takes profit when the price crosses the next grid line. This method can indeed generate a stream of profits for a while when the prices move up and down, but don’t move too far away from their initial position. Problem is that they eventually do… How much capital would you need for a grid trader to survive such a price move?
    Assume that you have n open trades and the price moves by x pips, x being much larger than the grid size. The move will then increase the balance by x pips and reduce the open trade value by about n/2 * x pips, causing an overall equity drawdown of (n/2-1) * x pips. On a Forex account with 10 mini lots trade size, 1 $ pip cost, and an average of 20 open trades, a 500 pips (5 cents) price move produces a 9*10*500*1$ = 45,000 $ drawdown. Such moves can occur several times per year. More capital and smaller trade sizes can delay, but not avoid that event.
    It’s easy to tweak grid trading systems for surviving any backtest, so they are often found in trade robots and on trade copy services. There are few exceptions where grid trading can make sense – for instance when runaway prices are hedged with some method, or when the price movement is limited by external factors such as a price cap.

Looking at all those methods it seems not really surprising that most traders lose money. But maybe I’m mistaken – if you have some solid evidence that one of the above methods does in fact work, please let me know!