For connecting the Zorro platform to a particular broker API, a DLL must be dropped in its Plugin folder. The DLL is then automatically recognized and appears in Zorro’s Broker selection box. For trading with the broker, the DLL exports functions for getting prices and sending orders. These functions are standardized and described in the Zorro manual. They are the same for all brokers. The following 4 functions are the most important:

BrokerOpen – identify the DLL.
BrokerLogin – initialize and store login credentials.
BrokerAsset – retrieve price and other parameters of an asset.
BrokerBuy2 – send a buy or sell order.

These four are sufficient for running most trading strategies. The DLL can export additional functions that are useful, but not mandatory, since Zorro works around them when they are not implemented:

BrokerAccount -get the account state.
BrokerTrade – get the trade or order state.
BrokerHistory2 -get price history.
BrokerCommand – set special modes, get order book, etc.

Zorro users have written DLLs for several broker APIs, but the code is often redundant and invents the wheel many times again. I’ll give here an example of a relatively lean broker DLL that can be used as template for individual REST API implementations. Bittrex is a US based digital currency exchange that supports several hundred crypto currencies and provides free API access. We’re first going to implement the 4 mandatory functions, following the API description on the Bittrex website:

https://bittrex.github.io/api/v3

We will use the free Visual C++ Community Edition from Microsoft. Aside from the exported functions, we need some internal functions to print messages, send HTTP requests, parse a JSON response, and generate a digital signature. For all this there’s ready code available on the Internet, but there are also ready functions in Zorro’s library. To make life easier, we simply import the Zorro function library to the DLL. The VC++ setup for generating such a DLL is described here:

https://zorro-project.com/manual/en/dlls.htm

The setup is for a C++ trading strategy, but works as well for a broker API DLL. First we’re going to implement the adminstrative stuff, the BrokerOpen and BrokerLogin functions:

#define PLUGIN_TYPE 2
#define PLUGIN_NAME "Bittrex V3"
#define DLLFUNC extern "C" __declspec(dllexport)

int (__cdecl *BrokerMessage)(const char *Text);
int (__cdecl *BrokerProgress)(const int Progress);
...

DLLFUNC int BrokerOpen(char* Name,FARPROC fpMessage,FARPROC fpProgress)
{
  strcpy_s(Name,32,PLUGIN_NAME);
  (FARPROC&)BrokerMessage = fpMessage;
  (FARPROC&)BrokerProgress = fpProgress;
  return PLUGIN_TYPE;
}

The BrokerOpen function is Zorro specific. It sets up two function pointers for printing message and sending heartbeats to Zorro. It’s needed for recognizing the DLL in the Plugin folder and letting the plugin name, “Bittrex V3”, appear in the broker scrollbox. 

Next we need the login function that’s called at the begin and the end of any algo trading session

struct GLOBAL {
  int PriceType,VolType,OrderType;
  double Unit; // smallest trade unit
  char Path[256]; // send path buffer
  char Key[256],Secret[256]; // credentials
  char Symbol[64],Uuid[256]; // last trade symbol and UUID
  char AccountId[16]; // account currency
} G; // parameter singleton
...

DLLFUNC int BrokerLogin(char* User,char* Pwd,char* Type,char* Accounts)
{
  if(User) { // login
    memset(&G,0,sizeof(G));
    strcpy_s(G.Key,User);
    strcpy_s(G.Secret,Pwd);
    return 1;
  } else  // logout
    return 0;
}

This function only sets up the G struct, a singleton that holds all global variables of the DLL. The User and Pwd arguments contain the Bittrex key and secret. They are taken from the Zorro login fields or from the account list. Type is the account type, ignored here because Bittrex has no demo accounts.  Account is the account identifier and also ignored since we’re not using sub-accounts.

Next comes the function to retrieve prices and other parameters of a particular cryptocurrency. It’s a bit more complex and requires a HTTP request to the API. Some requests require authentication, some not. For all this we first implement a send() function in the DLL that uses Zorro’s functions for sending http requests and generating a HMAC signature. The authentication method by the Bittrex REST API V3 is a bit complex:

To properly sign an authenticated request for the Bittrex v3 API, the following headers must be included: Api-Key, Api-Timestamp, Api-Content-Hash, Api-Signature. Api-Timestamp is the current time as a UNIX timestamp in epoch-millisecond format. Api-Content-Hash is a SHA512 hash of the request body, Hex-encoded (with no request body, a SHA512 hash of an empty string). For creating the Api-Signature, first generate a string by concatenating the following items: Contents of the Api-Timestamp header; the full URI used to make the request, including query string; the HTTP method of the request, in all caps (GET, POST, DELETE, etc.); Contents of the Api-Content-Hash header. Sign this string via HmacSHA512, using the API secret as the signing secret. Hex-encode the result and populate the Api-Signature header with it.

This is the implementation:

#define RHEADER "https://api.bittrex.com/v3/"
...

int sleep(int ms)
{
  Sleep(ms); 
  return BrokerProgress(0);
}

char* send(const char* Url,
  int Mode = 0,
  const char* Method = NULL,
  const char* Body = NULL)
{
  static char Url[1024], Header[2048], Signature[1024],
    Buffer1[1024*1024], Buffer2[2048];
  *Signature = *Header = 0;
  sprintf_s(Url,"%s%s",RHEADER,Url);
  if (Mode & 1) { // Authentication required
    strcpy_s(Header, "Content-Type:application/json");
    strcat_s(Header, "\nAccept:application/json");
    strcat_s(Header, "\nApi-Key: ");
    strcat_s(Header,G.Key);
    strcat_s(Header, "\nApi-Timestamp: ");
    __time64_t Time; _time64(&Time);
    static __time64_t Offset = 0;
    char* TimeStamp = i64toa(Time*1000 + Offset++); 
    strcat_s(Header, TimeStamp);
    strcat_s(Header, "\nApi-Content-Hash: ");
    char* Hash = hmac(Body, 0, 0, 512);
    strcat_s(Header,Hash);
    strcpy_s(Signature, TimeStamp);
    strcat_s(Signature, Url);
    if(Method && *Method)
      strcat_s(Signature, Method);
    else if(!Body)
      strcat_s(Signature,"GET");
    else
      strcat_s(Signature,"POST");
    strcat_s(Signature, Hash);
    strcat_s(Header, "\nApi-Signature: ");
    strcat_s(Header, hmac(Signature, 0, G.Secret, 512));
  }

  char* Response = Mode & 2? Buffer2 : Buffer1;
  int MaxSize = Mode & 2? sizeof(Buffer2) : sizeof(Buffer1);
  int Id = http_request(Url,Body,Header,Method);
  if(!Id) goto send_error;
// wait 30 seconds for the server to reply
  int Size = 0, Wait = 3000;  
  while (!(Size = http_status(Id)) && --Wait > 0)
    if(!sleep(10)) goto send_error;
  if (!Size) goto send_error;
  if(!http_result(Id,Response,MaxSize)) goto send_error;
  Response[MaxSize-1] = 0; // prevent buffer overrun
  http_free(Id);
  return Response;

// transfer unsuccessful?
send_error:
  if(Id) http_free(Id);
  return NULL;
}

The above code adds an authentication header with signature when Mode == 1. Mode == 2 selects a second, smaller response buffer and thus allows to evaluate two API responses at the same time. The sleep function generates a delay with a heartbeat to keep the Zorro window responsive during long requests. If it returns 0, someone has hit Zorro’s [Stop] key and all operations should be aborted. The Windows _time64 function has only 1-second resolution, so we add an incremented Offset for generating unique timestamps.

We’re now prepared to implement the BrokerAsset function:

double fixAmount(double Value)
{
  int Exp = log10(Value);
  if (Exp >= 0) Exp++;
  return pow(10,Exp);
}

DLLFUNC int BrokerAsset(char* Symbol,double* pPrice,double* pSpread,
double *pVolume, double *pPip, double *pPipCost, double *pMinAmount,
double *pMargin, double *pRollLong, double *pRollShort, double *pCommission)
{
  sprintf_s(G.Url,"markets/%s/ticker",fixSymbol(Symbol));
  char* Response = send(G.Url,2);
  if(!Response) return 0;
  double Bid = strvar(Response, "bidRate", 0.),
    Ask = strvar(Response,"askRate",0.);
  if (Ask > 0. Bid > 0. && pSpread) 
    *pSpread = Ask - Bid;
  double Last = strvar(Response, "lastTradeRate", 0.);
  if(Ask == 0. || G.PriceType == 2) 
    Ask = Last;
  if (Ask == 0.) return 0; // symbol does not exist
  if(pPrice) *pPrice = Ask;
  if(pVolume) {
    sprintf_s(G.Url,"markets/%s/summary",fixSymbol(Symbol));
    Response = send(G.Url,2);
    if (Response) {
      if (G.VolType == 4)
        *pVolume = strvar(Response, "volume", 0);
    else
        *pVolume = strvar(Response, "quoteVolume", 0);
    }
  }
  if (pMinAmount) { // get lot amount
    sprintf_s(G.Url,"markets/%s",fixSymbol(Symbol));
    Response = send(G.Url,2);
    if (Response) {
      *pMinAmount = fixAmount(strvar(Response,"minTradeSize",0.000001));
      if (pPip) {
        int Exp = strvar(Response,"precision",8);
        *pPip = pow(10,-Exp); 
        while (*pPip * *pMinAmount < 0.000000001)
         *pPip *= 10; // avoid too small pip cost
        if (pPipCost)
          *pPipCost = *pPip * *pMinAmount;
      }
    }
  }
  if (pMargin) 
    *pMargin = -100; // no leverage
  return 1;
} 

This function is supposed to return current price, current ask-bid spread, and current volume. It can also optionally request other asset specific parameters when available from the API. Otherwise Zorro will replace them with parameter values from the asset list. Broker APIs rarely provide all requested parameters. Make sure to only calculate and fill a parameter when its pointer passed to the function is nonzero. Most of the pointers are NULL most of the time.

The send function uses the second buffer because we’ll need price requests internally for calculating the account state. For getting lot amount, pip size, and pip cost, two more API requests are needed. The values from the response string are parsed with Zorro’s strvar function. The fixAmount function converts a value to its next-higher power of 10 – for instance, 7 is converted to 10 and 0.07 to 0.1. We’re doing this because we don’t want a strange number for the lot amount. The pip size is calculated from the precision, but we prevent it from going too small. Bittrex has no leverage, so the margin cost is always 100%.    

The next and final function sends an order to the API:

DLLFUNC int BrokerBuy2(char* Symbol,int Volume,double StopDist,double Limit,double *pPrice,int *pFill)
{
  if(!isConnected() || !Volume) return 0;

// compose the body
  char Body[256] = "{\n";
  strcat_s(Body,"\"marketsymbol\": \"");
  strcat_s(Body,fixSymbol(Symbol));
  strcat_s(Body,"\",\n\"direction\": \"");
  strcat_s(Body,Volume > 0? "BUY" : "SELL");
  strcat_s(Body,"\",\n\"type\": \"");
  strcat_s(Body,Limit > 0. ? "LIMIT" : "MARKET");
  strcat_s(Body,"\",\n\"quantity\": \"");
  double Size = labs(Volume);
  if(G.Amount < 1.) Size *= G.Amount;
  strcat_s(Body,ftoa(Size));
  if (Limit > 0.) {
    strcat_s(Body,"\",\n\"limit\": \"");
    strcat_s(Body,ftoa(Limit));
  }
  strcat_s(Body,"\",\n\"timeInForce\": \"");
  if ((G.OrderType&2) && Limit > 0.) 
    strcat_s(Body,"GOOD_TIL_CANCELLED"); // limit orders only
  else if (G.OrderType&1) 
    strcat_s(Body,"FILL_OR_KILL"); // fill all or nothing
  else 
    strcat_s(Body,"IMMEDIATE_OR_CANCEL");
  strcat_s(Body,"\"\n}");

  char* Response = send("orders",1,0,Body);
  if(!Response) return 0;

  char* Uuid = strtext(Response,"id","");
  if(!*Uuid) return 0; // failed
  strcpy(G.Uuid,Uuid);
  double Filled = strvar(Response,"fillQuantity",0);
  if(Filled == 0. && !(G.OrderType&2))
    return 0; // unfilled FOK/IOC order
  double Price = strvar(Response,"proceeds",0);
  if (Filled > 0. && Price > 0. && pPrice)
    *pPrice = Price/Filled;
  if (*pFill) *pFill = Filled/min(1.,G.Unit);
  return -1; 
}

The function first composes a message body in this JSON format (example):

{
  "marketSymbol": "ETH/BTC",
  "direction": "BUY",
  "type": "LIMIT",
  "quantity": "0.1",
  "limit": "0.008",
  "timeInForce": "IMMEDIATE_OR_CANCEL"
}

If the order is accepted, the response has this JSON format:

{
  "id": "12345-6789-007-4711",
  "marketSymbol": "ETH/BTC",
  "direction": "BUY",
  "type": "LIMIT",
  "quantity": "0.1",
  "limit": "0.008",
  "timeInForce": "IMMEDIATE_OR_CANCEL",
  "fillQuantity": "0.05",
  "commission": "0.000002",
  "proceeds": "0.00001",
  "status": "CLOSED"
}

The relevant fields, especially the id, the fill amount, and the fill price, are parsed from the response and returned. The return value -1 indicates to Zorro that the order generated no identifer number, but an UUID. It was stored and can be retrieved with a subsequent command. G.Unit is the lot amount that was generated with the previous BrokerAsset call. 

The remaining functions not described here – BrokerTrade, BrokerAccount, BrokerHistory2, and BrokerCommand –  are implemented in a similar way. You can find them in the full source code in the Source folder of the latest Zorro version. 

Update, November 2023: Bittrex recently announced that they will terminate their operation. So the Bittrex API plugin described herein lost its purpose, but you can still use the code as a template for connecting Zorro to any other broker or crypto exchange APIs – at least as long as they remain operational.  

A broker API allows software to trade, receive price quotes, and download price history. Those three functions are essential for an automated system. Good-to-have additional features are retrieving trade status, account status, and asset parameters. That’s six functions, or seven when you count login/logout. A broker API has often more than 100 functions. So you should assume that at least the 6 essential are more than covered. But sadly, it is not so. The debacle begins already with installing and starting up the API. 

API startup

Installing and starting a broker API can be so simple. The ideal case is a broker that provides no software at all, but just a protocol and a URL for a TCP/IP or UDP connection. You can then use well-established libraries such as curl or enet for connecting to the broker. In the second-best case you get an API DLL that you can redistribute to your clients. Your software then just calls the login function of that DLL, submits the authentification data and the connection is established. But for some unknown reason, many brokers feel obliged to provide connections to their APIs in much more complex and creative ways.

Broker A required any user to run an API installation package. After an installation dialogue with several Windows incompatibility warnings, the installer produced a bunch of DLLs. Then you can go through a lengthy procedure with many steps of building up all the internal tables and data structures that you need for connecting to your account. In a previous version, the installer also modified the Registry, so you could not step around it and distribute the DLLs directly to your clients together with your software. They had to go through the installation procedure themselves. This is fortunately not required anymore with the current version.

Broker B offers no direct connection at all. You have the choice of connecting either to their bloated Java trade platform, or to a small “Gateway” program. When you connect to the Java platform, your software must once per day log out, restart the platform, and log in again since the platform automatically shuts down every 24 hours. When you connect through the “Gateway”, you can not open their platform at the same time (not even from a different PC). So you can’t even check your positions. If you do, the Gateway connection breaks down and must be manually re-established. This is not a bug as you might think. It is intentionally implemented this way (according to the broker) as a service to their users.

Both broker A and broker B are major, world-wide operating Forex and stock brokers.  

Getting price quotes

Broker A keeps its asset prices in a “Quote Table” on his server. For getting a price quote you must first subscribe to that asset. For this you do not just call a function, you build up a “Request Factory” and generate a “Subscription Request”. Now you can enter the procedure of receiving a response on your request, which involves the creation of a “Response Listener”. Subscribing to an asset this way takes about 30 primary lines of C++ code – and this does not include the secondary code for sending the requests, generating listeners, and all the supplementary stuff. My clients can consider themselves fortunate that I don’t charge by code line. Getting the price itself then only requires looking up the asset in the server’s quote table.

Broker B makes it easy: You can not subscribe market quotes by API at all. You must do that manually on their website, and it is not free (except for Forex). For getting the price, a simple API call is sufficient. Or would be, if you knew the primary exchange, secondary exchange, asset type and sub-type, and other asset parameters. If any of them is not right, you get no price. And don’t think that these exchange and other parameters are documented somewhere. You’re supposed to find them out by trial and error, or to ask fellow programmers on the Internet.

Getting price history

Although any algorithm trades in real time, it still needs price history on startup for calculating initial values of its indicators and price-analyzing functions. With no access to price history, you had to wait a couple days before the first trade can be placed. Since this is not really practical, price history is an essential API function.

Broker A provides price history with no major problem. I can almost not believe it. Ok, you must again fire up the “Request Factory”, generate requests and listeners, but about 50 code lines enable you to download historical prices. The broker charges no fee for those prices (you can even download them with a demo account), and at least the recent data, from 2010 and above, is in acceptable quality. Eight out of ten points for the price history from broker A.

Broker B again got creative and enterprising. Their price history from demo accounts is worthless – you must indeed open a real account and deposit $10,000 when you want historical data. This sum, of course, is also required from a poor coder who only wants to implement a function for getting price history. Not that you get it then easily. Technically it’s simple – a few lines of code are enough – but you only have 60 requests. After that, the price history server is shut down for 10 minutes. 60 requests are enough for a single-asset system, but not for starting a portfolio system, and certainly not for backtests. Why this bizarre limit? Maybe broker B can’t afford a fast Internet connection for their price history server. Maybe they are using the CEO’s old PC, located in his living room, and the bandwidth goes down anytime when his kids play Tetris on it. Whatever the reason, that’s only one out of ten points for the price history from broker B.

Is there a broker with even zero points for the price history? Yes, there is: Broker C has no API function for retrieving historical prices at all. Apparently they had not expected that someone would really use their API. I had no choice but to advise my client to choose another broker for his algorithmic system.

Handling trades

We programmers tend to think binary. When we send a fill-or-kill order to the broker API, we assume that the position will be either opened (“yes”), or it will not (“no”). But from time to time, broker APIs produce a third result, like “maybe” or “I won’t tell”. A position that is maybe opened or maybe not can result in an orphaned trade. This is a trade opened by the broker, but not reported back to the trading software, and consequently not under automated control. This trade won’t be closed by a reversal or stop and can thus accumulate a large loss. It can blow the account. Orphaned trades are not a good thing.

Theoretically, they would be so easy to prevent. The API just needs to provide a simple order function with an identifier and a validity limit, like this:

openAtMarket(string Asset, int Amount, int ID, int SecondsValid);

ID = dear broker, please store this identifier together with the trade and use it for retrieving the trade status in case of doubt.
SecondsValid = dear broker, please disregard the order when after the given number of seconds your API still doesn’t know if it can fill it or not, or is unable to confirm it for whatever reason. 

That’s the ideal, but how’s the reality?

Broker A accepts a ton of parameters in his order functions, but no user-supplied trade ID and no SecondsValid limit. After sending an order and going through the usual “Request Factory” and “Response Listener” rigmarole, you’ll receive a confirmation and can retrieve the trade identifier for managing or closing it later. Or you’ll get notified that the order failed. Or neither of both. It happens that the API does not react at all on your order, maybe due to a server hickup, an Internet outage, or because it became confused by too many Factories and Listeners. Then you can only guess if your order made it to the server or not. If it did, you have an orphaned trade. But at least you can relatively easily identify and close it manually.

Broker B provides a simple order function, even with a user-supplied ID. And you get confirmation by a simple callback function that the order was filled or not. Usually not, since broker B needs again lots of information about the asset and the order routing, and won’t execute the order if something is missing or not fitting together. But at least you know the ID. This makes you think that you can later check the trade status. Think again: Broker B won’t reveal any information about your trades. At least not to you. In fact they do not store your trades at all. They only store net positions. So there is no way to identify orphans. Not even manually, since you can’t open the trade platform as long as an automated system is running.

Account parameters and trading costs 

Broker A provides a function where you can request leverage, pip size, pip cost, rollover, and any other parameters of the selected asset. Enough information to calculate margin and trading costs in advance, what is a good thing for an automated system – if there weren’t an important parameter missing. The API does not allow to retrieve the commission. You must enter it manually and store it per asset. Why this important parameter is not provided by the API remains broker A’s secret.

Broker B does not have this problem, as its API provides no asset and trading cost information at all – not even the leverage. For compensation, it supports a sort of “virtual order” that can be used to calculate profit, similar to Zorro’s “phantom trades”. Theoretically you could use that to calculate some of the asset parameters in a complex way from a series of virtual orders with different lot sizes. My clients didn’t pay me for that. For simulating a broker B account in a backtest, you must therefore enter all required asset parameters and trading costs manually in a spreadsheet – which is no easy task due to the complicated fee and margin structure of that broker.

There are APIs for almost all imaginable software tasks. And most are well structured nowadays and relatively easy to use. Why is it not so with most broker APIs? Dear brokers: If you think about providing API access to your customers, please look first into other broker’s solutions for learning how NOT to do it. And if really necessary, please add detailed documentation about which of the market or order parameters are really required and in which combination.

And if you then still are not sure how to implement your new API, just hire me. I had the dubious pleasure to implement dozens of APIs so far, and can show you how an API for algo trading should look like. And even though you’ll then have to hand over some shekels to my employer, you’ll make up for this by winning an excellent reputation. At least among the poor coders and hackers who have to implement your API.

Addendum (January 2017): For the sake of fairness it should be mentioned that broker B has meanwhile removed the bizarre 60 requests limit for historical data. One year of data is now available. Maybe they have read this blog. Or their CEO got a new PC for Christmas. Whatever the reason, it’s still not sufficient for backtests, but at least for live trading even large portfolio strategies.

Coding contracts can work out well or fail miserably. It depends on you and on the coder. But mostly on you. As a contract coder* who often hires other contract coders, I got meanwhile some experiences with both sides of the medal. So here are some coder hiring suggestions for increasing the chance to get something useful back for your money.

Find the coder

There are three Internet places where you can look for strategy coders: trader forums, freelancer websites, and tool vendors. The latter is my preferred way to acquire coding manpower. I check out advertised indicators, trading systems, or software tools that have some relation to the project at hand. If I like what I see, I contact the developer and ask if she or he does contract work. Most do. This way I know that the new coder has knowledge about the subject, can bring in her experience, and can do commercial quality work.

When contracting on trader forums or freelancer websites, be aware that coding abilities are not equally distributed among the population. Speed and code quality by freelancers differ by factors up to 10. So you need some means to judge the coding beforehand. Check out example projects (obviously, don’t hire someone who has nothing to show). If you can’t decide among five coders, hire all five. Give them all a small task, a little script, something that can be done in about 3 hours. Pay all five for the small task, then select the one with the best work for the real project. The invested money is well spent. A connection to a good coder is a very valuable asset.

Grill the coder

Before hiring, interrogate the freelancer about his skills and his approach to the project. And put attention on clear and prompt answers. Many issues arise from misunderstanding. Freelance coders must be able to express themselves not only in script, but also in plain English. As you’ll most likely communicate by email, test the promptness. If it takes your aspiring coder three days to respond on an email even in the negotiating phase, he’ll likely need three weeks to react on an issue after delivery and payment – if at all.

One important thing to ask is how the coder tests his scripts. In software development, proper test procedures are more important than writing code. A good coder has some test framework for putting his code under stress. This is twice important when the coder, as I sometimes do, sub-contracts parts of the task to other coders. Understanding and testing other people’s scripts is an additional effort that pays back in better code quality.

Naturally you should also make sure that the coder has basic understanding of trading, and knowledge in math and statistics – or can at least quickly come to grips with a new field of knowledge. A degree in math, physics, engineering, or computer science is of advantage, even if it pushes up the coder’s fees. But understanding the algorithm on a deep level can be essential for coding a serious trade system. My favorite way to grill a coder was giving her a sheet of paper and asking her for a quick proof of Pythagoras’ theorem. Unfortunately, this only works in a face to face interview, not by email.

Feed the coder

An essential part of cooperation with coders is your project description. Know your goals precisely, but not too precisely. Some clients are able to describe clearly what they want. Others send long and utterly confused project specifications that leave you completely in the dark of what you have to code. From some clients I got laconic flow diagrams like this: “Analyze price curve with neural network – open position when profitable – close position at price maximum – repeat”. Others have 22-page concepts with a complex strategy described up to the finest detail of the user interface button colors. And as they felt that all this could somehow limit the flexibility, they usually require any detail to be adjustable with many different parameters and modes.

A trading strategy is no iPhone app. No one knows beforehand if your idea will work or not. If not, any implemented additional details are wasted money. So split your project into milestones. Or let the coder do it. The first milestone is always the proof of concept: a script of just your trade idea, with no stops, extra filters, money management, user interface, or other gimmicks. This should be coded at first, and carefully tested in all possible variants. If this script turns out to be profitable, go on with the next step. Otherwise you won’t need all the rest.

If your trade idea or algorithm is very secret, the coder should offer you a non-disclosure agreement. This way he accepts the responsibility to keep your trade method for himself. Which he should do anyway, of course.

Don’t listen to the coder

At least not to me. I used to advise clients about the success chances of their concept – even before coding and testing it. For instance I declared a certain strategy doomed to fail since it seemed just a complex hodgepodge of conventional indicators. The client was unimpressed. He told me to code it nevertheless. Now, 18 months later, he’s still living from the returns of that system. Since then I refrain from unsolicited advising.

But you should very well listen to the coder when he suggests different methods of implementing your system, or modifications for simplifying the algorithm or making it more effective. The coder possibly knows your algorithm better than you.

Pay the coder

I will be frank: For almost all coders (I’m no exception), coding is one of the greatest things on earth. For some, it’s better than sex. We all would code for free if there were no alternative. However, professional ethics and our bank account require that we demand payment.

Since coding speed can be very different, it’s normally not advised to pay a coder by hour, unless you’ve worked with her a long time and know her well. Normally the coder will quote the project at a fixed price. The price should include not only the coding, but also a support and warranty period after delivery. It usually does not include training the script or adjusting the parameters. This is your task, as it’s very time consuming and thus would be expensive, but requires no code knowledge.

Just as coding speed and quality, coding prices vary strongly among freelancers. Obviously the cheap price is not always the best. On the other hand, if the project is really interesting and the coder would really like to get the job, you can often negotiate a discount. Some ballpark figures: My own contracts start at EUR 170 per strategy, which is my employer’s required minimum for a simple script. Contracts for complex projects with options trading or machine learning algorithms are in the EUR 1000-2000 range; individual user interfaces, and special analysis or input / output methods can raise the contract to EUR 10,000 or more. My largest contract so far was 1.5 million EUR. Sadly, the small contracts are much more frequent than the large ones…

Coders usually demand advance payment. How much depends on your prominence in relation to the coder. If you are Warren Buffett and the coder is an unknown guy from India, you’ll be able to negotiate payment after delivery. If you’re a nondescript retail trader and the coder is employed by a large company, he’ll likely ask you for 100% advance payment. In most cases the terms will be 50% in advance and 50% after delivery.

If the script works, pay the rest immediately. This won’t cost you much, but makes a good impression on the coder. I only ever had one single case where the contract price was not paid at all. This is always a bad idea: We coders talk to each other. Not paying a contract might get you in trouble to ever find a coder again. Or worse: You might find one who implements a little surprise in your trading script…

Be independent of the coder

Even when you hire a coder, it makes sense when you acquire a basic understanding of coding – for instance through the Zorro tutorial. It will take a day or two. But it will allow you not only to better judge the code quality, but also to modify the script beyond just setting parameters. Then you don’t need to always wait for a new quote by the coder. And you save money. Remember, I have to charge EUR 170 (at least) no matter if I re-write the script completely or only modify a single line.

Conclusion

Be very, very meticulous in selecting a coder and working with her or him, and try to understand as much of the code and its structure as possible. This can be the deciding factor for the success of your project. – In case you’re curious about my above mentioned one point five million contract, or about the single case when a contract was not paid, or about my most bizarre experience as a hired coder so far: You can read that story here. But it is not suited for minors. So if you’re below 18, please do not click on that link. Thank you.

* Some coders prefer to be referred to as ‘programmers’. The rationale is that a coder just converts an algorithm in code, while a programmer develops the algorithm. This distinction is not really useful: You can not code a trading strategy when you’re illiterate in algorithms. Thus, I don’t care if I’m a coder or a programmer as long as I’m paid.

Choice of languages

Algo trading systems are normally based on a script in some programming language. You can avoid writing scripts entirely by using a visual ‘strategy builder’, ‘code wizard’ or spreadsheet program for defining your strategy. But this is also some sort of programming, just in a different language that you have to master. And visual builders can only create rather simple ‘indicator soup’ systems that are unlikely to produce consistent trading profit. For serious algo trading sytems, real development, and real research, there’s no stepping around ‘real programming’.

You’re also not free to select the programming language with the nicest or easiest syntax. One of the best compromises of simplicity and object orientation is probably Python. It also offers libraries with useful statistics and indicator functions. Consequently, many strategy developers start with programming their systems in Python… and soon run into serious limitations. There’s another criterion that is more relevant for system development than syntax: execution speed.

Speed mostly depends on whether a computer language is compiled or interpreted. C, Pascal, and Java are compiled languages, meaning that the code runs directly on the processor (C, C++, Pascal) or on a ‘virtual machine’ (Java). Python, R, and Matlab is interpreted: The code won’t run by itself, but is executed by an interpreter software. Interpreted languages are much slower and need more CPU and memory resources than compiled languages. It won’t help much for trading strategies that they have fast C-programmed libraries. All backtests or optimization processes must still run through the bottleneck of interpreted trading logic. Theoretically the slowness can be worked around with ‘vectorized coding’ – see below – but that has little practical use.

R and Python have other advantages. They are interactive: you can enter commands directly at a console. This allows quick code or function testing. Some languages, such as C#, are inbetween: They are compiled to a machine-independent interim code that is then, dependent on implementation, either interpreted or converted to machine code. C# is about 4 times slower than C, but still 30 times faster than Python.

Here’s a benchmark table of the same two test programs written in several languages: a sudoku solver and a loop with a 1000 x 1000 matrix multiplication (in seconds):

Speed becomes important as soon as you want to develop a short-term trading system. In the development process you’re all the time testing system variants. A 10-years backtest with M1 historical data executes the strategy about 3 million times. If a C-written strategy needs 1 minute for this, the same strategy in EasyLanguage would need about 30 minutes, in Python 2 hours, and in R more than 10 hours! And that’s only a backtest, no optimization or WFO run. If I had coded the trend experiment in Python or R, I would today still wait for the results. You can see why trade platforms normally use a C variant or a proprietary compiled language for their strategies. HFT systems are anyway written in C or directly in machine language.

Even compiled languages can have large speed differences due to different implementation of trading and analysis functions. When we compare not Sudoku or a matrix multiplication, but a real trading system – the small RSI strategy from this page – we find very different speeds on different trading platforms (10 years backtest, ticks resolution):

• Zorro: ~ 0.5 seconds (compiled C)
• MT4:  ~ 110 seconds (MQL4, a C variant)
• MultiCharts: ~ 155 seconds (EasyLanguage, a C/Pascal mix)

However, the differences are not as bad as suggested by the benchmark table. In most cases the slow language speed is partically compensated by fast vector function libraries. A script that does not go step by step through historical data, but only calls library functions that process all data simultaneously, would run with comparable speed in all languages. Indeed some trading systems can be coded in this vectorized method, but unfortunately this works only with simple systems and requires entirely different scripts for backtests and live trading.

Choice of tools

Zorro is a software for financial analysis and algo-trading – a sort of Swiss Knife tool since you can use it not only for live trading, but also for all sorts of quick tests. It’s my software of choice for financial hacking because:

• It’s free (unless you’re rich).
• Scripts are in C, event driven and very fast. You can code a system or an idea in 5 minutes.
• Open architecture – you can add anything with DLL plugins.
• Minimalistic – just a frontend to a programming language.
• Can be automatized for experiments.
• Very stable – I rarely found bugs and they were fixed fast.
• Very accurate, realistic trading simulation, including HFT.
• Supports also options and futures, and portfolios of multiple assets.
• Has a library with 100s of indicators, statistics and machine learning functions, most with source code.
• Is continuously developed and supported (new versions usually come out every 2..3 months).
• Last but not least: I know it quite well, as I’ve written its tutorial…

A strategy example coded in C, the classic SMA crossover:

void run()
{
  double* Close = series(priceClose());
  double* MA30 = series(SMA(Close,30));	
  double* MA100 = series(SMA(Close,100));
	
  Stop = 4*ATR(100);
  if(crossOver(MA30,MA100))
    enterLong();
  if(crossUnder(MA30,MA100))
    enterShort();
}

More code can be found among the script examples on the Zorro website. You can see that Zorro offers a relatively easy trading implementation. But here comes the drawback of the C language: You can not as easy drop in external libraries as in Python or R. Using a C/C++ based data analysis or machine learning package involves sometimes a lengthy implementation. Fortunately, Zorro can also call R and Python functions for those purposes.

R is a script interpreter for data analysis and charting. It is not a real language with consistent syntax, but more a conglomerate of operators and data structures that has grown over 20 years. It’s harder to learn than a normal computer language, but offers some unique advantages. I’ll use it in this blog when it comes to complex analysis or machine learning tasks. It’s my tool of choice for financial hacking because:

• It’s free. (“Software is like sex: it’s better when it’s free.”)
• R scripts can be very short and effective (once you got used to the syntax).
• It’s the global standard for data analysis and machine learning.
• Open architecture – you can add modules for almost anything.
• Minimalistic – just a console with a language interpreter.
• Very stable – I found a few bugs in external libraries, but so far never in the main program.
• Has tons of “packages” for all imaginable mathematical and statistical tasks, and especially for machine learning.
• Is continuously developed and supported by the global scientific community (about 15 new packages usually come out every day).

This is the SMA crossover in R for a ‘vectorized’ backtest:

require(quantmod)
require(PerformanceAnalytics)

Data <- xts(read.zoo("EURUSD.csv", tz="UTC", format="%Y-%m-%d %H:%M", sep=",", header=TRUE))
Close <- Cl(Data)
MA30 <- SMA(Close,30)
MA100 <- SMA(Close,100)
 
Dir <- ifelse(MA30 > MA100,1,-1) # calculate trade direction
Dir.1 <- c(NA,Dir[-length(Dir)]) # shift by 1 for avoiding peeking bias
Return <- ROC(Close)*Dir.1 
charts.PerformanceSummary(na.omit(Return))

You can see that the vectorized code just consists of function calls. It runs almost as fast as the C equivalent. But it is difficult to read, it can not be used for live trading, and many parts of a trading logic – even a simple stop loss – cannot be coded for a vectorized test. Thus, so good R is for interactive data analysis, so hopeless is it for writing trade strategies – although some R packages (for instance, quantstrat) even offer rudimentary optimization and test functions. They all require an awkward coding style and do not simulate trading very realistically, but are still too slow for serious backtests.

Although R can not replace a serious backtest and trading platform, Zorro and R complement each other perfectly: Here is an example of a machine learning system build together with a deep learning package from R and the training and trading framework from Zorro.

More hacker’s tools

Aside from languages and platforms, you’ll often need auxiliary tools that may be small, simple, cheap, but all the more important since you’re using them all the time. For editing not only scripts, but even short CSV lists I use Notepad++. For interactive working with R I recommend RStudio. Very helpful for strategy development is a file comparison tool: You often have to compare trade logs of different system variants and check which variant opened which trade a little earlier or later, and which consequences that had. For this I use Beyond Compare.

Aside from Zorro and R, there’s also a relatively new system development software that I plan to examine closer at some time in the future, TSSB for generating and testing bias-free trading systems with advanced machine learning algorithms. David Aronson and Timothy Masters were involved in its development, so it certainly won’t be as useless as most other “trade system generating” software. However, there’s again a limitation: TSSB can not trade or export, so you can not really use the ingenious systems that you developed with it. Maybe I’ll find a solution to combine TSSB with Zorro.

References

TIOBE index of top programming languages

Speed comparison of programming languages

Update (November 2017). The release of new deep learning packages has made TSSB sort of obsolete. For instance, the H2O package natively supports several ways of features filtering and dimensionality reduction, as well as ensembles, both so far the strength of TSSB. H2O is supported with Zorro’s advise function. Still, the TSSB book by Davin Aronson is a valuable source of methods, approaches, and tips about machine learning for financial prediction.

Download links to the latest versions of Zorro and R are placed on the side bar. A brief tutorial to both Zorro an R is contained in the Zorro manual; a more comprehensive introduction into working with Zorro can be found in the Black Book.

Because the money amount always corresponds to the same amount of private and public debt, this debt amount also must inevitably grow, in spite of all the political lamentoes. Reducing public debt would either destroy money or increase private debt proportionately. This happened in fact in the United States around the turn of the millennium, when then-President Bill Clinton managed to get by without borrowing, and even achieved a budget surplus. Which caused interests to drop and banks to look elsewhere for distributing their money. The indirect result of Clinton’s good deed was a massive increase in private debt that eventually led to the mortgage crash of 2007.

How to acquire it in large amounts

Money is considered a good thing in almost all cultures. After all, it allows you to do the things you want, and – even more important – not to do things you don’t want to do. It thus represents freedom. You can get to it with different methods. The most obvious is taking away other people’s money. Here’s the Top Ten fortunes by known villains, according to Forbes (in US $):

1. Hugo Drax – 7.6 billion
2. Auric Goldfinger – 6.5 billion
3. Max Zorin – 5.3 billion
4. Lex Luthor – 4.7 billion
5. Franz Sanchez – 1 billion
6. Ernst Stavro Blofeld – 640 million
7. Karl Stromberg – 640 million
8. Elektra King – 420 million
9. Francisco Scaramanga – 115 million
10. Dr. Julius No – 110 million

But the most successful in money taking are not, as you might think, drug cartel bosses or leaders of criminal underground organizations, but presidents and other heads of state. They can take their share of money with no risk, since they need not fear the law. Here’s the Top Ten of the acquired fortunes by this way (in US $):

1. Muammar Gaddafi, Libya – 55 billion
2. Hosni Mubarak, Egypt – 50 billion
3. Mohamed Suharto, Indonesia – 25 billion
4. Alexander Lukashenko, Belarus – 12 billion
5. Mobutu Sese Seko, Congo – 7 billion
6. Ben Ali, Tunisia – 4 billion
7. Gnassingbé Eyadéma, Togo – 4 billion
8. Obiang Nguema, Equatorial Guinea – 3 billion
9. Slobodan Milosevic, Serbia – 1 billion
10. ‘Baby Doc’ Duvalier, Haiti – 600 million

This list does naturally not include assets of monarchs such as the Sultan of Brunei, who have no need of pilfering because the country belongs to them by law anyway. Or of dictators like Wladimir Putin, whose estimated 125 billion booty (plus a 17,000 sqft palace) officially does not belong to them, but is kept for them by friendly oligarchs. The listed sums must also be considered in relation to the economy of the country. To bag 600 million in grinding poor Haiti is a much more impressive performance than the lousy one billion that Milosevic could siphon off in industrialized Serbia. But as long as you’re neither a supervillain, nor a head of state, nor both at the same time, you have no choice but to use other means to acquire money. There’s also the method of buying something cheap and selling it dear. Not as profitable as being a head of state, but it still can produce some handsome gains (annual income in US $):

1. Jim Simons, Renaissance – 1.7 billion
2. Ken Griffin, Citadel – 1.7 billion
3. Raymond Dalio, Bridgewater – 1.4 billion
4. David Tepper, Apaloosa – 1.4 billion
5. Izzy Englander, Millenium – 1.1 billion
6. David Shaw, Shaw Group – 750 million
7. John Overdeck, Two Sigma – 500 million
8. David Siegel, Two Sigma – 500 million
9. Andreas Halvorsen, Viking – 370 million
10. Joseph Edelman, Perceptive – 300 million

All in this list acquired their wealth with algorithmic trading. Which is the topic of most of the rest of this blog. It does not produce any goods. But on the other hand, it does not steal from anyone. On the contrary, private, small-scale financial trading can boost demand and soften economic inequality. It can redistribute money from the rich to the poor. So it should be rewarded by the government, for instance by a tax exemption. Well, one can dream, at least…

Why financial hacking?

Part of my job is developing financial tools and trading systems for clients. So far we coded hundreds of trading strategies with all sorts of algorithms for all sorts of financial instruments. Some worked and fulfilled the client’s expectations. Some failed miserably. And some worked in the backtest, but not in live trading. Coming from a background of theoretical physics and computer game programming, I wondered why trading seems not to be an exact science at all. What is the difference between a successful and a doomed strategy? And how can you determine that before actually trading it?

On this blog I’ll attempt a hacking approach to algorithmic trading. Hacking is nothing illegal, it’s just a pragmatic way to solve problems. Hackers prefer experiment over theory. They don’t give a damn about the wisdom of gurus or authorities. So I’ll start with considering all praised trade systems worthless and all “trader’s wisdom” irrational and nonsense until proven otherwise. I will try to evaluate by systematic experimenting whether, why, when, and how algorithmic trading does work. My goal is to find out how it can be a reliable income source for a private trader. This might require complex statistical or machine learning algorithms – but that’s no big deal with today’s software tools. All scripts and software to the articles will be put up for download, so anyone interested can reproduce all the results and use the strategies. After all, successful private trading is for the common good.

As this blog is about algorithmic trading, I’m going to post here a lot of algorithms and source code. Naturally not any trader is able to read code. On the other hand, some basic code and math understanding is required for making sense of the articles. To go from zero to a full understanding of the articles on this blog, here’s a list of Useful Books.