Whatever software we’re using for automated trading: We all need some broker connection for the algorithm to receive price quotes and place trades. Seemingly a simple task. And almost any broker supports it through a protocol such as FIX, through an automated platform such as MT4™, or through a specific broker API. But if you think you can quickly hook up your trading software to a broker API, you’re up for a bad surprise. Dear brokers – please read this post and try to make hacker’s and coder’s lifes a little easier!
A broker API allows software to trade, receive price quotes, and download price history. Those three functions are essential for an automated system. Good-to-have additional features are retrieving trade status, account status, and asset parameters. That’s six functions, or seven when you count login/logout. A broker API has often more than 100 functions. So you should assume that at least the 6 essential are more than covered. But sadly, it is not so. The debacle begins already with installing and starting up the API.
Installing and starting a broker API can be so simple. The ideal case is a broker that provides no software at all, but just a protocol and a URL for a TCP/IP or UDP connection. You can then use well-established libraries such as curl or enet for connecting to the broker. In the second-best case you get an API DLL that you can redistribute to your clients. Your software then just calls the login function of that DLL, submits the authentification data and the connection is established. But for some unknown reason, many brokers feel obliged to provide connections to their APIs in much more complex and creative ways.
Broker A required any user to run an API installation package. After an installation dialogue with several Windows incompatibility warnings, the installer produced a bunch of DLLs. Then you can go through a lengthy procedure with many steps of building up all the internal tables and data structures that you need for connecting to your account. In a previous version, the installer also modified the Registry, so you could not step around it and distribute the DLLs directly to your clients together with your software. They had to go through the installation procedure themselves. This is fortunately not required anymore with the current version.
Broker B offers no direct connection at all. You have the choice of connecting either to their bloated Java trade platform, or to a small “Gateway” program. When you connect to the Java platform, your software must once per day log out, restart the platform, and log in again since the platform automatically shuts down every 24 hours. When you connect through the “Gateway”, you can not open their platform at the same time (not even from a different PC). So you can’t even check your positions. If you do, the Gateway connection breaks down and must be manually re-established. This is not a bug as you might think. It is intentionally implemented this way (according to the broker) as a service to their users.
Both broker A and broker B are major, world-wide operating Forex and stock brokers.
Getting price quotes
Broker A keeps its asset prices in a “Quote Table” on his server. For getting a price quote you must first subscribe to that asset. For this you do not just call a function, you build up a “Request Factory” and generate a “Subscription Request”. Now you can enter the procedure of receiving a response on your request, which involves the creation of a “Response Listener”. Subscribing to an asset this way takes about 30 primary lines of C++ code – and this does not include the secondary code for sending the requests, generating listeners, and all the supplementary stuff. My clients can consider themselves fortunate that I don’t charge by code line. Getting the price itself then only requires looking up the asset in the server’s quote table.
Broker B makes it easy: You can not subscribe market quotes by API at all. You must do that manually on their website, and it is not free (except for Forex). For getting the price, a simple API call is sufficient. Or would be, if you knew the primary exchange, secondary exchange, asset type and sub-type, and other asset parameters. If any of them is not right, you get no price. And don’t think that these exchange and other parameters are documented somewhere. You’re supposed to find them out by trial and error, or to ask fellow programmers on the Internet.
Getting price history
Although any algorithm trades in real time, it still needs price history on startup for calculating initial values of its indicators and price-analyzing functions. With no access to price history, you had to wait a couple days before the first trade can be placed. Since this is not really practical, price history is an essential API function.
Broker A provides price history with no major problem. I can almost not believe it. Ok, you must again fire up the “Request Factory”, generate requests and listeners, but about 50 code lines enable you to download historical prices. The broker charges no fee for those prices (you can even download them with a demo account), and at least the recent data, from 2010 and above, is in acceptable quality. Eight out of ten points for the price history from broker A.
Broker B again got creative and enterprising. Their price history from demo accounts is worthless – you must indeed open a real account and deposit $10,000 when you want historical data. This sum, of course, is also required from a poor coder who only wants to implement a function for getting price history. Not that you get it then easily. Technically it’s simple – a few lines of code are enough – but you only have 60 requests. After that, the price history server is shut down for 10 minutes. 60 requests are enough for a single-asset system, but not for starting a portfolio system, and certainly not for backtests. Why this bizarre limit? Maybe broker B can’t afford a fast Internet connection for their price history server. Maybe they are using the CEO’s old PC, located in his living room, and the bandwidth goes down anytime when his kids play Tetris on it. Whatever the reason, that’s only one out of ten points for the price history from broker B.
Is there a broker with even zero points for the price history? Yes, there is: Broker C has no API function for retrieving historical prices at all. Apparently they had not expected that someone would really use their API. I had no choice but to advise my client to choose another broker for his algorithmic system.
We programmers tend to think binary. When we send a fill-or-kill order to the broker API, we assume that the position will be either opened (“yes”), or it will not (“no”). But from time to time, broker APIs produce a third result, like “maybe” or “I won’t tell”. A position that is maybe opened or maybe not can result in an orphaned trade. This is a trade opened by the broker, but not reported back to the trading software, and consequently not under automated control. This trade won’t be closed by a reversal or stop and can thus accumulate a large loss. It can blow the account. Orphaned trades are not a good thing.
Theoretically, they would be so easy to prevent. The API just needs to provide a simple order function with an identifier and a validity limit, like this:
openAtMarket(string Asset, int Amount, int ID, int SecondsValid);
ID = dear broker, please store this identifier together with the trade and use it for retrieving the trade status in case of doubt.
SecondsValid = dear broker, please disregard the order when after the given number of seconds your API still doesn’t know if it can fill it or not, or is unable to confirm it for whatever reason.
That’s the ideal, but how’s the reality?
Broker A accepts a ton of parameters in his order functions, but no user-supplied trade ID and no SecondsValid limit. After sending an order and going through the usual “Request Factory” and “Response Listener” rigmarole, you’ll receive a confirmation and can retrieve the trade identifier for managing or closing it later. Or you’ll get notified that the order failed. Or neither of both. It happens that the API does not react at all on your order, maybe due to a server hickup, an Internet outage, or because it became confused by too many Factories and Listeners. Then you can only guess if your order made it to the server or not. If it did, you have an orphaned trade. But at least you can relatively easily identify and close it manually.
Broker B provides a simple order function, even with a user-supplied ID. And you get confirmation by a simple callback function that the order was filled or not. Usually not, since broker B needs again lots of information about the asset and the order routing, and won’t execute the order if something is missing or not fitting together. But at least you know the ID. This makes you think that you can later check the trade status. Think again: Broker B won’t reveal any information about your trades. At least not to you. In fact they do not store your trades at all. They only store net positions. So there is no way to identify orphans. Not even manually, since you can’t open the trade platform as long as an automated system is running.
Account parameters and trading costs
Broker A provides a function where you can request leverage, pip size, pip cost, rollover, and any other parameters of the selected asset. Enough information to calculate margin and trading costs in advance, what is a good thing for an automated system – if there weren’t an important parameter missing. The API does not allow to retrieve the commission. You must enter it manually and store it per asset. Why this important parameter is not provided by the API remains broker A’s secret.
Broker B does not have this problem, as its API provides no asset and trading cost information at all – not even the leverage. For compensation, it supports a sort of “virtual order” that can be used to calculate profit, similar to Zorro’s “phantom trades”. Theoretically you could use that to calculate some of the asset parameters in a complex way from a series of virtual orders with different lot sizes. My clients didn’t pay me for that. For simulating a broker B account in a backtest, you must therefore enter all required asset parameters and trading costs manually in a spreadsheet – which is no easy task due to the complicated fee and margin structure of that broker.
There are APIs for almost all imaginable software tasks. And most are well structured nowadays and relatively easy to use. Why is it not so with most broker APIs? Dear brokers: If you think about providing API access to your customers, please look first into other broker’s solutions for learning how NOT to do it. And if really necessary, please add detailed documentation about which of the market or order parameters are really required and in which combination.
And if you then still are not sure how to implement your new API, just hire me. I had the dubious pleasure to implement dozens of APIs so far, and can show you how an API for algo trading should look like. And even though you’ll then have to hand over some shekels to my employer, you’ll make up for this by winning an excellent reputation. At least among the poor coders and hackers who have to implement your API.
Addendum (January 2017): For the sake of fairness it should be mentioned that broker B has meanwhile removed the bizarre 60 requests limit for historical data. One year of data is now available. Maybe they have read this blog. Or their CEO got a new PC for Christmas. Whatever the reason, it’s still not sufficient for backtests, but at least for live trading even large portfolio strategies.
18 thoughts on “Dear Brokers…”
>You’re supposed to find them out by trial and error, or to ask fellow programmers on the Internet.
You use reqContractDetails. You can give it, e.g. a symbol and instrument type and it returns all the info you need. Which you should then save in a local db for future use.
Of course you shouldn’t be using Broker B for data at all.
Broker B sounds a lot like Interactive Brokers!
Geeze. That’s unfortunate, I’m not looking forward to having to deal with this as I get my system up and running. Any broker you recommend for the best API?
I can not recommend any broker API as 100% perfect, but Oanda has a relatively good API. But it also depends on what you want to trade. Forex brokers offer usually better APIs than stock brokers, and newcomers have better APIs than old-established brokers – which is at least an encouraging sign. Dependent on the API structure, you’ll need between one afternoon and one week for the implementation. So it’s a manageable task even with a very bad API.
Broker B has its challenges no doubt. Java bloat is still a huge problem. Thinkorswim’s java bloat is even worse. Anyways I believe there is a workaround the problem of Broker B’s problem – to concurrently connect with the gateway and trading platform. Using the account management website portal one can create a secondary username that is tied to the same account. Login to the gateway with the secondary username and login to the platform with the primary username.
Thanks for the hint! The problem with this workaround: you can only create secondary user names for the main account, not for additional accounts such as a paper account that you’re normally using when you code and test an API implementation. But for trading with the main account this does indeed step around the connection issue of broker B.
The ideal broker for algo trading would provide a binary protocol for MD and trades (also instruments description and metadata on start up or by request). The historical MD and trades available as the same binary format messages but downloaded as files. Is there such a broker even for one venue in one single country? (I don’t know one.)
I am reaching out to you from IG Markets – global leader in CFDs.
We are interested in discussing potential marketing collaboration. We are looking for algo trading communities to work with. IG provides a strong platform supporting Web API, Auto trading and MT4. We would love to see if affiliation or IB partnership opportunities with financial-hacker.com.
I’d like to schedule a convenient time to have a chat and discuss further.
Would you be available sometime next week?
Digital Marketing Specialist
IG, Level 15, 55 Collins Street, Melbourne VIC 3000
D: +61398601751 | T: +61398601711
40 YEARS OF TRADING
INDICES | SHARES | FOREX
COMMODITIES | BINARIES
We tried a number of Forex APIs, and eventually settled on Dukascopy JForex.
It’s a huge, rambling Java API, but very flexible and powerful, with excellent access to decent quality historical data. It was one of very few that enabled us to execute the more unusual aspects of our strategy. You can run strategies on their servers (if you trust them with your code), client side within their trading client, or client side with their SDK. Data goes back and forth as Java objects, so unless you set up a fancy bridge you’re working in Java.
You run across the odd bug but there’s usually another way to do things, and the one time we hit a show-stopper they fixed it for us.
Commission and spread are very competitive and execution has been OK so far. An actual Swiss bank, so the regulation gives a measure of comfort. In the UK, Atom8 white labels the Dukas marketplace and offers tax benefits to UK residents.
@Ariel Lao I have tried IG in UK and to be honest the experience with the API support and customer service was not good at all, so not good our team gave up on working with IG.
I’ll contact you privately and hopefully after talking to You i’ll happily come back here to say things are different.
Gradually internet search engine began identifying these techniques, as well as have actually taken positive techniques.
This is my first time visit at here and i am truly happy to read
all at single place.
It looks like Broker B has eliminated their $10,000 initial deposit requirement. It is now $0.
As of Forex: run Metatrader in a Docker env, develop an EA accepting commands and requests via zeroMq, control the MT-EA from any other programming language with nearly no latency.
This way you will get a “nice” API but it takes some time to develop this architecture.
It’s not difficult to develop, but it will no advantage over other means of interfacing with Metatrader.