Capitalmind
Capitalmind
Actionable insights on equities, fixed-income, macros and personal finance Start 14-Days Free Trial
Actionable investing insights Get Free Trial
Charts & Analysis

Creating Automated Trading Systems

As a geek and a trader, I’ve always been fascinated by the thought of “automated trading”. The idea is to have algorithms recognise patterns in stock market data, and automatically take on positions when something is favourable. But this is not just random pattern recognition – the whole area of technical analysis is a science that goes back 200 years (or some crazy number, basically it existed before I was born. And before you were born)

So the idea is that you first create a hypothesis. The hypothesis can be something simple (“Stocks go up in April”) to something complex (“A positive slope ADX with DI+ greater than DI-, combined with a reversing slower acceleration parabolic SAR switchover has a positive expectancy”). If you think the second one is a joke, think again – a number of traders use it.

The hypothesis can then be tested using historical data, automatically, by a computer program. The concept is simple – start with some equity at some past date. With the data on that date, would you take the position? If so, allocate the equity appropriately, reducing brokerage and other transaction costs. Also deal with “slippage” – a difference between noted price and what you will actually get (effect of a moving market).

Then go to the next available date, and consider your options again. Would you get out of your current position? (a “stop-loss” or “book-profit”) Would you reverse and short? (for systems that go both ways) Would you take on a fresh position?

All these are trading rules.

Consider also: Position sizing. How much do you allocate per trade? At what point does it make sense to get out on a portfolio level? Say you allocate maximum 2% risk per position – meaning for a 10 lakh portfolio the stop loss per position is Rs. 20,000 regardless of other considerations.

Some of these rules come together as “money management” rules.

The results of a historical back-test will show:

  • Is the system profitable? Obviously you should be net positive over a reasonably long period. But in the interim smaller periods can be net negative – a system that is profitable over a year, could lose money on a monthly basis but have one big month that makes it work. (Such systems should be handled with care)
  • What’s the maximum drawdown? A system will have a drawdown – the highest peak-to-trough difference. This is an indication of the highest percentage you could have lost had you entered at a different point.
  • Number of trades, no. of wins and losses, and average win and loss. Important figures to find out probabilities.
  • Expectancy: This is mathematically (No. of wins)x(average win) – (No. of losses)x(average loss(). If this figure is negative, the system isn’t quite working for you. But this figure is only relevant for a very large number of trades – don’t bother about this below 40-50 trades.

Once we have a back tested system, we have to analyse what works well and what does not. A system with phenomenal profitability – like 500% or something – is unrealistic , so you should be skeptical. Lots of things can make good results, but be a bad system in reality. Firstly, bad data – not all data available is representative of a real world – for example a few weeks ago, NSE reports that Reliance opened at Rs. 3,000 and that was the high of the day. The problem is it could be one trade of 1 share at this price, but if that makes your system look good, you are going to be in trouble when you really implement it. So discard such odd data points.

Second, you can overfit the system to past data. Let’s say the system was “Stocks go up in April”. That didn’t work. SO you said, ok let’s do “Stocks go up in May”. That doesn’t work either. So you add more rules like “April of every year which is an even number, but ignoring April 15, 16 and 17, unless one of them is a weekend, in which case you don’t take April 10, 11 and 12”. This may give you a phenomenal result tested in the past 10 years, but you have an overfit system – it is very unlikely to work in the future.

Once you have a system you like – I will talk about how to choose a system in a different post – you can then “paper trade” it. Meaning, run it and take the next few signals on some “virtual” money, and see if it works over a good period (of say 40-50 trades).

And if both the back test and paper-trade yield good results, you can start putting real money behind the calls. Remember you should continue to backtest as you move forward to see if the market scenario changes results dramatically.

If we were able to identify, back-test and paper trade systems, then can we set up a computer to trade some money automatically? This saves us time and effort to have to keep watching tickers – and instead, focus on building better systems. In theory this is possible but most brokers in India are loath to provide API access to their systems. There could be regulatory hassles to this too.

But my geekiness has prevailed and I think it’s possible to do – create a system, paper trade it and convert it to auto-trade. I have built two such systems, but what I will do is to take a simple system in another post and demonstrate how a system can be built, tested and paper-traded.

This is an exciting field – very old in the U.S. but extremely worthwhile in India to investigate. Even if one avoids technicals, a system based on “news” and “fundamentals” can be build – the code required for it is not dramatically different.

This, both Kaushik and I think, is the future of the stock markets in India. In the U.S. nearly 30% of volumes come from automated systems. It’s time this happens in India – not just will volumes increase, but more discipline will be maintained as computers can be made to follow rules strictly.

Let’s look at this some more in subsequent posts, but tell me – what do you think?

  • Arun says:

    >Of course, as a concept, this should work in India, as it has worked in the US. All you need are the right mathematical models, and the right programmers, and enormous leverage. But the problem I see is the amount of money that can be made from this.

    The problems –
    — In the US, there was a lot of money to be made in the dawn age of program trading, as there were very few people doing it. So there were a lot of inefficiencies to exploit. But now the field is crowded with so many people chasing the same opportunities with the same statistical models. Now many of these quant funds will move en-masse to India if SEBI regulates and mandates APIs for program trading. Actually the foregin brokers will come in and do it themselves as they already have the systems that can handle volumes from program trading desks. So any new opportunity is limited for a new entrant, as the foreign quant desks just have to recalibrate their models for the Indian market
    — So you can have an advantage only if you have a unique stat model that nobody else has thought of so far. Which is very difficult, as most of these techniques have been researched to death
    — Once you have these models, programming these systems is not easy. These are similar to hard realtime systems in terms of response parameters, and also have to process ginormous tick data. Very dificult to fund those kind of programmers in india
    — You will need enormous leverage to get this to work. Because with so many people chasing the same algos, the profit per trade is so miniscule. And if something turns against you, you are wiped out.

    But anyway, I have a feeling that a few very early startup Indian quant funds will make out well, and get bought over by some foreign quant fund.

  • Anonymous says:

    >500% returns? what if trade derivatives and consider only margin as investment.

    Also do you think India can offers(today)inefficient enough markets for outsize gains.
    would like to know your experiences.

  • Anonymous says:

    >All I wish to say dear friend, is it’s gonna be wastage of your intellect, time n energy. I dunno why but have seen such things don’t work consistently.
    A wellwisher

  • Deepak Shenoy says:

    >Arun: True, the opportunities are rare but I’ve seen enough out there that make it worthwhile. And yes, the threat is that foreign funds may come and plug in their systems (but they could do so already). Now with DMA in the picture a lot more opportunities will go away, I’m sure, but as much as program trading continues to work in the US, it will work here as well.

    A stat model needn’t be unique, it just needs to be constantly evaluated. Most of the money is in the thinking, not the doing. The analogy with the US is interesting – although most quant funds came up through teh 80s and 90s, many consistently made money through the last decade as well.

    – I’m not speaking of high frequency trading, which requires tick level data and analysis. That’s like phase 4 🙂 I’m speaking of trend following or mean reversion systems, and I don’t think we need many programmers. Just a few smart ones, and I know I can find them.

    – I don’t believe in over leveraging and we’re not doing what LTCM did. (stat-arb with micro earnings) We are actually going directional, and leverage factors are of the order of 2.5x rather than the 30x you’re thinking about.

    Are the results possible with less leverage? I think so – one of the systems I’m currently trading has a near 80% return this year and about 4% in the last one month (averages about 6 trades a month). Which is quite ok by me – institutions may demand higher returns though.

    But good points and thanks for your comments.

  • Deepak Shenoy says:

    >anon1: Yes, I’m actually speaking of trading derivatives (otherwise there is no leverage :)) But I get very conservative and consider about double the margin required, to make up for bad days.

    Auto-trading is not about efficiency – it’s rare that markets are efficient anyhow – but it’s about what kind of opportunities are there. 6 months ago, classic arb was the vogue – you coudl get 10-12% a year with practically no risk. Today those opportunities have vanished. But there are others – pairs trading, mean reversion, breakouts/trend following strategies still work. I’m trading one right now which “seems” to work, let me give it a few months to test (what’s better than to put real money on it)

    Wellwisher: thanks mate, I know this is fraught with danger. But what is life without some calculated risks – and in this case, it brings together two passions, programming and markets.

    I would love to know what went wrong with the other systems you talked about. Of course, systems will fail, they are supposed to, after a while. That’s why you have to constantly research and find new systems.

  • Anonymous says:

    >Dear friend
    I’m an old guy who’s been around for a while now. Lemme tell ya few big reasons for failure:
    1. Lack of discipline by the traders
    2. System being followed by many becoming obsolete
    3. Mr. Market doesn’t unfortunately function systematically. In fact it punishes AUTO in the long run, look at the failures in US. Do’nt believe in guys who ‘claim’ they have been successful. They’ve a motive behind it.

    To quote Jim Rogers: “I’ve never met a rich technician……unless he is the one who sells his services”.

    And about your programming passion, I can understand how you feel the urge to apply it here. But remember the words of relatively unknown legend called Seykota, “Everyone gets what they want from Markets” We are all in Markets to make money not to satisfy our other passion here.

    Lemme end this comment of mine with another great thought from Seykota: “Easily identifiable patterns seldom repeat”.

    Sorry it might be sounding harsh. But life & Markets are simple ….(although not easy) ….don’t make it complex.

    A wellwisher

  • Deepak Shenoy says:

    >Anon: I understand your points and yes, we have to battle similar situations.

    Systems become obsolete. We know that, and i’ve literally seen them go way. But there’s a way to figure it out and constantly research.

    Jim Rogers hasn’t met many of the people that got rich with technicals; to your quote – Ed Seykota himself, or Bill o’neill (IBD was loss making, the real income was trading)

    We know what we want from the markets, and we’ve done research on not just Seykota but a number of others (read about the Turtles, whose 1986 based rules are public today, and about how, despite having strong, profitable rules, many didn’t succeed)

    What most people want from the markets is the excitement. For me, the passion is profit. Tech and auto-systems are tools in which I have an edge, at least in these markets.

    Seykota’s quote is right, but a lot of times people quote big people just to make or prove a point. They usually don’t have too much experience doign the same thing Seykota has, and therefore words get taken in the wrong context.

    What would be good to know is if you have any experience in the trading system world, in India, what kind of systems they were, and how they went horribly wrong (or seemed to have). It’s all right to say, don’t do this, I know better than you – but I’m not the kind of person who pays homage to age, when what really matters is experience.

    I understand this experience doesn’t come cheap – still, I ask you more details about how your auto-trading went, what were your drawdowns, what kind of historical back-tested results you had etc. The biggest trends were in 2006 and 2007, if you had even the silliest of systems, they would have worked like a charm. In fact even in Oct 2007 and Jan 2008, signals to exit were present in most systems since Jan 12 (even trend following systems!)

    To your last point (“People only claim to be successful”) – honestly, everyone does so. I’ve traded discretion, and I’ve started to trade systems, and I can see the phenomenal difference in discipline. If discipline makes a trader successful, then automated systems is MUCH better than discretion. Seykota, whom you quoted a lot, is a system trader and swears by it.

  • Roshan says:

    >Deepak,
    As far as I remember, Automated trading was illegal. I think every strategy has to be approved by SEBI before you can take it live and per se I think they have ever given approval for only one strategy i.e. inter exchange arbitrage. I personally feel ‘automated trading’ is very risky. More interesting is ‘program trading’. However, even that is insignificant in the Indian markets (whenever they decide to allow it) from the perspective of actual money to be made in relation to the capital and effort that requires to be invested.

  • Deepak Shenoy says:

    >Roshan: Any links on it being illegal? The recent news on DMA was that SEBI was allowing automated trading. It had allowed it for cash-futures arb too, but now it’s across the board.

    Program trading is the same – only orders are not directly placed with the exchange automatically. (Is that what you mean?)

    There are larger opportunities that we think – exactly because there is not much research in the area. A lot of foreign funds don’t come in because of the unavailability of high speed high freq data, live order books etc. I think there are much better opportunities in them, in futures and in options.

  • Anonymous says:

    >Dear Wellwisher,

    Seykota is pure system trader.. no gut feel or emotional discretions here.

    Also he actually (shudder)*programs* some at this late age.

    Maybe,(being the old hand) you’ve heard of cognitive consistency in your take.

  • Roshan says:

    >Deepak, I think what SEBI has allowed is Direct Market Access i.e. an ability for clients to link directly to the exchange – though linked to their broker – and execute trades. They are not (probably) allowed to have those trades coming from bots. I guess they have given this access under pressures from FIIs who have probably experienced dealing with unethical brokers who when they know a big fund is buying, try to hitch a free ride.

    For program trading – look at Wikipedia. It’s a got a more rigid U.S. definition.

    cheers,

  • Deepak Shenoy says:

    >Roshan: SEBI’s DMA circular says:
    “Some of the advantages offered by DMA are direct
    control of clients over orders, faster execution of client orders, reduced risk of errors
    associated with manual order entry, greater transparency, increased liquidity, lower
    impact costs for large orders, better audit trails and better use of hedging and arbitrage
    opportunities through the use of decision support tools / algorithms for trading.”

    and this is what it allows, so DMA is everything – algorithmic trading, auto trading, everything – more like Program trading.

    Yes, DMA will help prevent front-running, and it will also allow automated trading systems to generate orders. We as individuals have always been allowed to program trade (there is no rule disallowing it) – till now institutions were not allowed to. Now they can, so the field is level – so when volumes increase strategies can be tested better.

  • Arun says:

    >”- I don’t believe in over leveraging and we’re not doing what LTCM did. (stat-arb with micro earnings) We are actually going directional, and leverage factors are of the order of 2.5x rather than the 30x you’re thinking about.”

    Deepak, I think the problem is scalability. Funny thing how software concepts apply here.. heh… It might be able to make 10% returns “almost” risk-free a some strategy with 2.5x leverage. But whats the fund size you are talking about. Does this strategy(ies) scale to handle
    — Million $
    — Tens of million $
    — Hunderds of million $
    — billion (not possible anyway with a single strategy)

  • Anonymous says:

    >Talking about quant funds in India, Lotus India has the AGILE fund and it is not working well. I know there’s a geeky bit of you that wants to go in for this and as a programmer I know its fun. But at the end of the day you need a strategy [mind you I am not saying an algorithm, since there can’t be any that is axiomatic across time] which I am not sure would generate you how much. However if you are talking about risk free 10% in 4 months then I think it’s a good project to invest oneself in. Another thing I am curious to understand is that these quant funds rely on trendlines right? In these days of wild gyrations there ain’t any trendline. What do you do then?

    –Arpan

  • Deepak Shenoy says:

    >arun: Scale is an issue with some of the option strategies we use, which tops up at about 5-6 cr. But there are futures and cash strategies, specially those that have a multi-day swing hold in them, that can take much higher volume. Money under management can likely be 100-200 cr. on a futures strategy and an equivalent amuont on cash – without impacting market volumes too much.

    Still, these facts will change over the next two-three years when volumes increase dramatically.

    If we can scale this out to commodities and bonds and all sorts of other stuff that can be traded (even outside India) then we can really achieve scale.

  • Deepak Shenoy says:

    >Arpan: AGILE is a crap version of quant. It’s a long only, 11 stock, restricted to large cap fund which will obviously go down when the market tanks. And the market has seriously tanked. In fact the strategy doesn’t even allow it to stay in cash!

    One system we have developed has generated around 4% this month, of which I have gathered around 2% as I traded it in the middle (was long in the beginning of the month, switched to short later, now out in cash. Two trades)

    Quants don’t need to be trendline based. Trend followign strategies usually require a trend to sustain, and many strategies work on trends. You can have arbs – classic arb, pair trading, mean reversion etc. You can have contrarian strategies (very heavy risk but still).

    And even in trends while there is no longer term trensd, there are still much shorter term trends. Weekly trends and daily trends (measured over minutes) are evident, but in the last one month we have seen a strong 20 day uptrend and we are in a strong downtrend now. Most short term trend following strategies would work. (Not to say they will continue to, but constant research and tracking is important)

  • Siddharth says:

    >Deepak,

    Philosophical bottomline (IMHO) is that years later when you look back where you stand right now and regret that you didnt give it a try, then you know what to do now.

    As far as I know there are plenty bots where I have put little money in BullionVault> Online gold trading.

    And if one thinks of Murphy’s Law in every step then that wont be helpful, right?

    Good Luck.
    Siddharth.

  • Deepak Shenoy says:

    >Siddharth : brilliantly put. Yes, I agree – and some of what you said rang home. One of the discussions we had was “We’re never going to be able to do this again later”. Because then we won’t be able to risk all this money.

    Gotta try. Can’t be afraid of failure.

  • ratheesh says:

    >Great blog. Am a regular reader.

    1) For automated trading in US market just would like to point to
    http://www.equitrader.com
    There are even contests.

    (For some reason it didnt interest me in that, may be because it is US market . The people dont share much (as individual trading algorthms are private (rightly). Hence there is a steep learning curve. Also I dont have a background in finance. )

    2) I do active investing in Indian stock market (started only few months back). So automatic trading environment for Indian stock market may be interesting for many Indian investors.

    3) Off-topic and a Request
    In another post when you have time can you please explain what determines the price of a stock ?. Meaning how a stock price is recalculated after every trade.
    I and my friends do investing in Indian stock markets but none of then actually know how exactly price of a stock is calculated. Each person is saying one-one thing [It is clear they dont know the answer. None of us are from finance background]
    Even though I did a google search I didnt get a satisfactory answer.

    Here are my doubts
    (a) Say stock price today is 100Rs. One person sells 1 stock at Rs 80. What will then be the current price of stock – Rs 80 ? [That defies common sense that 1 trade can reduce the price by 20% and vice versa on the rise]. So how is the price calculated in a more fair way ?

    (b) When I try to sell a stock for market price if there are no buyers will Exchange or company buy the stock ?.
    What happens during crashes, when large number of traders sell. If nobody buys them, will the price come down or stay constant.

    (c) How does exchange does the matching of a bid and sell. [ie what is the logical algorithm it is uses to match a buy and sell. For example is it
    highest buy value to highest offer price or
    highest buy value to lowest offer price
    etc
    ]

    thanks

  • Anonymous says:

    >Great topic.

    I can add for currency markets. As you know, currencies are the biggest “Asset” class which trade globally, and it comes as no surprise that most of the trading systems and mathematical models have been beaten to death in the markets. There are so many Arb models, trend following models, mean reverting models, that I cannot explain.

    I can give a personal example. I had gone into a currency trading dealing room of a big house in Singapore, and there I saw many computers dealing with each other on the Aussie Dollars.

    Models are also based on Fundamentals and yield differentials, which get auto-mated orders based on yield between currencies being seperated by certain basis points.

    BTW…I completly agree with Deepak on the research part. As William Eckhardt put it…”You have to run as fast as you can just to stay in your place”, otherwise you would get outdated in the market. Constant Research is everything.

    Ed Seykota proposes 2 systems in the market which one must have to be suscessful. 1) Actual trading system which gives you the signal on what to do in the markets and 2) self system to be able to follow the trade which the mathematical model has generated. Its in the second system where one fails.

  • Anonymous says:

    >hi Deepak,

    Could you leave a quick comment/link on where one can get the data for Indian markets ?
    Does one need to subscribe to the data feeds from BSE ? Are there alternatives ?
    Thanks,

  • bharat says:

    >learnt reliably that some HNIs in Mumbai are using auto trading to trade on NSE. Recently I saw on freelance programming site, a project by Punjab based trader for auto trading software to create auto trading software having interface with ODIN.It seems auto trading has already started whether SEBI,NSE,approves it or not.I am exploring it’s use for arbitrage trading where complex logic is not needed to generate trade signals.Programme has to locate price differences in two markets and place trades.

  • Anonymous says:

    >Hi,

    Read your views on Automated Trading. I have been using one system to trade markets. However, I have to be glued on screen to trade. Can you refer me someone who can prepare code to place Trades Automatically on ODIN/NEAT usng a pre-defined Algorithm? You may contact me at rajiv.khanna.345@gmail.com Thanks in advance for your help.

  • Deepak Shenoy says:

    >Thanks Rahul, and I found and joined that group as well.

  • Joe Magaro the Trend Following Guy says:

    >I'm so glad that someone finally said it – Position Sizing is one of the most important aspects of successful trading. So many people out there just totally miss that crucial fact, and then they wonder why in the world they don't make any money. Even some of the best books out there only focus on when to buy and sell, and never how much to buy and sell. How specifically do you personally size your positions? I use a volatility adjusted formula based on the average true range and how much I'm willing to lose on a single trade…