From Rags to Automated Rags... my attempt at automating Forex.


sven

Active Member
Kick it off

After a quick intro in Hi there...time to get serious?
Followed by a basic brain dump over on Can't measure success without setting a Goal..or two...and you are gonna need a plan.

I basically set out a very high level outline of where i am and where I want to get to and the things that have caused me issue in the past.

I put out a basic straw man of the following...
  • Platform choice.
    • Define requirements
    • Research the options
    • Evaluate shortlist
    • POC
    • Choose.
  • Broker choice.
    • Supported brokers for platform.
    • Additional Api requirements?
    • Trust
    • Funding.
  • Define System Pipeline
    • High level approach to Algo research, dev through to deployment. Timeframe/instruments etc. How to evaluate, discard, test deploy etc. How to monitor post live!!, how to integrate new systems into the portfolio.
    • Risk parameters and kill switch.
  • Algo!!!!
    • ideas and the pipeline.... loop loop loop.. Go live!!!... monitor as per pipeline.
    • GOTO Algo...

Some great points from @jack that I may have it backwards choosing platform and broker before attempting to investigate an edge. There is a lot of sense in that statement and obviously dipped in some experience.
I did mull it over and ultimately discounted (for now) the idea of getting into data /edge first and then looking for a broker that meets my needs for access in favour of just looking for a platform then broker.

Why if its great advice sven?
  1. I've been plagued/burnt in the past where I've went down a rabbit hole building testing frameworks and playing with jupyter notebooks. I have always struggled to get systems to the market.
  2. I'm primarily interested in time series and not order book based algos. Not to say I wouldn't but I simply don't have any insight on them. I think the tools for TS are freely available and won't require me to be bespoke.

But ultimately I'm most concerned about not getting anything to market and for now at least want to just keep it as vanilla as possible. I realise that will likely result in vanilla results but it's a start.


So.. the sooner I start this the quicker I can toss it in the bin and comeback to @jack 's Idea lol.

With that in mind and the absence of a better home, I'll treat it like a journal.. you can see my random thoughts and feel free to chime in.

Next up.... Platform Choice.
 

jack

Administrator
Staff member
Why if its great advice sven?
  1. I've been plagued/burnt in the past where I've went down a rabbit hole building testing frameworks and playing with jupyter notebooks. I have always struggled to get systems to the market.
  2. I'm primarily interested in time series and not order book based algos. Not to say I wouldn't but I simply don't have any insight on them. I think the tools for TS are freely available and won't require me to be bespoke.
Mm.. could you describe the kinds of pitfalls you ran into doing this? Worked out a few possible edges on paper only to realize they don't materialize when trading live markets?

This is where a lot of quants / analysts benefit from working with experienced traders (specifically ones experienced in DMA and market structure.)

Could you give an example of a rabbit hole you went down that didn't yield the results you first expected (strategy wise)?
 

sven

Active Member
Mm.. could you describe the kinds of pitfalls you ran into doing this?
Oh thats easy.. I basically spent months making cool infrastructure and tools and webapps and forgot one of the primary goals..... alpha lol

Worked out a few possible edges on paper only to realize they don't materialize when trading live markets?
I think most edges don't materialize :/ I did the classic, gridding systems look like they can't lose etc.. long before I made any effort to understand risk or expectancy etc.. plug that bad boy in and test over a short period lol.

This is where a lot of quants / analysts benefit from working with experienced traders (specifically ones experienced in DMA and market structure.)
I'm sure. if I had 1/2 a clue on DMA and market structure I'd be dangerous. I think given my current restrictions I would have to look towards crypto to gain exposure. Not sure I've got what it takes lol.
 

jack

Administrator
Staff member
How's progress?

No one subject is hard, there's just a lot of individual things to understand... it might appear daunting but keep chipping away and you'll be surprised how much you can pick up even in a few months time.
 

sven

Active Member
How's progress?
Somewhere between zero and none I would say jack. pretty much planned tbh. I’ve been trying to give my self a couple of full weeks away from everything except family over the holidays.

That mission is complete and I’m back to work tomorrow. Likely be a bit of carnage for a week or so as projects get back on track at work. Family goals done, work and pick up the gym/ health goals again after the short break.

I have made a little progress on platforms and brokers which I will write up soon but short version is pepperstone uk and first account there is the
ctrader Razor. Played with a bunch of demos and brokers but pepperstones support were great.

so a little write up in the works to keep me honest on why and next mission pipeline and ideas.
Happy new year!
 

sven

Active Member
Platform Choice
Rather than litter threads across the FXGears-a-sphere I'll just keep them inline unless they warrant their own thread.

  • Define requirements
  • Research the options
  • Evaluate shortlist
  • POC
  • Choose.
pepperstone uk, ctrader with a razor account (comms + spread)

Define requirements
Before I started looking at platforms and the likes I needed a basic idea what I wanted and the criteria of which I would judge them. So with that in mind I jotted down a mini wish list of the ideal platform. It looked a little like this and in no particular order.
  • Full auto
    • Comprehensive support for EA/automated trading out of the box.
    • multi pair system dev and testing.
    • multi timeframe support.
    • clear documentation
    • existing resources on the Tinterweb
    • optimisation tools.
  • Discretionary /semi-auto
    • Ability to customise the interface with trading panels for order entry, risk management etc.
    • range bars, tick charts(although not as truthful on FX obviously but do help normalise the bar sizes.)
    • decent range of timeframes.
  • Ability to interface with different brokers and not be tied.
  • Support account hedging (to make life simple with multiple systems in the same account)
  • Mobile/Web support

Research the options
Well if you have been messing about with FX for any length of time you will know that Metatrader is pretty much the defacto platform of choice by traders and brokers. Of course a relatively new cTrader by spotware is around and supported but then the well can get a little dry unless you prefer a platform like quantconnect(which has some merit unless you want to trade using your amazing skills manually)

Heck check out Jacks excellent Best practice accounts for MT4/5 and cTrader to see some of the common pro's and con's of both plafforms as well as 2 brokers that support them.

Of course you can build your own or use frameworks that others have build such as backtrader in python. Then it becomes more of a question o brokers and if they support an API of some form.

I'm interested in platforms so lets stick to MT + cTrader
Now I'm sure that some(maybe) will pick me up and say "Hey XXX can do that!!!!". Probably can. I probably missed it...breath.


CriteriaMT4/5cTrader
Support for EA/automated tradingCome on. You know that MT is has approximately 700 million indicators and EAs. They have been around since the internet was born.. don't believe me? check out the interface ;)

It has its own language which is kinda c/c++ ish..
cTrader now supports algos in the same platform as the trading app. Once upon a time it was an additional cAlgo app but they merged it into cTrader and call it 'automate'

Its in C#
Multi pair backtestingYes MT5Yes
Multi TimeframeYesYes
Clear DocumentationYes MQL5Yes cTrader Guides and of course its generic c#
Existing ResourcesLoads/the most/Every whereSome / algos marketplace and can install with one click.
Optimisation toolsMT5 is quite comprehensive.

  • genetic optimisation
  • 1/2 decent charts for reviewing parameter space
  • easy click forward testing
  • agents to speed up work.
cTrader provides a somewhat limited amount of tools.
  • Genetic optimisation
  • Custom Fitness function (to allow you to build your own evaluation function in optimisation.. Nice)
Interface CustomisationYes - own custom componentYes - own custom components
Bar TypesRange bars/tick bars only through indicators and offline charts AFAIKNative range bars and Tick charts, renko and others.
Timeframeslots / more with offline chartslots
Broker supportEveryoneSeveral and usually decent ones
Account Hedging supportYes MT5 now supports(after much pushing by clients)Yes
Mobile/Web supportMobile app is quite good.
Web app is ok.
Mobile app is quite good
Web is close to desktop with limited chart types (time + range)



Evaluate shortlist
Frankly there is little between them. However, after playing with them for sometime I got a feel for how their different tools sat with me.

I just couldn't muster any love for MT's UI. God I know in the scheme of automating systems this it pretty low on the criteria but it just made me sad opening it up. cTrader on the other hand felt fresh and almost a joy to use. After a few years of TradingView charts being available we have been spoilt in the joy of typing a few letters for the pair chooser to pop up, typing a few numbers for the timeframes to come up etc. Neither MT/CT were close but its clear CT sucks less lol.

CTs DOM entry and order management is years ahead of MT. You can do partial closes, breakevens etc without algo requirements. Even the web app supports this.

Range bars and tick charts in CT are really good. They are native and it feels like it vs indicators running to generate offline charts in MT. This is a big deal for me. Discretionary shortterm trading I prefer to use tick charts instead of 3-5min charts. I know they are not as useful in Forex but with a sensible broker they can still help to normalise the bar sizes and make things a little more sensible. Range bars are something I had already done some initial testing with years ago.
They are great at removing the whipsaws that happen from low volatility periods as the bars don't print. Also if you take a look at the distribution of returns on them they tend to look like an M.. ie almost no chance of zero movement and higher chance of full movement which is pretty much the opposite of normal time based bars.
With range bars you only need 1 number to represent the entire movement of that bar and its easy to normalise as R/c-o.. +1 full up bar.. +0.75 went south for 25% and then finished up.. etc. There has to be an edge there right? somewhere?. I digress but has to be useful for ML..

c#. Now we know in the perfect world we would have our own systems running in our own frameworks and plug into our chosen brokers. Using MT this is a challenge albeit not impossible. CT uses c# and it would be a relatively simple job to take Algos from that and build your own framework that mocks the API or simply rewrite to use your own API (keeping the key events like start, on tick, on bar etc)
AND... you have pretty much the universe of c# available to you. Yes in MT you can write some DLL's or bridges are available but its not quite the same.


Optimisation in MT stood out for over CT based on its charts making it easier to review the parameter space. Its clear that CT has someway to go however of course it does have the extensive fitness functionality as well as allowing you to create custom Fitness functions. This is quite a bit deal if you have your preferred way of evaluating performance. CT results are available and can be exported so external charting could be done easily.


Proof of Concept (POC)
I used MT4 previously messing about for years(why?) I know it intimately. I demoed a MT5 account and spent as much time in it as I could. I didn't build anything as I've done it before. Took a look at the key differences between MT4/5

cTrader I traded discretionary for a week or so on demo (yey I'm rich with demo money) to get a feel for the tool. I also, downloaded a few sample algos and indicators. built a trade entry panel for fun.


Choose.
So when I started I thought I'll be rational, base it on the facts and see what can be done by the various offerings.
...in the end though it comes down to the feels you have. Thats as important as facts given you will need to enjoy the process.

So.... cTrader won this round. Not really much between them and of course each has strengths and weaknesses but its important to remind ourselves of my goals.
I want to automate systems robustly with an off the shelf offering and also discretionary trade as a secondary goal.

The reality is that seeking Alpha is what is hard. Neither cTrader or MT will limit me in functionality or brokers. Neither would prevent me from taking a winning algo and redeploying in another framework/api or whatever in more than a handful of days/weeks.

So at the end of the day just need to make a choice. believe in it and move forward. We can always come back to it.
 
Last edited:

sven

Active Member
Broker Choice

Broker choice.
  • Supported brokers for platform.
  • Additional Api requirements?
  • Trust
  • Funding.
See: Jacks.. Best forex plaforms for demo/practice accounts
Life's too short.. I chose Pepperstone UK. The end

Ok maybe a little more to it than that. If I had wanted to use MT4 I could have stuck with Oanda (and will but just for the REST api and spreadbetting should I get some traction).

But I decided on cTrader as we know. With that I checked out a few brokers.

Trust and ease of funding and withdrawals
I see people pushing unregulated brokers that are based in the freaking Caymans on reddit and other places because "leverage man". Anyone that is treating this like a business and doing any kind of due diligence will push those straight in the bin.

...push..

That leaves regulated and in the UK thats going to be the FCA with some ESMA overhead so I checked out a few; Pepperstone UK, FX Pro and IC Markets.

Pepperstone were the only ones that reached out to me and checked if I had everything I needed.. tried to get an understanding of who I am and what I want.. They didn't push any live account or recommend anything but they did reach out. The others I got a flurry of emails but thats about all.

Type of account
FX is typically a spread game with a bunch of made up terms like ECNs / STP and the likes when they are maybe not that in the strictest definition. Ultimately I want a NDD platform. I'm not overly concerned if the broker takes the other side but I would rather have transparency in my pricing.

cTrader makes it pretty much impossible for a broker to get tricky as cTrader essentially white labels and own the infrastructure. That said you can usually choose between spread based or comm's + spread.

Pepperstones Razor account is comms+spread. A basic bit of maths and you can see that their spread based offering is competitive and their Razor is tight spreads with decent enough comms to work out the cheapest. cTrader at Pepperstone has some "weird" pricing model for comms in that its 7 units per Lot round trip in the Base CCY... so 7£ if GBP pairs.. $7 for USD pairs etc.. A little quirky but I guess you can take advantage by choosing scalping pairs carefully if its really that important but on all pairs I calculated it came out cheaper than comms only (assuming sensible spreads at time)

Most of the brokers will give somekind of discount for volume and of course one day thats going to be something I care about..

In the end, I've went with PepperStone UK as they have a very competitive offering and excellent customer service, strong track record and regulatory oversight.
Their only real downside is lack of spreadbetting which of course may or may not matter in the long haul once I make my millions.
 

sven

Active Member
Define System Pipeline

Define System Pipeline
  • High level approach to Algo research, dev through to deployment. Timeframe/instruments etc. How to evaluate, discard, test deploy etc. How to monitor post live!!, how to integrate new systems into the portfolio.
  • Risk parameters and kill switch.
Well I have been giving the subject of system development some thought. On one hand its somewhat instinctive but of course on the other we need to treat this as a business. We need to try and avoid as many biases as we can.

For me at least a pipe really just a decision tree or flowchart that has criteria or rules of thumb that determine how we transition from one to the other.

Why bother?
"if its profitable trade it.. else just tweak it...etc"
-- said no one out loud.

  1. Procrastination for one. Man I'm amazing at it.
  2. Reducing Bias... I'll just tweak it.. optimise a little more...
  3. Moving toward the goal.
  4. Time management. Putting enough work into each facet of the process.
  5. Defining success or failure up front to allow us to think clearly when needed.

For fun (lol) I'll go into a little more detail on above.

If anyone has read The Lean Startup (@jack you may enjoy / feel free to add affiliate link if you do) then you know that the whole idea is to come up with millions of ideas and fail fast and fail early. Don't invest effort too early and just refine as the feedback tells you. Thats the idea here. Don't waste time in later stages.



Time Management
If you have a roadmap of how to get to your 'end' goal it is much easier to know how you are progressing. Its also important to understand where the time is going and where it should be going.

Value as we pass through the pipeline (when we do it correctly)

pmid.png

As we continue through our pipe we invest more and more time on the process. Its obvious that anything you spend time on you can lose objectivity. Painters for example after spending a while on a painting will often use a mirror to complete the final details. Sure they like carrying a mirror but it allows them to see things for the first time and with fresh eyes.

The longer you spend on something the more likely you are to invest mentally in the success of it. This is something software developers and the early days of RAD realised. If you show a user a fancy mocked screen they are reluctant to tell you it sucks and you don't want to hear it. If you sketch together you are happy to toss them in the bin.




Wastage of Effort(if we do it right)
pmid-inv.png

So that means in an ideal world we would throw aways the garbage early on and as we pass through the pipeline.. Each step forward is an investment in effort and we lose objectivity.



Speaking of Steps.. what are they?
Now this is deliberately vague and geared towards 'Model based systems' rather than datamining / ML etc.

Some notes...
  • Its entirely possible to add a million more steps and break each down to definite steps AND I intend on doing just that.
  • Its just a high level guess at what I think I need to do, Obviously its without any background or proven track record.
  • There is likely some iteration between stages but of course some things should not (out of sample validation back to optimisation for example)
  • There are hidden / not defined Criteria for moving to the next stage.. Like what the hell decides if its good enough to go to validation?
  • The colouring tries to represent the frequency of which I'd expect to make it through the steps.. Millions of concepts not so many deployments etc.
  • The general idea is to try and dedicate the appropriate amount of time to each stage.. instead of 90% on entry signals ;)


pipeline.png




So I'm treading a fine line between making progress and disappearing on an academic exercise defining the 'perfect' pipeline but thats how I roll. I need to formalise this else I will be dev/code/optimise-tastic

Ok.. Next step in a week or so's time for a Model Backlog. Define a backlog or ideas, concepts, prioritise them and then start investigating...

Feedback / criticism gratefully accepted as usual. I plan on refining as I learn stuff but now is a good time to tell me I've missed any key points/stages or I'm wasting my time.
 

Top