Python Development Environment / Jack's Technology Stack

jack

Administrator
Staff member
Could you detail the advantages to using Python over MQL.
(sorry for the late reply, I didn't see your message for some reason.)

With anything, there's trade-offs.

When using Python, you gain access to every library available to python to use with your project. Want a web interface? Done.. want a GUI for custom controls? Done... want to use the most widely supported and accessible machine learning libraries and tools? Done... want to integrate 3rd party data into your trade decisions? Easily done!..

The drawback is, should your broker not already provide a Restful API or Python native API, you'll likely have to write some sort of bridge code yourself (such as the agents I describe to connect to MT5.) It's a bit clunky, and it takes a little effort, but it works.

Other benefits include writing code once that will be easily ported to new platforms. If you do something advanced in MT5 (such as using machine learning,) it's gonna be a real pain to port it outside of MT5 later on should you ever use a different broker who doesn't support MT5. For instance, many decent traders "grow up" to using a broker like Interactive Brokers, who doesn't offer MT4/5 at all (and before anyone tries to link to some 3rd party tool or bridge service.. they are horrendous for use with Interactive Brokers.)

Or say you want to get into futures with existing strategies you have running in MT5... that limits yourself to maybe 1 major broker and FCM out there and their offering, while fine, is limited. Having your code written in a language that's generic and only requiring you to connect the dots to a new broker's API means it's a lot easier to port to one of many decent futures brokers should you ever want to make this switch.

You can build in some fault tolerance... setup a heartbeat monitor on your primary instance and have some sort of risk and order manager kick in should your main machine go down... MT5 doesn't do anything like this.

Finally... another big advantage I noticed was running 'light'... often I can run dedicated python code on a micro server (without a graphical interface) in Amazon's cloud... you might run a total bill of $2-3 bucks a month to power multiple strategies... You simply need a beefier machine if you're gonna run MT5 and a graphical interface, typically costing 10x as much.

Between all the 3rd party libraries available for python, and all the nifty setups you can do, it really comes down to flexibility.
 

sven

Well-Known Member
@jack I don't see any mention of 'backtesting' algos. Given you are using a hybrid MT/python setup how do you backtest algos?

I get there are limitations to historical testing and many don't hold a great deal of weight in it but its safe to say you need some comfort there is an edge(albeit transient for most)

I'm sure there are different needs for book based algos etc but as a general approach...
Do you backtest?
if so, how?

I do like forward testing but there is a place for backtests. curious.
ta
 

jack

Administrator
Staff member
@jack I don't see any mention of 'backtesting' algos. Given you are using a hybrid MT/python setup how do you backtest algos?

I get there are limitations to historical testing and many don't hold a great deal of weight in it but its safe to say you need some comfort there is an edge(albeit transient for most)

I'm sure there are different needs for book based algos etc but as a general approach...
Do you backtest?
if so, how?

I do like forward testing but there is a place for backtests. curious.
ta
That's a good question. I have a few approaches but the issue isn't quite backtesting on it's own, but what exactly you want to prove with said backtest.

For time series analysis (which is not my bread and butter but likely where most people start in the algo trading world,) you could use something like MT4/5, MultiCharts, or NiinjaTrader to test and prove trading ideas. However, the downside of time series analysis are rampant though:

  • Candlesticks are painted based off either ticks (time of sales, transaction records,) or in MT4's case, the bid price. Both are narrow slices of the total picture of what's happening in the market at a given point of time. You do not see where people have resting orders
  • Candlesticks / bar data is also VERY machine readable, and very accessible to market participants. Meaning if RSI + MACD with any given settings could yield a profit and prove alpha, you better believe some computer has already chewed through every combination and the results are known and/or already crushed out.
  • Candlesticks / bar data is also very low resolution.. it's just Open, High, Low, Close (and maybe Volume,) within a given period of time. 4 or 5 data points to summarize what happened over 60 seconds. Look at 1 minute of the SPY ETF's TOS (tick data, transaction records, time of sales, etc..) data to realize the volume of information being devolved into just 5 float numbers.

A lot of my strategies depend on things like knowing where the spread / L1 / NBBO prices are compared to where the security is trading actively (transactions,) which cannot be tested on platforms like MT4... or, for that matter, cannot be traded on MT4 since anything dealt to you over MT4 is on a made market and no one is going to cross the spread and hit your resting order like you'd find on a proper exchange. But I digress..

So... to overcome this.. I use a bit of a hybrid between using MT5 / NinjaTrader to 'replay' data via their strategy tester features (which load my code as indicators and pipe out the market replay data to my algo platform, which then makes trading decisions and communicates back, and some 3rd party tools like BuildAlpha for quick vetting of simple ideas. The market replay method is slow... very slow... so I'll sometimes prototype an idea in NT8 or MT5 directly to vet it should I want to run a test across more than a few months worth of data at a time, or I'll use BuildAlpha with custom imported data should the idea rely on just simple time series analysis.

I'd like to avoid writing a backtesting engine directly into my algo platform because this would be hella time consuming and I'd feel like I'm reinventing the wheel. I can see such functionality being helpful, but I can also use other platform to do this for me without the time commitment to coding it myself.

As a side note: I also trust forward testing a lot more than back testing, plus I'm actively watching the market, order books, transaction feeds, etc.. while the bots are working with small risk in live, or in sim. That way the feedback I get is FAR more useful than just straight P/L numbers.

Anyhoo.. hope that answers your question.
 

sven

Well-Known Member
I'd like to avoid writing a backtesting engine directly into my algo platform because this would be hella time consuming and I'd feel like I'm reinventing the wheel.
Phew I thought you were going to say you built a full blown backtesting framework and I was going to have to leave the room ;)


I guess due to the markets I'm limited in I'm going to have to focus on timeseries and hope I can exploit some mediocre feature. Would not know where to begin on L1 and such.. one for a little further down my journey I suspect.

thanks again.
 

rod178

Well-Known Member
Jack et al

Have you tried Deep learning with Python?

refer
Francois Chollet - Deep Learning with Python-Manning Publications (2017)
 

jack

Administrator
Staff member
Jack et al

Have you tried Deep learning with Python?

refer
Francois Chollet - Deep Learning with Python-Manning Publications (2017)
I haven't gone as far as deep learning yet... I've done a bit of work around random forest regression, decision tree algos, reinforcement learning, and ensembles / combos of them.

I'll look into your source on deep learning.
 

rod178

Well-Known Member
Jack

Are you running both Python and MT5 under Linux?
Are you running on a VPS or locally?
 

jack

Administrator
Staff member
Are you running both Python and MT5 under Linux?
MT5 with my ZeroMQ powered agent/connector running as an EA: It works both on Windows and Linux, but I primarily use Windows right now. In the near future I'll be migrating some tasks that involve MT4/5 and my platform to a light VPS and that will likely be Linux.

My platform in Python can run under Linux and Windows.

That said, I can also setup a Windows machine to run MT5, then connect to the agents on MT5 over the network from a Linux machine running my platform's client. Nothing has to be running on the same box.. It's kinda nifty.

A big reason I adopted ZeroMQ when designing the foundation of my platform was the ability to spin out services to other machines on the network if needed. A simple switch from inproc to TCP sockets is just one line of code to change. Heck.. one of the bigger changes I made recently was to make a log service on a different box so none of the clients processing strategies would be bothered with disc/IO operations, and instead of having logs on each local machine it would just be collected and stored in one place (with the exception of the code crashing, which dumps everything locally too.)

Are you running on a VPS or locally?
Locally, but I'm about to put a few tasks/strats in the cloud on a linux VPS... Starting with the stuff I do in crypto since it's a 24/7 market.
 

rod178

Well-Known Member
MT5 with my ZeroMQ powered agent/connector running as an EA: It works both on Windows and Linux, but I primarily use Windows right now. In the near future I'll be migrating some tasks that involve MT4/5 and my platform to a light VPS and that will likely be Linux.

My platform in Python can run under Linux and Windows.

That said, I can also setup a Windows machine to run MT5, then connect to the agents on MT5 over the network from a Linux machine running my platform's client. Nothing has to be running on the same box.. It's kinda nifty.

A big reason I adopted ZeroMQ when designing the foundation of my platform was the ability to spin out services to other machines on the network if needed. A simple switch from inproc to TCP sockets is just one line of code to change. Heck.. one of the bigger changes I made recently was to make a log service on a different box so none of the clients processing strategies would be bothered with disc/IO operations, and instead of having logs on each local machine it would just be collected and stored in one place (with the exception of the code crashing, which dumps everything locally too.)

Locally, but I'm about to put a few tasks/strats in the cloud on a linux VPS... Starting with the stuff I do in crypto since it's a 24/7 market.

You seem to be doing very much what I am planning, so remain interested on your progress,
I am currently running MT4 on both Win7 and Wine(under Mint).

Still have not implemented ZMQ and still feel more comfortable with C, MQL and Ruby than Python, all about available time.

Eventually I would prefer to dump Windows completely and run all on a Linux VPS.
On a VPS, would it be preferable to run MT4 on a Win Virtual Machine or run under Wine.

Initially I would prefer to run both ZMQ and MT4 on the same Linux VPS, do you see any issues with this?

Eventually I would like to concurrently run the three as Servers (VPS) via one ZQM Client(VPS)

My EA runs off H1 and Trades about twelve times a month an each Major Pair although currently I restrict to one concurrent pair and one Broker
Currently I use three Brokers, AmiB, ICM and PepperS.

Have you considered using cTrader as your Server?
 

jack

Administrator
Staff member
You seem to be doing very much what I am planning, so remain interested on your progress,
I am currently running MT4 on both Win7 and Wine(under Mint).

Still have not implemented ZMQ and still feel more comfortable with C, MQL and Ruby than Python, all about available time.
ZeroMQ has C and Ruby bindings as well. It's just a messaging library that's wrapped or ported to various languages. The idea is, if you're more comfortable in C, you can write code that works with ZMQ and communicates in a way that my Python code can interface with via ZMQ as well. In my case, I have some code written in MQL5 (the connecting agents running on MT5,) some in C# (the connecting agents for NinjaTrader 8,) some in GoLang (part of my logging server,) and a lot in Python... all of which works together as one 'system' via ZeroMQ.

Eventually I would prefer to dump Windows completely and run all on a Linux VPS.
On a VPS, would it be preferable to run MT4 on a Win Virtual Machine or run under Wine.

Initially I would prefer to run both ZMQ and MT4 on the same Linux VPS, do you see any issues with this?

Eventually I would like to concurrently run the three as Servers (VPS) via one ZQM Client(VPS)
It will work this way, yes.. I had it working myself and plan on going this route for a few jobs/tasks as per my last message.

See here too: https://github.com/dingmaotu/mql-zmq
Go down to the files and install part and read the: "Note for WINE users" section.

My EA runs off H1 and Trades about twelve times a month an each Major Pair although currently I restrict to one concurrent pair and one Broker
Currently I use three Brokers, AmiB, ICM and PepperS.

Have you considered using cTrader as your Server?
I have used cTrader in the past with Pepperstone. However, Pepper stopped supporting Canadian clients, so while I wish I could still use them (as they are a very solid broker,) I don't believe there is a cTrader broker I'd consider using that also supports residents of Canada. Thus, I focused on other platforms.
 
What version of python 3 are you using? Have you looked into 3.8+ (the latest), as it allows for easily writing async functions, which will obviously help with I/O, plus the utilization of multiprocessing for parallelism. Also, do you use any memory efficiency libraries such as Struct or memory views (pointer logic) for ensuring no data is unnecessarily memory greedy? The combination of these tools allows for hyper efficient server usage.

I’m also curious how frequently your strategies are trading. I’ve found for myself, that I prefer less frequent, longer term buy/hold + options strategies for investing, which require much less of a robust system.

Also, last question, for the sake of speed have you tried to locate the servers of the brokers you’re placing trades with to ensure that whatever servers you’re trading from are as close as possible, to reduce latency?
 
Thought of another question. Does your system make use of any proxies or IP rotation to prevent potential blocks? Is that necessary in your case? I suppose if you’re using an API to trade, it’s already defined rates.
 

jack

Administrator
Staff member
What version of python 3 are you using? Have you looked into 3.8+ (the latest), as it allows for easily writing async functions, which will obviously help with I/O, plus the utilization of multiprocessing for parallelism. Also, do you use any memory efficiency libraries such as Struct or memory views (pointer logic) for ensuring no data is unnecessarily memory greedy? The combination of these tools allows for hyper efficient server usage.
I just upgraded to 3.8.0 a few months ago. Nothing major changed in my code for this, I was just waiting for all the libraries I used to release something stable on 3.8.

A new thing I'm experimenting with now is running python embedded within my project. Like, literally putting the whole python environment in a subdirectory of my application so all dependencies and required files just come along with the application. The only setback here is managing my algo stack this way becomes platform dependent if I want to use the embedded python along with it...

Asyncio is for threading and event loops, not multiprocessing. A new thread != a new process. That said, I don't really use the programming pattern that Async pushes because of how I use ZMQ. ZMQ supports Asyncio through a module they provide, but really it's just so you can code with Async style syntax / pattern and not required to use ZMQ for event processing. Soooo.... **shrug** (I do like the pattern Asnyc uses, and Twisted kinda has something like it as well with how they decorate functions.. but I'm not going to rewrite what I already have since functionally it does the same thing.)

That said, regarding the comment you made about Async and multiprocessing: Async is for threading and event loops, not multiprocessing. That's a different beast entirely.

And no, I'm not looking at struts or memory management libraries yet... I'm not nearly hitting any memory limits while running my algo platform. I can't say the same for the Pandas + Excel + database work I'm doing right now as part of my idea generation pipeline.. lol..

I’m also curious how frequently your strategies are trading. I’ve found for myself, that I prefer less frequent, longer term buy/hold + options strategies for investing, which require much less of a robust system.
I have some FX stuff that trades a few times a month. I have some equities stuff that range from 600-2800 transactions a day depending on various factors.. though I don't run such strats every day.

One BIG mistake smart people make when starting with algo trading is to blindly follow their "preference" in trading style. They let their inexperienced opinion and ego dictate where and what they should trade. Instead, the data and any signs of traction should be proving where their time needs to be invested. If that's on daily data, cool, but it just as well might be found in order book data on a market that no one is even talking about online. Just saying.

Also, last question, for the sake of speed have you tried to locate the servers of the brokers you’re placing trades with to ensure that whatever servers you’re trading from are as close as possible, to reduce latency?
I do not target ideas where sub 30 milliseconds makes the difference between profit and loss. I'm just not interested in playing that speed game.

But that said, yes, I do have a few things I'm running now that are in the cloud and closer to the execution server they interface with. Depending on the system I'm referring to that's already in place in the cloud, latency varies between 1ms and 5ms away. None of them would be much worse off if they were 30-60ms away... being in the cloud is more about keeping a consistent connection and resiliency.

Hope that answers your questions. Cheers!
 

jack

Administrator
Staff member
Thought of another question. Does your system make use of any proxies or IP rotation to prevent potential blocks? Is that necessary in your case? I suppose if you’re using an API to trade, it’s already defined rates.
Zero reason to ever need to do this if you're a good actor on the exchange and following any compliance rules set by your broker. So I'm really tilting my head here on why you're asking.

Let me ask you: Why do you think you'd get "blocked" for your actions?

Or do you mean things like polling rate limits? That's usually on an account level, not controlled on the IP level.. so maybe there's a concept here you might be conflating with something else?
 

rod178

Well-Known Member
Zero reason to ever need to do this if you're a good actor on the exchange and following any compliance rules set by your broker. So I'm really tilting my head here on why you're asking.

Let me ask you: Why do you think you'd get "blocked" for your actions?

Or do you mean things like polling rate limits? That's usually on an account level, not controlled on the IP level.. so maybe there's a concept here you might be conflating with something else?
at least it gave me a perplexed laugh
 

jack

Administrator
Staff member
Hi Jack,

Is your python code open-source for adding further extensions?
It is modular so I can add new sources of data and brokers as needed.

However, if you're asking if the code itself is opensource and public, the answer is no. It's not available to the public and not for sale.
 

kramu

New Member
Hi Jack,

No worries, I can understand. I was planning on doing something similar as I had to keep with multiple API's for different brokers.
Do you think you can share have you abstracted the incoming feed? (If you have any reference for incoming abstract.py it would be useful, it can avoid me potential rewrite of API multiple times)
 
Top