MT4 -> Multi-R sessions for tick-analysis

The Shared-Memory between multiple R sessions mentioned in my previous post got me thinking … quite some potential indeed. As a result, I investigated further using (calling) multiple R sessions from the same MT4 script. Specifically, I wanted to have a clearer understanding of the time required to performed lightning fast & dead slow processing, while being in the same MT4 script and without preventing real-time tick analysis (that is HFT potential at least on the receiving+processing side).

I drafted the following scenario to compare performances:

  1. Single R session called at every tick-update, displaying Bid/Ask @tick-update and using ChartSeries to plot either 500 bars or 10,000 bars – Bars are generated in the Init() function.
  2. 2 R sessions called sequentially, first R session displaying the Bid/Ask @tick-update, the second using ChartSeries to plot either 500 bars or 10,000 as previously
  3. 2 R sessions, but where the second R session is executed asynchronously, the first session plotting the Bid/Ask in @tick-update, the second using ChartSeries to plot 500bars/10,000bars.

The 10,000bars chart is used to further amplify potential anticipated observations made with the 500bars (that is a delay, or even a loss, of ticks due to the time required to “sequentially” render the chart). The two major area of concerns I had were the cooperation of 2 R sessions within the same script and the second the time required to perform the asynchronous call.

RESULTS are presented thereafter; y-axis is the number of ticks (GbpUsd) and x-axis is the time required to perform a complete MT4 Start-loop in [ms].

CONCLUSION: I have been very surprised by the behavior of the Asynchronous mode of MT4R. The cost of checking if an Asynchronous call is pending of finished prior re-issuing another Asynch call is about 1ms. This, added to the 1-2ms required to display Bid/Ask @tick-update depicted an extremely fast data processing system.

Although I’m not too happy about the set-up as I think a c++ multithreaded (or mpi) would have been better (through Rcpp or else), I can’t really discard the obvious that based on these simple tests, the R Asynch multisession set-up seems perfectly feasible for a Live Trading System. I even think I will push the limits to testing HFT trading, for which I will probably have to recompile Blas and Atlas anyways.

Finally, upcoming strategies should be design so as to separate real-time processing from asynch processing. As well, I did not yet evaluate the time required to send trade signals from R -> MT4, neither did I evaluate the Broker time to accept such signal (what’s the point of processing data at 2-5ms if trade signals are executed at 250ms)

1 with 500bars: .                   1 with 10,000bars:

————————————————————————————————————————————————————————————————————————————–

————————————————————————————————————————————————————————————————————————————–

2 with 500bars: 3 with 10,000bars:

Advertisements

Multithreading in R (or other types of non-sequencial programming)

Considering forwarding tick data from MT4 to R requires less than 2ms, but that charting 4 different time-frame (1min, 15min, 30min and 1hour) at each tick-update may require more than 250ms (depending on the number of bars in history), I think it is fundamental to investigate further in different types of non sequential programming and R, specifically multithreading or parallel programming.

Indeed, although charting is perfectly useless for our purpose of algorithmic trading, it is a valuable example to demonstrate a main loop that still receives tick updates at a rate of few milliseconds, while another “thread” would chart different time-frame every 1/2 seconds, without impacting the real-time gathering of ticks.

Along with the evaluation of MT4->R data transfer investigation, this is another major area to clarify prior stepping into the system design world (the fun part), considering our goal is to achieve a high-quality trading platform with Open Source Softwares.

Currently, I’m investigating:

  • Parallel computing in R through Rmpi, Snow. I found that particular example applied to optimizing an algorithmic system on backtested data at Revolutionalytics. This comforts me in saying that I’m probably following the right path. A list of High performance R computing can be found here.
  • Multi-threaded strategies through Rcpp
  • Shared-Memory between multiple R sessions.  The author mentioned the motivation of his example to be the developement of a High Frequency Trading systems. For that purpose, R nws package can be further investigated.
  • Again, one may also use socket connections through RServer.

On the Shared-Memory between multiple R sessions, I wanted to add that this strategy may support the development of multicurrency trading systems. Each currency would have its own Metatrader EA that at least reports the Bid/Ask continuously for that given PAIR, in a Shared-Matrix space …. actually quite a lot to think about but i shall not yet go that way 😉

Transfering MT4 quotes in R

The more I get into this, the more I adjust my objectives. Currently, I’m willing to make the most of the Quantmod and Blotter R-packages, while eventually building a MT4 equivalent of the InteractiveBroker-R_API. As a first step, I was willing to Chart ticks from MT4 in R using the Quantmod’s chartSeries()

There are different options to export ticks from MT4:

  1. Using the DDE – the major drawback, if I’m not mistaken, is this is an export-ONLY mechanism. Considering I’m willing to use R to generate the trade signals, the DDE is not an option.
  2. Using sockets on 127.0.0.0 – the major issue is the extra-work required to design the communication protocol. However, there are some interesting sockets ideas. I believe this is a similar, apparently efficient, COM/DCOM server design.
  3. Through a database interface such as MySQL.dll for MT4 – on the bright side, it allows for the easy management of tick data (using timestamps as main key), but I believe this would bring quite some extra-complexity and speed issue. Here is an example.
  4. Using an EA/Indicator with the R.dll
  5. Using an EA/Indicator with any [custom].dll

I have decided to follow the 5th alternative refered to as the MT4R interface library, specifically that proposed by 7bits on ForexFactory. 7bits’s interface starts Rterm.exe as a background process and communicate with its command line via stdin/stdout. Although I’m a little concerned by real-time efficiency (I read the following: For 1 minute bar trading, with limited computational overhead, R is a perfectly workable solution. Inside of that time-frame, things may get difficult), this seems a straight forward first-move, that has lots of benefits and may allow live-trading through R (providing one is not HFT or scalping).

Thereafter is a screenshot of my R-Chart from MT4 tick, along with a video of the overall performance regarding R-tick “real-time” performance (although my code is not yet designed for speed).

I have uploaded a video of both MT4 next to R-chart so as to have a sense of the R-charting delay: Megaupload link ( .swf playable in any browser). As you may notice, at some specific time R is lagging behind MT4. However, one should keep in mind that, at every tick, I pass the last 500 OHLC+Volume quotes from MT4 to R, whereas a more logical approach would simply append that last new tick. Nonetheless, this gives some insight in R’s performance once we’ll start performing real processing through R, to eventually lead to a trade signal.

From that first test, I was relatively concern by the ability to quickly send MT4 ticks to R in real-time. I decided to write the simplest transfer logger in that I simply send Bid/Ask prices @everytick from MT4 to R (assigning 2 double [variables] in R) while measuring execution time. Surprisingly I found out only 1-4 ms are required. Here is my logged-video Megaupload link. I quickly added a line to pass a Matrix[1000] that I initialized in the Init() that resulted in an average increase of 2ms (Megaupload link). I will further investigate different processing scenario to have a clear picture of MT4->R->MT4 live-trading set-ups.

I will be putting my simple code examples @ xmph.forex-googlecode.

“R” scripts for Metatrader ….. The first steps of many for proper trading

I have designed and developed Automated Trading Systems (ATS) for few years now. Since trading live, I have been fairly profitable (that is, I achieved my goals). However, I had the unsatisfactory feeling that I was missing some information to increase the level of trust in my systems … I was missing the proper tools to closely examine, and follow-up on the Real Equity Curve of an ATS.

I define the Real Equity Curve (REC) as the accrual of the

In-Sample backtesting Period + Out-Sample backtesting Period + Paper Traded Period + Live Trading Period.

There are quite a bit of information on the web for such but I did not find any Free Open Source tools to perform such assessment, specifically for Metatrader. Indeed, I do use MT4 for trading my strategies, and it’s fine with me; no issue nor any concerns. You might think otherwise but I will not go there … that’s far, far from my current objective.

Moving from one thread to another, I found this particular one: Synthetic hedges, cointegration, mean reversion and similar stuff (related to: MQL4->R-project – Interface Library).     Well, this was more than I needed to finally get a kick-start attempting to fill my needs for REC analysis.

I have started a GoogleCode SVN which contains my R scripts along with example files. My objectives are to design and implement the following:

  • Analysis of multiple MT4 StrategyTester [.htm] files … as I realized most, if not all, the software out-there are commercial solutions
  • Analysis and Follow-up of ATS through Real Equity Curve evaluation
  • Provide a detection for ATS performance Break-Down … to determine as soon as possible when things get ugly

In a second session, and because I stumble across few interesting information here and there, I will be looking at the following:

  • Strategy Optimization of MT4 ATS through R
  • Enhanced the analysis of ATS performance, through System Consistency evaluation for example
  • Enhanced the REC evaluation with Monte-Carlo analysis and Bayesian Statistics
  • and eventually combine the above to investigate Computer Generated ATS (CGATS).

As a first start, thereafter I present 2 examples of my programming kick-start: (1) The aggregation of multiple StrategyTester reports and, (2) the visual follow-up of an optimized MT4 Moving Average from backtesting to paper-trading. Every information and material needed can be found at http://code.google.com/p/rscripts4metatrader/

(1) Using: MT4_StratRep_Analyzer()

(2) Using: MT4_RealEquityCurve_Analysis ()

Metatrader Optimization on the Moving Average EA for the period 2008.01 -> 2009.01

3    2408.02    694    1.27    3.47    784.52    18.86%    MovingPeriod=60    Lots=0.1     MaximumRisk=0.02     DecreaseFactor=3     MovingShift=6