MT4 -> Multi-R sessions for tick-analysis

The Shared-Memory between multiple R sessions mentioned in my previous post got me thinking … quite some potential indeed. As a result, I investigated further using (calling) multiple R sessions from the same MT4 script. Specifically, I wanted to have a clearer understanding of the time required to performed lightning fast & dead slow processing, while being in the same MT4 script and without preventing real-time tick analysis (that is HFT potential at least on the receiving+processing side).

I drafted the following scenario to compare performances:

1. Single R session called at every tick-update, displaying Bid/Ask @tick-update and using ChartSeries to plot either 500 bars or 10,000 bars – Bars are generated in the Init() function.
2. 2 R sessions called sequentially, first R session displaying the Bid/Ask @tick-update, the second using ChartSeries to plot either 500 bars or 10,000 as previously
3. 2 R sessions, but where the second R session is executed asynchronously, the first session plotting the Bid/Ask in @tick-update, the second using ChartSeries to plot 500bars/10,000bars.

The 10,000bars chart is used to further amplify potential anticipated observations made with the 500bars (that is a delay, or even a loss, of ticks due to the time required to “sequentially” render the chart). The two major area of concerns I had were the cooperation of 2 R sessions within the same script and the second the time required to perform the asynchronous call.

RESULTS are presented thereafter; y-axis is the number of ticks (GbpUsd) and x-axis is the time required to perform a complete MT4 Start-loop in [ms].

CONCLUSION: I have been very surprised by the behavior of the Asynchronous mode of MT4R. The cost of checking if an Asynchronous call is pending of finished prior re-issuing another Asynch call is about 1ms. This, added to the 1-2ms required to display Bid/Ask @tick-update depicted an extremely fast data processing system.

Although I’m not too happy about the set-up as I think a c++ multithreaded (or mpi) would have been better (through Rcpp or else), I can’t really discard the obvious that based on these simple tests, the R Asynch multisession set-up seems perfectly feasible for a Live Trading System. I even think I will push the limits to testing HFT trading, for which I will probably have to recompile Blas and Atlas anyways.

Finally, upcoming strategies should be design so as to separate real-time processing from asynch processing. As well, I did not yet evaluate the time required to send trade signals from R -> MT4, neither did I evaluate the Broker time to accept such signal (what’s the point of processing data at 2-5ms if trade signals are executed at 250ms)

1 with 500bars: .                   1 with 10,000bars:

————————————————————————————————————————————————————————————————————————————–

————————————————————————————————————————————————————————————————————————————–

2 with 500bars: 3 with 10,000bars:

Multithreading in R (or other types of non-sequencial programming)

Considering forwarding tick data from MT4 to R requires less than 2ms, but that charting 4 different time-frame (1min, 15min, 30min and 1hour) at each tick-update may require more than 250ms (depending on the number of bars in history), I think it is fundamental to investigate further in different types of non-sequencial programming and R, specifically multithreading or parallel programming.

Indeed, although charting is perfectly useless for our purpose of algorithmic trading, it is a valuable example to demonstrate a main loop that still receives tick updates at a rate of few milliseconds, while another “thread” would chart different time-frame every 1/2 seconds, without impacting the real-time gathering of ticks.

Along with the evaluation of MT4->R data transfer investigation, this is another major area to clarify prior stepping into the system design world (the fun part), considering our goal is to achieve a high-quality trading platform with Open Source Softwares.

Currently, I’m investigating:

• Parallel computing in R through Rmpi, Snow. I found that particular example applied to optimizing an algorithmic system on backtested data at Revolutionalytics. This comforts me in saying that I’m probably following the right path. A list of High performance R computing can be found here.
• Multi-threaded strategies through Rcpp
• Shared-Memory between multiple R sessions.  The author mentioned the motivation of his example to be the developement of a High Frequency Trading systems. For that purpose, R nws package can be further investigated.
• Again, one may also use socket connections through RServer.

On the Shared-Memory between multiple R sessions, I wanted to add that this strategy may support the development of multicurrency trading systems. Each currency would have its own Metatrader EA that at least reports the Bid/Ask continuously for that given PAIR, in a Shared-Matrix space …. actually quite a lot to think about but i shall not yet go that way

Transfering MT4 quotes in R

The more I get into this, the more I adjust my objectives. Currently, I’m willing to make the most of the Quantmod and Blotter R-packages, while eventually building a MT4 equivalent of the InteractiveBroker-R_API. As a first step, I was willing to Chart ticks from MT4 in R using the Quantmod’s chartSeries(). 

There are different options to export ticks from MT4:

1. Using the DDE – the major drawback, if I’m not mistaken, is this is an export-ONLY mechanism. Considering I’m willing to use R to generate the trade signals, the DDE is not an option.
2. Using sockets on 127.0.0.0 – the major issue is the extra-work required to design the communication protocol. However, there are some interesting sockets ideas. I believe this is a similar, apparently efficient, COM/DCOM server design.
3. Through a database interface such as MySQL.dll for MT4 – on the bright side, it allows for the easy management of tick data (using timestamp as main key), but I believe this would bring quite some extra-complexity and speed issue. Here is an example.
4. Using an EA/Indicator with the R.dll
5. Using an EA/Indicator with any [custom].dll

I have decided to follow the 5th alternative refered to as the MT4R interface library, specifically that proposed by 7bits on ForexFactory. 7bits’s interface starts Rterm.exe as a background process and communicate with its command line via stdin/stdout. Although I’m a little concerned by real-time efficiency (I read the following: For 1 minute bar trading, with limited computational overhead, R is a perfectly workable solution. Inside of that time-frame, things may get difficult), this seems a straight forward first-move, that has lots of benefits and may allow live-trading through R (providing one is not HFT or scalping).

Thereafter is a screenshot of my R-Chart from MT4 tick, along with a video of the overall performance regarding R-tick “real-time” performance (although my code is not yet designed for speed).

I have uploaded a video of both MT4 next to R-chart so as to have a sense of the R-charting delay: Megaupload link ( .swf playable in any browser). As you may notice, at some specific time R is lagging behind MT4. However, one should keep in mind that, at every tick, I pass the last 500 OHLC+Volume quotes from MT4 to R, whereas a more logical approach would simply append that last new tick. Nonetheless, this gives some insight in R’s performance once we’ll start performing real processing through R, to eventually lead to a trade signal.

From that first test, I was relatively concern by the ability to quickly send MT4 ticks to R in real-time. I decided to write the simplest transfer logger in that I simply send Bid/Ask prices @everytick from MT4 to R (assigning 2 double [variables] in R) while measuring execution time. Surprisingly I found out only 1-4 ms are required. Here is my logged-video Megaupload link. I quickly added a line to pass a Matrix[1000] that I initialized in the Init() that resulted in an average increase of 2ms (Megaupload link). I will further investigate different processing scenario to have a clear picture of MT4->R->MT4 live-trading set-ups.

I will be putting my simple code examples @ xmph.forex-googlecode.

Enhancing R Graphic packages

This will be the last thing I’ll post prior starting to connect R with Metatrader. Indeed, as you may suspect already I use this blog as my lab-book. The following might be quite general, but it fits in my overall ambition to build a personal quant-trading architecture as professional as I can.

Graphic visualization is a major part of a trading system design. To me, nice & clean graphs are as important as their content. One needs to appreciate looking at the information to understand it, or at least to spend some time trying !

Here are the few framework I found that could interact with R:

• Tulip is an advanced visualization framework that may support the development of interactive charts
• Stanford Protoviz through R-Web-Vis, also for Web deployment (demo site)
• Jung, a java interactive graph rendering framework
• GGplot2 is a static graphic library more polished than the original R-one
• Graphviz

Finally, few web resources for graph examples:

• R-Climate
• RGraphics exampels from Auckland univ.
• and the famous: R-addictedtor

Algorithmic Trading in R: Available Information

After digging through different blogs and info on the web I found an amazingly strong and developping community supporting R for specific applications in algorithmic trading. Thereafter I present some of the major information that I believe are essential to the novice R algorthmic trader, hoping this will reduce the learning curves of the few following that path:

• Designing & Backtesting strategies: Modified Donchian Channel – Most example codes I found are available in my Code Examples section;
• Developping Indicators and Market scanners: Basic Indicators in R, Cointegration analysis, Cointegration with R through Metatrader, Prediction Analysis ;
• Live trading: InteractiveBroker API for R ;
• Mailling lists & User groups: R-Sig-Finance, RMetrics and Rmetrics user list, R-bloggers, inside-R, RinFinance, Revolutionanalytics in Finance ;
• and Few talks worth watching: UCLA talks, UCLA talks2, Jim Simons from Renaisssance ;
• R libraries specific to trading & financial analysis:

1. QuantMod: development, testing, and deployment,
2. TTR: technical trading rules,
3. PerformanceAnalytics: econometric functions for performance and risk analysis of financial instruments or portfolios,
4. XTS: extensible time-series,
5. Blotter/QuantStrat: Transaction-oriented infrastructure for defining instruments, transactions, portfolios and accounts for trading systems and simulatio – build, and back-test strategies
6. R-QuantLib: R interface to the QuantLib library
7. CRAN Finance & Econometrics packages

A nice summary can be found here.

Now I’m early in my project, which aims at using R to provide trading-signals to a Metatrader platform, but there is a specific information that I’m not yet able to find: Is R efficient enough (fast enough if you will) to allow live trading on a per-tick basis, per-second basis or per-? basis ? What is the ideal architecture to perform live trades: asynchrone R calculation for major analysis, combined to c++ functions and/or Metatrader functions to closely watch market opportunities? I was not able to find anyone detailing his/her Live trading set-up involving R which is a pity as a lack of design would result in basically restarting from scratch or living with inefficient set-ups. I guess I will have to experience the trial & error investigation.