1

I'm trying to implement a live trading tick data streaming from a MT4 terminal to a Postgres database. To accomplish this, I'm using dwx-zeromq-connector and I had success when doing this for 2 currency pairs.

The way the connector works is by setting a pub/sub ports and stream data from MT4 using a python script. Now I want to stream 28 currencies, so the number of charts in the MT4 terminal is 28 and the problems of slow subscribers feels very heavy. The messages queue up and no matter what I try, it does not seems to process them any faster.

What I have tried so far is:

  • 28 different sub sockets for all pub ports using multiprocessing and daemon threads (not at the same time)
  • 1 socket for all 28 pub ports using multiprocessing and daemon threads (not at the same time)
  • 28 different classes running 1 daemon thread each, one per pub
  • 1 class having 7 different threads and each thread has 1 socket that manages 4 pub ports

None of them improves the performance. Now, looking at this zmq guide, the Black Box Pattern seems to be suitted for what I need. However, I was wondering if I should reduce the amount of pub ports by making that the publishers publish information to only 1 pub socket.

So, according to the zmq guide Black Box Pattern, I guess that I need to use only 1 publisher and then push the messages to worker, right? Based on all this, is it possible that many publishers publish messages to only one port using tcp protocols? And is the zmq guide what I should really use or I'm missing something? The specs of the VPS I'm using is 2 vCPU and 12 GB RAM.

If any code is needed I can post it gladly.

EDIT 1:

This is the OnTick function, as requested:

void OnTick()
  {
   /*
      Use this OnTick() function to send market data to subscribed client.
   */

   if(CheckServerStatus() == true)
     {
      if(Publish_MarketData == true)
        {
         //for(int s = 0; s < ArraySize(Publish_Symbols); s++)
         //  {
         string _tick = GetBidAsk(Publish_Symbols);
         if(print_pub_data)
           {
            Print("Sending " + Publish_Symbols + " " + _tick + " to PUB Socket");
           }


         InformPullClient(pubSocket, StringFormat("%s %s", Publish_Symbols, _tick));
         //}
        }
     }
  }
xerac
  • 147
  • 8
  • 1
    How did you test the problem was on python side, whereas MT4 Terminal was not the performance blocker having 28 MT4-GUI-Graphs? How do you inject the data in MQL4? A script-based injection or an EA-based event-driven, blocking OnTick(){...} event-loop data injection? – user3666197 Dec 23 '20 at 06:30
  • 1
    Having used MT4 / ZeroMQ integration for about 12+ years ( yes, indeed ages before Darwinex got merely Est.'d ) there are ways to go ahead & improve the performance / latency, it would be also nice to post a hwloc / lstopo ( binaries available from OpenMPI site https://www.open-mpi.org/software/hwloc/v2.4/ ) output for the VPS-hosted virtual machine, to see the actual skeleton X-ray'd :o) that was delivered to you. – user3666197 Dec 23 '20 at 09:05
  • Hi @user3666197. I tested it by running the same 28 charts and having all the 28 EA's connected to each port, but just subscribing to 1 currency and then increasing the amount of subscribers until it "lags" and messages starts to queue up. When doing this, I found that if I use 1 subscriber up to 3-4 its okay. Actually, from the 4th subscriber and beyond, the problem starts arising. – xerac Dec 23 '20 at 09:31
  • I'm curious how you are still using ZMQ and MT4 and if you do, the way you do it. What was your experience and if you are doing (or did) tick data streaming, how did you overcome this problem (if you ever encounter it). – xerac Dec 23 '20 at 09:32
  • 2
    Always welcome @xeras Would you mind to complete the post with OnTick(){...} code to see your setup configuration details. Yes, I use the ZeroMQ in production inside a low-latency loop, shaving down all possible [us], to keep the E2E response time under about ~80 [us] for data exchange, leaving reasonable amounts of time for ML/AI-predictive engine to respond to any new QUOTE-arrival ( per-tick ML-predictions in real-time ). – user3666197 Dec 23 '20 at 10:26
  • Thank you, @user3666197. Your last comment sounds like what I'm trying to build and really excites me. Sure, I'll post the OnTick, but its no different from the one provided in the `dwx-zmq-connector`. I'll post it in a while. – xerac Dec 24 '20 at 12:29

1 Answers1

1

Q : "...is it possible that many publishers publish messages to only one port using tcp protocols?"

Yes, it is, yet given a few details posted above, this is rather an anti-pattern, introducing many risks a live-streaming solution ought reduce, not increase.

How to do this safer ?

  • ignore the "ready-made" connector :
    so as to avoid all there present overheads you simply cannot control & handle in volume

  • refactor the data-flow :
    a performance motivated data-flow system ought reduce to minimum any sort of data re-wrapping, here the MQL4-code may & should for minimum latency (to avoid blind-spots of not seeing a next QUOTE-arrival due to not having yet finished the OnTick(){...}-blocking code-block - MQL4-code can self-diagnose its waiting times at about a microsecond resolution). For ultimate performance / minimum further L3-transport-layer latency & ZeroMQ buffer-sizings all QUOTE-data ( Ask, Bid, Broker-side last known server Time )-values ought be rather binary-mapped to a reasonably wide bitmap to be carried over any protocol/wire (attempts to send JSON(!)-decorated text representations of these data is an awful sin per-se, never do this in production, the less in an attempt to run more than 1+28+1+28 MQL4 & ZeroMQ threads on just a 2 vCPU device. Details about the "stolen" CPU-ticks for a VM hosted on VPS are no less important, yet are way beyond the footprint of this post)

  • test all Zero-Copy, protocol-less Transport-Class use and buffer-overflow avoidance possible :
    ZeroMQ PUB/SUB Scalable Formal Communication Pattern Archetype is not the best choice here. Depending on the ZeroMQ API version, some perform TOPIC-filtering on the PUB-side ( soon overloading the PUB-(MQL4) side at no positive value added for doing that ) or on the SUB-side(s) ( thus overloading the SUB-side node, be it on the same VPS/VM node or distributed elsewhere, again at only a negative value added for
    doing that ). Reimagine the processing so as to ultra-fast off-load the sending side's OnTick(){...}-sending sprint, using PUSH/PULL or even PAIR/PAIR, best used on a protocol-less inproc:// Transport-Class, if host/hosting O/S permit these to operate error-free. That would yield you a Zero-Copy, Protocol-less, Zero-re-Buffering collection of all QUOTE-data into one MT4-Terminal service-graph, from which it can operate at maximum performance, minimum latency to collect indeed fast all QUOTE-data in a fashion, that off-loads those many respective currency-specific OnTick(){...}-located senders.

  • refactor the code for performance boost by a separation of concerns :
    the previous trick to ultra-fast collect all QUOTE-data inside a separated MQL4-code permits to add more services to the "Collector", where performance ceiling permits to do so ( we use a remote-CLI ( keyboard+terminal / scriptable-CLI ) so as to control this MT4-hosted service-process using a user-defined command-language, distributed-logging to offload any shave-able latency off the MT4-Terminal node, and a few more )

  • scale the processing capability :
    the ZeroMQ messaging layer has several strong performance boosters, yet here you primarily need to coordinate this with the VPS-provider not to neglect the actual (there hosted) VM-host, under which the MT4-Terminal operates. Off-loading may leave but the MT4-Terminal process running inside this VM, having the "rest" of the data-flow processing somewhere else, not impacting the current VM having but 2 vCPU-cores (co-shared with any other VPS-hosted VMs, bearing thus any & all adverse effects thereof - vRAM Virtual Memory swapping, co-shared vCPU-core cache areas devastations, co-shared vCPU-core thermal hopping injected from "other / neighbouring" workloads co-hosted on the same physical VPS-node )

ZeroMQ tooling has started as an AMQP / FinTech lingua franca / replacement, where Pieter Hintjens & Martin Sustrik's teammates have designed a luxurious messaging/signaling framework for our small-scale algo-trading deployments.

user3666197
  • 1
  • 6
  • 50
  • 92
  • Hi. So, if I understood this well, despite the fact its possible, it's not really recommended, right? An approach of creating a subscriber that subscribe to 28individual pub sockets should be a better approach, right? – xerac Dec 25 '20 at 23:35
  • 1
    Thank you. So, what is VPS-provider please? – Avv Oct 04 '22 at 20:55