Distributing Real-Time Market Data Using ZeroMQ / NFS?

Question

Suppose that you have a machine that gets fed with real-time stock prices from the exchange. These prices need to be transferred to 50 other machines in your network in the fastest possible way, so that each of them can run its own processing on the data.

What would be the best / fastest way to send the data over to the other 50 machines?

I am looking for a solution that would work on linux, using python as the programming language. Some ideas that I had are:

(1) Send it to the other machines using python's ZeroMQ module

(2) Save the data to a shared folder and have the 50 machines read it using NFS

Any other ideas?

score 1 · Answer 1 · answered Jul 01 '11 at 17:00

1

I'm pretty sure sending with ZeroMQ will be substantially quicker than saving and loading files.

There are other ways to send information over the network, such as raw sockets (lower level), AMQP implementations like RabbitMQ (more structured/complicated), HTTP requests/replies, and so on. ZeroMQ is a pretty good option, but it probably depends on your situation.

You could also look at frameworks for distributed computing, such as that in IPython.

answered Jul 01 '11 at 17:00

Thomas K

39,200
7
84
86

thank you. Do you know if `zeroMQ` is faster / slower than HTTP requests/replies? – user3262424 Jul 01 '11 at 18:25
faster... there is no need for http protocol in your case.. i would go with zeroMQ using pub/sub sockets – Felipe Cruz Jul 01 '11 at 18:45

score 1 · Accepted Answer · answered Jul 01 '11 at 18:49

1

I would go with zeromq with pub/sub sockets..

in your 2 option, your "clients" will have to refresh in order to get your file modifications.. like polling.. if you have some write error, you will have to handle this by hand, which won't be easy as well..

zeromq is simple, reliable and powerful.. i think that perfectly fit your case..

answered Jul 01 '11 at 18:49

Felipe Cruz

940
5
14

Cruz: which message pattern of `zeroMQ` should I use in this case? – user3262424 Jul 01 '11 at 22:50
start reading this: http://zguide.zeromq.org/page:all#Getting-the-Message-Out basically a pub/sub with subscribers filtering stocks they want – Felipe Cruz Jul 02 '11 at 01:47

score 0 · Answer 3 · answered Jul 10 '11 at 21:38

0

Definatly do NOT use the file system. ZeroMQ is a great solution wiht bindings in Py. I have some examples here: www.coastrd.com. Contact me if you need more help.

answered Jul 10 '11 at 21:38

Mike Trader

8,564
13
55
66

thank you. So `zeroMQ` is the way to go for distributing the data to the other machines? – user3262424 Jul 10 '11 at 22:37

Distributing Real-Time Market Data Using ZeroMQ / NFS?

3 Answers3