How Scalable is ZeroMQ?

Question

How scalable is ZeroMQ? I'm especially interested in understanding its potential for running on a large number (10,000 - 15,000) of cores.

http://stackoverflow.com/questions/8361738/is-zeromq-really-reliable-and-scalable — Mitch Wheat, Feb 14 '12 at 23:53

score 10 · Accepted Answer · answered Feb 15 '12 at 01:00

10

We've tried to make it as scalable as possible but I personally tested only on up to 16 core boxes. Up to that limit we've seen almost linear scaling.

answered Feb 15 '12 at 01:00

sustrik

144
1
2

score 10 · Answer 2 · answered Feb 15 '12 at 18:59

You don't mention whether your 10k or 15k cores are on the same box or not.

Let's assume they are. Every two years the number of cores on a box can, theoretically, double. So if we have 16-core boxes today, it'll be 16K cores in 20 years.

So now, your question is maybe, "will ZeroMQ help my application scale to such huge numbers of cores, so that it will scale over the next 20+ years?" The answer is "yes, but only if you use it properly". This means designing your application using inproc sockets and patterns that properly divide the work, and flow of data. You will need to adjust the architecture over time.

If your question is, "can I profitably use that many cores between multiple applications", the answer lies with your O/S more than ZeroMQ. Can your I/O layer handle the load? Probably, yes.

And if your question is, "can I use ZeroMQ across a cloud of 10K-16K boxes", then the answer is "yes, this has already been proven in practice".

Thanks Pieter, I do mean cores spread across multiple boxes. Do you know of any papers that discuss this level of scalability, or any reference sites of this size? — Mark Harrison, Feb 15 '12 at 20:20
So even with OpenAMQ we scaled in production to 3K-4K boxes (each multiple cores), talking to a single broker. This is the basic level of scaling of any 1-N flow in 0MQ: limited only by the number of sockets your OS can handle. — Pieter Hintjens, Feb 16 '12 at 21:38

score 6 · Answer 3 · answered Feb 15 '12 at 01:41

Note that although ZeroMQ is multithreaded internally, it may not be wise to rely solely on that to scale it up to large numbers of cores. However, because ZeroMQ uses the same API for inter-machine, inter-process and inter-thread communication, it is easy to write application using ZeroMQ that can move seamlessly into a one-process-per-core scenario or into a grid fabric of many, many machines.

ZeroMQ already has a reputation for being the fastest structured messaging protocol around so if you were going to do benchmarks to choose a technology, ZeroMQ should definitely be one of them.

The two big reasons for using ZeroMQ are its easy-to-use cross-language API (see all the examples on the ZeroMQ Guide site) and its low overhead both in terms of bytes on the wire, and in terms of latency. For instance ZeroMQ can leverage UDP multicast to run faster than any TCP protocol, but the application programmer doesn't need to learn a new API. It is all included.

How Scalable is ZeroMQ?

3 Answers3