18

I'm building a small system that contains many parts and I want to use a message pub/sub service to communicate between parts.

I read about some message queue services like RabbitMQ and ZeroMQ but I feel they are too complicated and feel like it was born for distributed system. All parts of my system will be written in C++/Linux and place on a small Raspberry Pi CPU, so I dont need feature like scalable, cross-platform, other language clients ...

Can you guys give me some advice about services or libraries that fit my need?

iammilind
  • 68,093
  • 33
  • 169
  • 336
Yoshi
  • 563
  • 1
  • 6
  • 17
  • Do you need to use a lib or service? Or could you, given the prerequisites, just use sockets or pipes? That would be, likely, more efficient for a Raspberry. – Lorenzo Dematté Jan 22 '13 at 09:00
  • 1
    ZeroMQ is very simple to use. – Kimi Jan 22 '13 at 09:01
  • 1
    @dema80: I prefer service but lib is ok. Can you suggest any implement use sockets? – Yoshi Jan 22 '13 at 09:07
  • As other suggested, you can roll you own, using in process communication (if all components are in the same process, maybe using threads) or IPC, writing a server. It depends also on the framework/lib you want to use. If you are using QT, for example, using sockets is very easy and you can find a lot of examples directly in their docs. – Lorenzo Dematté Jan 22 '13 at 09:32
  • 1
    As an alternative... you may consider dbus http://www.freedesktop.org/wiki/Software/dbus – Lorenzo Dematté Jan 22 '13 at 09:33
  • 2
    ZeroMQ is not pub/sub - it is rather message oriented sockets. Take a look at Apache Qpid - it implements AMQP in C++, the code doesn't look bad. – Maxim Egorushkin Jan 22 '13 at 09:36
  • @MaximEgorushkin ZeroMQ implement pub/sub see http://learning-0mq-with-pyzmq.readthedocs.org/en/latest/pyzmq/patterns/pubsub.html or http://zguide.zeromq.org/php:chapter5 – mpromonet Jan 27 '15 at 19:42

3 Answers3

8

It's not that hard to do yourself actually.

First of all you need to define the protocol to be used. It can be very simple; like just a message type field, a payload size field, and the actual payload. The message types you need SUBSCRIBE, UNSUBSCRIBE and PUBLISH. The payload for the SUBSCRIBE and UNSUBSCRIBE messages is the name of a channel to subscribe to/unsubscribe from. The payload for the PUBLISH message is the channel name and the actual data (together with the size of the data of course).

To connect all subscribers you need a central server. All subscribers/publishers needs to connect to this server. The server program keeps a collection of queues, one for each channel. When a subscribe or publish message arrives to the server for a channel that doesn't exist, create a new message queue for this channel. For each channel the server also needs a collection of all clients subscribes to that channel. When a publish message arrives to the server, it's added to the end of the queue for the channel in question. While a channel queue is not empty, send a copy of it to all subscribers for that channel, and when all have received it then the message can be removed from the queue.

The hard part of the server is probably going to be the communication part. The easy part will be all queues and collections, as you can use the C++ standard containers for all of them (e.g. std::queue for the actual queue, std::unordered_map for channels, and std::vector for the collection of connected clients.)

The clients are very simple, all the need to do is being able to send the subscription and publish messages to the server, and receive the publish messages from the server. The hard part will once again be the actual communication part.

Postscript:

I've never actually built such a system my self, all of the above was just directly of the top of my head. An experienced programmer shouldn't need more than a couple of hours to implement the basics, maybe a couple of days for an inexperienced one.

For the communication you could use e.g. Boost ASIO, maybe use one threads per channel. And you can use something like Boost property tree to construct/parse JSON or XML messages.

However, all of this is kind of reinventing the wheel, when you could probably start using one of the existing systems like RabbitMQ in a couple of hours, saving you a lot of time (and a lot of bugs!)

Some programmer dude
  • 400,186
  • 35
  • 402
  • 621
  • 3
    I second the idea, but only if the "framework" you are going to build is super-simple and needs to be lightweight, otherwise.. do not reinvent the wheel! I built this kind of system for a past project, in .NET, in just a couple of days, everything included. It was fun, and was blazing fast. And since it was very economical when we later needed to scale out, we threw it away and replaced it without looking back. In C++, I would use Boost as well. – Lorenzo Dematté Jan 22 '13 at 09:39
  • What do you mean by a channel? In client server system, a client always initiate the connection. Do you mean, this connection should never be closed? – Kumar Roshan Mehta Mar 30 '18 at 00:06
  • The client here is meant as subscriber. – Kumar Roshan Mehta Mar 30 '18 at 00:13
7

As far as lightweight servers go, Redis supports pub/sub commands.

The Redis code itself is extremely tight (only a couple files), it's single-threaded (use an event-loop), and the memory consumption is quite low (compared to other Queing systems I have seen).

Matthieu M.
  • 287,565
  • 48
  • 449
  • 722
  • 1
    I really like Redis and already used it as a cache/store service on some private projects. Just curious if it fit on a small CPU like raspberry PI? – Yoshi Jan 22 '13 at 09:56
  • @Yoshi: compared to any of the other queuing system (ActiveMQ, RabbitMQ, HornetQ, ...) it is certainly more likely to fit. I would be more concerned with RAM, but you can always remove the Lua JIT etc... to get a lighter process. – Matthieu M. Jan 22 '13 at 10:10
  • It seems redis is only good for text-based messages. What about binary? – liuyanghejerry Feb 20 '13 at 10:14
  • @liuyanghejerry: it is agnostic toward keys and values alike, and treat the content as raw memory; so no issue. – Matthieu M. Feb 20 '13 at 10:43
3

I know it is late but may be useful for others. I implemented a basic pub/sub in C++ using boost.

CppPubSub

Usage is very simple. From one end publish your data (generic map) on a channel and other side subscribe for same channel and receive the generic map again.

// you should create a singleton object of NotificationService class, make it accessible throughout your application. 
INotificationService* pNotificationService = new NotificationService();

// Subscribe for the event.
function<NotificationHandler> fnNotificationHandler = bind(&SubscriberClass::NotificationHandlerFunction, this, std::placeholders::_1);
subscriptionToken = pNotificationService->Subscribe("TEST_CHANEL", fnNotificationHandler);

// Publish event
NotificationData _data;
_data["data1"] = "Hello";
_data["data2"] = "World";
pNotificationService->Publish("TEST_CHANEL", _data);

// Unsubscribe event.
pNotificationService->Unsubscribe(subscriptionToken);
Anil8753
  • 2,663
  • 4
  • 29
  • 40