4

There are many open source Message queues for Linux, and I need some help deciding what I should go for.

My problem is simple - I get sent a list of files that needs to be processed. Each job can't be split up, but they are self contained and can be spread to multiple computers.

I'm thinking of solving this using a message queue. Multiple clients send a message to a central queue. Each queue has a number of subscribers that will take jobs from that queue when they have finished processing the current job.

Ideally it should have the following qualities

  • Message queue must be able to store unprocessed messages in case of a shutdown/reboot
  • A job can only be processed by a single subscriber (don't want duplicate jobs)
  • The subscribers should be able to send jobs of their own, that will be processed by a different set of subscribers.

Can anyone suggest a simple to use message queue?

6 Answers6

1

You have RabbitMQ and ZeroMQ, but afaik ZeroMQ doesn't store unprocessed messages in case of a crash. They're both open source and use AMQP, an open messaging protocol.

Marco Ramos
  • 3,120
  • 23
  • 25
  • ZeroMQ does not use AMQP - though it is a networking library written by the same people that designed and wrote the AMQP protocol and first implementation of it. – Alister Bulman Jun 05 '11 at 20:06
0

Considering your requirements and the context of your use case, KubeMQ could be a suitable option for your message queue solution. KubeMQ is a self-hosted, Kubernetes-native message broker that offers a wide range of features that align well with your needs:

Reliable Message Storage: KubeMQ can store unprocessed messages reliably, even in the event of a shutdown or reboot. This ensures that no messages are lost, and processing can resume once the system is back online.

Exclusive Message Processing: KubeMQ supports exclusive message processing. This means that a job can only be processed by a single subscriber, which prevents duplicate processing and ensures that each job is handled exactly once.

Multiple Queues: You can set up multiple queues in KubeMQ, each with its own set of subscribers. This allows you to logically separate different types of jobs and have different subscribers processing them.

Scalability and Load Distribution: KubeMQ is designed to work in distributed environments. You can easily scale the number of subscribers and distribute the load across multiple machines to handle processing of your self-contained jobs.

Simplicity: KubeMQ is relatively easy to use and integrates well with Kubernetes environments. It provides APIs for different programming languages, making it accessible to a wide range of developers.

Advanced Features: KubeMQ offers various messaging patterns beyond simple queues, such as publish-subscribe, request-response, and more. This flexibility can be useful if your requirements evolve over time.

Self-Contained Setup: KubeMQ can be deployed as a Docker container or using Kubernetes, allowing you to manage and scale your messaging infrastructure as needed.

Given your criteria, KubeMQ provides a well-rounded solution that meets your requirements for reliable message storage, exclusive processing, multiple queues, and scalability. It offers these features in a user-friendly way, making it a strong candidate for your message queue solution.

Zain Malik
  • 1
0

a very simple to use is memcacheq, which uses the same API as memcached, so you can use the same libraries. it uses a BDB backend, so it's not RAM-only like memcached

Javier
  • 9,268
  • 2
  • 24
  • 24
0

Beanstalkd is a simple job-queue system that matches your basic needs. It can use a binary log to provide persistance if the queue itself fails and will only allow one worker to have a job at once, though jobs are also set with a timeout, so if they are not deleted, or returned to the queue before that, they are made available again (in case of worker problems).

I did a presentation on beanstalkd for a local user-group, which has some more information.

Alister Bulman
  • 1,624
  • 13
  • 13
0

I just went through this in my latest architecture planning..

Basically.. "the message queues".. all have problems that none of them guarantee both of the follow characteristics at the same time..

Guarantee recieving a message
Guarantee no duplicate messages

So what is currently offered as an open source solution cannot perform these two imperative tasks simultaneously.. (unless your want to spend 50K with IBM)

There is one great video which suggests that cassandra can handle this with quorum reads/writes, but is not taking into account concurrency on a high scale:/

In the end i settled on REDIS actually ( i avoided the clustered solution )

Simply and effectively single threaded.. (to avoid duplicity) Offers a atomic BlockonPop or even a multicast pubsubhubbub feature for queue workers..

a homegrown solution was developed to manage "lost jobs" that never arrived.. ( reliability )

Its quite a simple model actually.. seemingly easy to maintain aswell..

Hope this helps..

Arenstar
  • 3,602
  • 2
  • 25
  • 34
0

I've used Beanstalkd for this type of task. It can be configured to persist jobs to disk between reboots. To help with removing duplicates, I pushed job identifier into memcached - if the job was in memcache already, delete it rather than queueing in Beanstalkd.

dkam
  • 149
  • 3