Java & RabbitMQ - Queueing & Multithreading - Or Couchbase as Job-Queue

Question

i have one Job Distributor who publishes messages on different Channels.

Further, i want to have two (and more in the future) Consumers who work on different tasks and run on different machines. (Currently i have only one and need to scale it)

Let's name these tasks (just examples):

• FIBONACCI (generates fibonacci numbers)
• RANDOMBOOKS (generates random sentences to write a book)

Those tasks run up to 2-3 hours and should be divided equally to each Consumer.

Every Consumer can have x parallel threads for working on these tasks. So i say: (those numbers are just examples and will be replaced by variables)

• Machine 1 can consume 3 parallel jobs for FIBONACCI and 5 parallel jobs for RANDOMBOOKS
• Machine 2 can consume 7 parallel jobs for FIBONACCI and 3 parallel jobs for RANDOMBOOKS

How can i achieve this?

Do i have to start x Threads for each Channel to listen on on each Consumer ?

When do i have to ack that?

My current approach for only one Consumer is: Start x Threads for each Task - each Thread is a Defaultconsumer implementing Runnable. In the handleDelivery method, i call basicAck(deliveryTag,false) and then do the work.

Further: I want to send some tasks to a special consumer. How can i achieve that in combination with the fair distribution as mentioned above?

This is my Code for publishing

String QUEUE_NAME = "FIBONACCI";

Channel channel = this.clientManager.getRabbitMQConnection().createChannel();

channel.queueDeclare(QUEUE_NAME, true, false, false, null);

channel.basicPublish("", QUEUE_NAME,
                MessageProperties.BASIC,
                Control.getBytes(this.getArgument()));

channel.close();

This is my code for the Consumer

public final class Worker extends DefaultConsumer implements Runnable {
    @Override
    public void run() {

        try {
            this.getChannel().queueDeclare(this.jobType.toString(), true, false, false, null);
            this.getChannel().basicConsume(this.jobType.toString(), this);

            this.getChannel().basicQos(1);
        } catch (IOException e) {
            // catch something
        }
        while (true) {
            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                Control.getLogger().error("Exception!", e);
            }

        }
    }

    @Override
    public void handleDelivery(String consumerTag, Envelope envelope, AMQP.BasicProperties properties, byte[] bytes) throws IOException {
        String routingKey = envelope.getRoutingKey();
        String contentType = properties.getContentType();
        this.getChannel().basicAck(deliveryTag, false); // Is this right?
        // Start new Thread for this task with my own ExecutorService

    }
}

The class Worker is started twice in this case: Once for FIBUNACCI and once for RANDOMBOOKS

UPDATE

As the answers stated, RabbitMQ would not be the best solution for this, but a Couchbase or MongoDB pull approach would be best. I'm new to those systems, is there anybody that could explain to me, how this would be achieved?

Answer 1

Here's a conceptual view of how I would build this on couchbase.

1. You have some number of machines to process jobs, and some number of machines (maybe the same ones) creating jobs to do.
2. You can create a document for each job in a bucket in couchbase (and set its type to "job" or something if you're mixing it with other data in that bucket).
3. Each job description, along with the specific commands to be done, could include the time it was created, the time it is due (if there's a specific time due) and some sort of generated work value. This work value would be arbitrary units.
4. Each consumer of jobs would know how many work units it can do at a time, and how many are available (because other workers may be working.)
5. So a machine with, say, 10 work units of capacity that has 6 work units being done, would do a query looking for jobs of 4 work units or less.
6. In couchbase there are views which are incrementally updated map/reduce jobs, I think you'll only need the map phase here. You'd write a view that lets you query on time due, time entered into the system and number of work units. This way you can get "the most overdue job of 4 work units or less."
7. This kind of query, as capacity frees up, will get the most overdue jobs first, though you could get the largest overdue job, and if there are none then the largest not-overdue job. (Where "overdue" is the delta between current time and the due date on the job.)
8. Couchbase views allow for very sophisticated queries like this. And while they are incrementally updated, they are not perfectly realtime. Thus you wouldn't be looking for a single job, but a list of job candidates (ordered however you wish.)
9. So, the next step would be to take the list of job candidates and check a second location - possibly a membase bucket (eg: RAM Cache, non-persistant) for a lock file. The lock file would have multiple phases (here you do a little bit of partition resolving logic using CRDTs or whatever methods work best for your needs.)
10. Since this bucket is ram based it's faster than views and going to have less lag from total state. If there's no lock file, then create one with a status flag of "provisional".
11. If another worker gets the same job and sees the lock file, then it can just skip that job candidate and do the next one on the list.
12. IF, somehow two workers attempt to create lock files for the same job, there will be a conflict. In the case of a conflict you can just punt. Or you can have logic where each worker makes an update to the lock file (CRDT resolution so make these idempotents so that siblings can be merged) possibly putting in a random number or some priority figure.
13. After a specified period of time (probably a few seconds) the lock file is checked by the worker, and if it has not had to engage in any race resolution changes, it changes the status of the lock file from "provisional" to "taken"
14. It then updates the job itself with the status of "taken" or some such so that it will not show up in the views when other workers are looking for available jobs.
15. Finally, you'll want to add another step where before doing the query to get these job candidates I described above, you do a special query to find jobs that were taken, but where the worker involved has died. (eg: jobs that are overdue).
16. One way to know when workers die, is that the lock file put in the membase bucket should have an expiration time that will cause it to disappear eventually. Possibly this time could be short and the worker simply touches it to update the expiration (this is supported in the couchbase API)
17. If a worker dies, eventually its lock files will dissipate and the orphaned jobs will be marked as "taken" but with no lock file, which is a condition the workers looking for jobs can look for.

So in summary, each worker does a query for orphaned jobs, if there are any, checks to see if there is a lock file for them in turn, and if none then creates one and it follows the normal locking protocol as above. If there are no orphaned jobs then it looks for overdue jobs, and follows the locking protocol. If there are no overdue jobs, then it just takes the oldest job and follows the locking protocol.

Of course this will also work if there's no such thing as "overdue" for your system, and if timeliness doesn't matter then instead of taking the oldest job you can use another method.

One other method might be to create a random value between 1-N where N is a reasonably large number, say 4X the number of workers, and have each job be tagged with that value. Each time a worker goes looking for a job, it could roll the dice and see if there are any jobs with that number. If not, it would do so again until it finds a job with that number. This way, instead of multiple workers contending for the few "oldest" or highest priority jobs, and more liklihood of lock contention, they would be spread out.... at the cost of time in the que being more random than a FIFO situation.

The random method could also be applied in the situation where you have load values that have to be accommodated (So that a single machine doesn't take on too much load) and instead of taking the oldest candidate, just take a random candidate form the list of viable jobs and try to do it.

Edit to add:

In step 12 where I say "possibly putting in a random number" what I mean is, if the workers know the priority (eg: which one needs to do the job most) they can put a figure representing this into the file. If there's no concept of "needing" the job, then they can both roll the dice. They update this file with their role of the dice. Then both of them can look at it and see what the other rolled. If they lost then they punt and the other worker knows it has it. This way you can resolve which worker takes a job without a lot of complex protocol or negotiation. I'm assuming both workers are hitting the same lock file here, It could be implemented with two lock files and a query that finds all of them. If after some amount of time, no worker has rolled a higher number (and new workers thinking about he job would know that others are already rolling on it so they'd skip it) you can take the job safely knowing you are the only worker working on it.

Answer 2

First let me say that I haven't used Java for communicating with RabbitMQ so I wont be able to provide code-examples. That shouldn't be a problem however since that's not what you're asking about. This question is more about the general design of your application.

Lets break it down a bit, because there's a lot of questions going on here.

Dividing the tasks on different consumers

Well one way to do this is to use round-robin, but that is rather crude and doesn't take into account that different tasks may take a different amount of time to finish. So what to do. Well one way to do this is to set the prefetch to 1. Prefetching means that the consumer caches the messages locally (note: the message is not consumed yet). By setting this value to 1 no prefetching will occur. This means that your consumer will only know about and only have the message which it is currently working on in memory. This makes it possible to only receive messages, when the worker is idle.

When to acknowledge

With the setup described above it is possible to read a message from the queue, pass it on to one of your threads, and then acknowledge the message. Do this for all the available threads -1. You don't want to acknowledge the last message, because that means that you'll open up for receiving another message which you won't be able to pass to one of your workers yet. When one of the threads finishes, that's when you acknowledge that message, this way you'll always have your threads working with something.

passing on special messages

This depends on what you wan't to do, but in general I'd say that your producers should know what they are passing on. This means that you'd be able to send it to a certain exchange or rather with a certain routing-key that would pass on this message to a proper queue which will have a consumer listening to it that knows what to do with that message.

I'd recommend you read up on AMQP and RabbitMQ, this might be a good startingpoint.

caveats

There is one major flaw in my proposal and in your design, and that is that we ACK the message before we're actually done with processing it. This means that when(not if) our application craches, we have no way of recreating the ACKed messages. This could be solved if you know how many threads you're going to start beforehand. I don't know if you can change the prefetch count dynamically, but somehow I doubt that.

Some thoughts

From my, albeit limited, experience with RabbitMQ you shouldn't be scared of creating exchanges and queues, these can greatly improve and simplify your application design if done correctly. Maybe you shouldn't have an application that starts a bunch of consumer-threads. Instead you might want to have some kind of wrapper that starts consumers based on available memory in your system or something similar. If you do that you could make sure that no messages are lost, should your application crash, since if you do it like that, you'll, of course, acknowledge the message when you're done with it.

Recommended reading

• RabbitMQ tutorials
• Understanding AMQP

Let me know if something is unclear or if I'm missing your point and I'll try to expand upon my answer or improve it if I can.

Answer 3

Here are my thoughts on your question. As @Daniel mentioned in his answer, I believe this is more a question of architectural principles than of implementation. Once the architecture is clear, the implementation becomes trivial.

First, I would like to address something related to scheduling theory. You have very long-running tasks here, and if they are not scheduled in the proper manner, you will either (a) end up running your servers at less than full capacity or (b) taking much longer to finish the tasks than otherwise possible. So, I have some questions for you related to your scheduling paradigm:

1. Do you have the ability to estimate how long each job will take?
2. Do the jobs have a due date associated with them, and if so, how is it determined?

Is RabbitMQ Appropriate in this case?

I do not believe RabbitMQ is the proper solution to dispatch extremely long-running jobs. In fact, I think you are having these questions as a result of the fact that RabbitMQ is not the right tool for the job. By default, you do not have enough insight into the jobs before you remove them from the queue to determine which should be processed next. Second, as mentioned in @Daniel's answer, you probably won't be able to use the built-in ACK mechanism, because it would be probably be bad for a job to get re-queued whenever the connection to the RabbitMQ server fails.

Instead, I would look for something like MongoDB or Couchbase to store your "queue" for jobs. Then, you can have full control over the dispatching logic, rather than rely on the built-in round-robin enforced by RabbitMQ.

Other considerations:

Further, i want to have two (and more in the future) Consumers who work on different tasks and run on different machines. (Currently i have only one and need to scale it)

In this case, I don't think you want to use a push-based consumer. Instead, use a pull-based system (in RabbitMQ, this would be called Basic.Get). By doing this, you would take the responsibility for job scheduling

Consumer 1 has 3 threads for FIBONACCI and 5 threads for RANDOMBOOKS. Consumer 2 has 7 threads for FIBONACCI and 3 threads for RANDOMBOOKS. How can i achieve this?

In this case, I'm not sure I understand. Do you have ONE fibonacci job, and you are somehow executing it in parallel on your server? Or do you wish for your server to execuite MANY fibonacci jobs at the same time? Assuming the latter, you would create the threads to do the work on the server, then assign jobs to them until all your threads are full. When a thread becomes available, you would poll the queue for another job to start.

Other questions you had:

• Do i have to start x Threads for each Channel to listen on on each Consumer ?
• When do i have to ack that?
• My current approach for only one Consumer is: Start x Threads for each
• Task - each Thread is a Defaultconsumer implementing Runnable. In the handleDelivery method, i call basicAck(deliveryTag,false) and then do the work.
• Further: I want to send some tasks to a special consumer. How can i achieve that in combination with the fair distribution as mentioned above?

I believe the above questions will cease to be questions once you move the dispatching responsibility from RabbitMQ server to your individual consumers as mentioned above (and by consumer, I mean consuming threads). Furthermore, if you use something more database-driven (say Couchbase), you will be able to program these things yourself and you can have complete control over the logic.

Using Couchbase

While a detailed explanation of how to use Couchbase as a queue is beyond the scope of this question, I can offer a few pointers.

• First, you'll want to read up on Couchbase
• I recommend storing jobs in a Couchbase bucket, and rely on an indexed view to list the available jobs. There are lots of options for how to define a key for each job, but the job itself will need to be serialized into JSON. Perhaps use ServiceStack.Text
• When a job is pulled to be processed, there will need to be some logic to mark the job's status in Couchbase. You will need to use a CAS method to make sure that someone else hasn't taken the job for processing the same time you have.
• You will need some sort of policy for clearing out failed and completed jobs from your queue.

Summary

1. Do not use RabbitMQ for this
2. Use parameters of each job to come up with an intelligent dispatching algorithm. I can help you with this once I know more about the nature of your jobs.
3. Pull jobs into the workers based on the algorithm in #2, rather than pushing them from the server.
4. Come up with your own way of keeping track of the status of jobs across your system (queued, running, failed, succeeded, etc.) and when/whether to re-dispatch stalled jobs.

Answer 4

If are using spring or willing to use spring then you can use the spring listener container support to achieve it. That will provide you a similar callback kind of programming model that you are looking for.

Sample code from the Spring AMQP Reference documentation

@Configuration
public class ExampleAmqpConfiguration {

    @Bean
    public MessageListenerContainer messageListenerContainer() {
        SimpleMessageListenerContainer container = new SimpleMessageListenerContainer();
        container.setConnectionFactory(rabbitConnectionFactory());
        container.setQueueName("some.queue");
        container.setMessageListener(exampleListener());
        return container;
    }

    @Bean
    public ConnectionFactory rabbitConnectionFactory() {
        CachingConnectionFactory connectionFactory = new CachingConnectionFactory("localhost");
        connectionFactory.setUsername("guest");
        connectionFactory.setPassword("guest");
        return connectionFactory;
    }

    @Bean
    public MessageListener exampleListener() {
        return new MessageListener() {
            public void onMessage(Message message) {
                System.out.println("received: " + message);
            }
        };
    }
}

Answer 5

Recently I pushed branch bug18384 that changes the way callbacks are sent to Consumer implementations.

Following this change, the Connection maintains a dispatch thread that is used to send callbacks to the Consumers. This frees Consumers call blocking methods on the Connection and Channel.

A question came up on Twitter about making this configurable, allowing for a custom Executor to be plugged into the ConnectionFactory. I wanted to outline why this is complicated, discuss a possible implementation and see if there is much interest.

First off, we should establish that each Consumer should only receive callbacks in a single thread. If this is not the case, then chaos will ensue and Consumers will need to worry about their own thread safety beyond that of initialisation safety.

With only a single dispatch thread for all Consumers, this Consumer-Thread pairing is easy honoured.

When we introduce multiple threads, we have to ensure that each Consumer is paired with only one thread. When using the Executor abstraction, this prevents each callback dispatch from being wrapped up in a Runnable and sent to Executor, because you cannot guarantee which thread will be used.

To get around this, the Executor can be set to run 'n' long-running tasks (n being the number of threads in the Executor). Each of these tasks pulls dispatch instructions off a queue and executes them. Each Consumer is paired with one dispatch instruction queue, probably assigned on a round-robin basis. This is not too complicated and will provide simple balancing of dispatch load across the threads in the Executor.

Now, there are still some problems:

1. The number of threads in an Executor is not necessarily fixed (as with ThreadPoolExecutor).
2. There is no way, via Executor or ExecutorService to find out how many threads there are. Thus, we cannot know how many dispatch instruction queues to create.

However, we can certainly introduce a ConnectionFactory.setDispatchThreadCount(int). Behind the scenes this will create an Executors.newFixedThreadPool() and the correct number of dispatch queues and dispatch tasks.

I'm interested in hearing if anyone thinks I'm overlooking some simpler way of solving this, and indeed if this is even worth solving.