2

i have the following situation:

  1. Read data from database
  2. do work "calculation"
  3. write result to database

I have a thread that reads from the database and puts the generated objects into a BlockingQueue. These objects are extremely heavy weight hence the queue to limit amount of objects in memory. A multiple threads take objects from the Queue, performs work and put the results in a second queue. The final thread takes results from second queue and saves result to database.

The problem is how to prevent deadlocks, eg. the "calculation threads" need to know when no more objects will be put into the queue. Currently I achieve this by passing a references of the threads (callable) to each other and checking thread.isDone() before a poll or offer and then if the element is null. I also check the size of the queue, as long as there are elements in it, the must be consumed. Using take or put leads to deadlocks.

Is there a simpler way to achieve this?

beginner_
  • 331
  • 1
  • 3
  • 14
  • may be a dupe of http://stackoverflow.com/questions/5326013/proper-implementation-of-producer-consumer-scenario-and-graceful-termination-of –  Sep 06 '11 at 05:19

2 Answers2

0

One of the ways to accomplish would be to put a "dummy" or "poison" message as the last message on the queue when you are sure that no more tasks are going to arrive on the queue.. for example after putting the message related to the last row of the db query. So the producer puts a dummy message on the queue, the consumer on receiving this dummy message knows that no more meaningful work is expected in this batch.

Scorpion
  • 3,938
  • 24
  • 37
  • What I have is more like a producer-consumer/producer-consumer pattern. The middle part consumes data and then forwards the result to another consumer. The thing is that I have multiple instances of this "middle part". This is where the hard work is done. Problem is the "dummy data" can pass through this chain faster than the previous "real data" because no work is done on the dummy. – beginner_ Sep 15 '11 at 05:37
  • dummy data or the posion message was just to avoid the deadlock situation; so you know no more messages are going to arrive. you might as well use a countdown latch or something to track the completion of already started tasks. – Scorpion Sep 15 '11 at 06:25
0

Maybe you should take a look at CompletionService

It is designed to combine executor and a queue functionality in one. Tasks which completed execution will be available from the completions service via 

completionServiceInstance.take()

You can then again use another executor for 3. i.e. fill DB with the results, which you will feed with the results taken from the completionServiceInstance.

Svilen
  • 1,377
  • 1
  • 16
  • 23
  • This does not help me because I'm not passing around a ton of tasks but Objects generated from database. I have only 3 tasks which must run in parallel 1. reading from database 2. do work 3. write result to database. They have to be in parallel to limit memory usage because reading and writing is a lot faster than the actual work. – beginner_ Sep 13 '11 at 04:39