Different results in run and deburg mode with Multithread producer-consumer

Question

In a JPA project, I have two threads: producer and consumer.

Producer should get the search result of twitter query from another class and put it in a LinkedBlockingQueue, thread consumer should consume the result and use another class to persist them to MYSQL.

Here I show you first the main class, producer and consumer :

Main Class:

public class RunQuery {

    public final static EntityManagerFactory   
  emf=Persistence.createEntityManagerFactory("mypu");

    public static void main(String[] args) throws Exception

        {
            org.apache.log4j.BasicConfigurator.configure(new 
     NullAppender());

            BlockingQueue<Pair<String, QueryResult>> tweetBuffer=new 
     LinkedBlockingQueue<Pair<String,QueryResult>>();

            // Creating Producer and Consumer Thread
            Runnable producerThread = new 
         TwitterStreamProducer("producer",tweetBuffer,args);

            Runnable consumerThread = new 
       TwitterStreamConsumer("consumer",tweetBuffer);

           Thread producer=new Thread(producerThread);
           Thread consumer=new Thread(consumerThread);

           producer.start();  

           Thread.sleep(100);

            consumer.start();               
        }

Producer thread:

   public class TwitterStreamProducer implements Runnable{

    private final BlockingQueue<Pair<String, QueryResult>> tweetBuffer;
    private final ResultsController resultsController;
    private String[] Keywords;
    private String name;

    public TwitterStreamProducer(String name,
            BlockingQueue<Pair<String, QueryResult>> tweetBuffer, 
           String[] 

        keywords) {

        this.tweetBuffer=tweetBuffer;
        this.resultsController=new ResultsController();
        this.Keywords=keywords;
        this.name=name;

    }

    public void run() {

        for(String key:Keywords)
        {
        boolean interrupted = false;
        try {
            this.tweetBuffer.put( new Pair<String,QueryResult>
      (key,resultsController.search(key)) );
            Logger.getLogger("TwitterApp").info("one result added to the
       queue, current size:"+tweetBuffer.size());
        }

        catch (InterruptedException e) {
              interrupted = true;
        //  Logger.getLogger("Producer").info( this.name+ " Interrupted
            "+e.toString() );
            }
        catch(Exception e)
        {
            interrupted = true;
        }
        finally {
            if (interrupted)
                Thread.currentThread().interrupt();
                }

        }


    }

Consumer Thread:

public class TwitterStreamConsumer  implements Runnable{

    private final BlockingQueue<Pair<String, QueryResult>> tweetBuffer;
    private final ResultsController resultsController;
    private String name=null;

    public TwitterStreamConsumer(String name,
            BlockingQueue<Pair<String, QueryResult>> tweetBuffer) {

        this.name=name;
        this.tweetBuffer=tweetBuffer;
        this.resultsController=new ResultsController();
    }

    public void run() {


        while(! tweetBuffer.isEmpty())
        {
        try {
            Logger.getLogger("TwitterApp").info(this.name+" has consummed
     queue, current size of queue:"+tweetBuffer.size());
            resultsController.parse(this.tweetBuffer.take());
        } catch (InterruptedException e) {
        Logger.getLogger("Consumer").info(this.name+ " interrupted   
    "+e.getMessage());
        }

        }

    }

}

If other information is needed, I will provide here.

This is the Logger result, When I run it, the producer always produces 8 results and after that nothing more happens and the application will not be interrupted or does not produce any error:

Jul 12, 2015 12:07:22 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:1
Jul 12, 2015 12:07:23 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:2
Jul 12, 2015 12:07:23 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:3
Jul 12, 2015 12:07:24 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:4
Jul 12, 2015 12:07:24 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:5
Jul 12, 2015 12:07:25 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:6
Jul 12, 2015 12:07:25 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:7
Jul 12, 2015 12:07:26 PM main.TwitterStreamProducer run
INFORMATION: one result added to the queue, current size:8

When I debug it with the break point on the beginning of run method in consumer thread, then it work properly.

Thank you in advance for your suggestions.

score 2 · Answer 1 · answered Jul 12 '15 at 10:22

while(! tweetBuffer.isEmpty())

This will skip the loop if the queue is empty. So if the producer takes some time before storing something in the queue, your consumer doesn't do anything. Get rid of that check. Just use a loop that stops when the thread is interrupted, or when the queue contains a fake end-marker element, and repeatedly takes the next element from the queue.

score 1 · Answer 2 · answered Jul 12 '15 at 10:22

1

Your consumer will always run into an empty queue and die. When you debug it, you automatically make it wait long enough for the producer to fill the queue.

answered Jul 12 '15 at 10:22

Fildor

14,510
4
35
67

Thanks all of my friends. now the problem is that the producer always produces all the expected results and after that consumer starts to consume. Since I changed the classes, I made another question here : http://stackoverflow.com/questions/31367478/consumer-starts-after-producer-has-finished-his-work – jd466 Jul 12 '15 at 12:05

score 1 · Accepted Answer · answered Jul 12 '15 at 10:24

1

Your consumer ends in the first run. When it executes for the first time, the buffer should be empty and then it doesn't enter in the while loop and ends the run method and the Thread.

You have to loop waiting for the buffer and, sometimes, if the buffer is empty this shouldn't end the consumer. It has just to wait for the next message. You could do this with a code like this one:

public class TwitterStreamConsumer  implements Runnable{

    private final BlockingQueue<Pair<String, QueryResult>> tweetBuffer;
    private final ResultsController resultsController;
    private String name=null;
    private boolean stopped;

    public TwitterStreamConsumer(String name, BlockingQueue<Pair<String, QueryResult>> tweetBuffer) {

        this.name=name;
        this.tweetBuffer=tweetBuffer;
        this.resultsController=new ResultsController();
    }

    public void stop() {
        stopped = true;
    }

    public void run() {
        stopped = false;


        while(! stopped) {
            try {
                resultsController.parse(this.tweetBuffer.take());
                Logger.getLogger("TwitterApp").info(this.name+" has consummed queue, current size of queue:"+tweetBuffer.size());
            }
            catch (InterruptedException e) {
                Logger.getLogger("Consumer").info(this.name+ " interrupted "+e.getMessage());
            }
        }
    }
}

To stop the thread you have to do something like that:

consumer.stop();
consumerThread.interrupt();

answered Jul 12 '15 at 10:24

Sébastien Doncker

290
4
11

1

Why use a (non-volatile) stopped boolean instead of simply check if the current thread has been interrupted? – JB Nizet Jul 12 '15 at 10:26
Thread can be interrupted for many reasons (depending of the complexity of the application). The stop boolean provide an easy way to be sure the purpose of the interruption is to stop the consumer execution. – Sébastien Doncker Jul 12 '15 at 10:29
Thanks, where to put the above code? consumer.stop(); consumerThread.interrupt(); – jd466 Jul 12 '15 at 10:29
@javadeveloper in your main when you want to stop your program. – Sébastien Doncker Jul 12 '15 at 10:30
Then what would be a condition for that? – jd466 Jul 12 '15 at 10:31
@Fildor If this is the case, he can effectively avoid the stop process and just do what JB Nizet suggests : use Thread.currentThread().interrupted() to test if the thread is interrupted. – Sébastien Doncker Jul 12 '15 at 10:32
@javadeveloper How do you stop your program for now? – Sébastien Doncker Jul 12 '15 at 10:33
You can see the programm, I do not stop it. – jd466 Jul 12 '15 at 10:38
@javadeveloper Ok, I believe I understand you want your program ends after all keywords have been consumed. To do that, you have to be sure, in your consumer, there will be no other messages to treat. To be sure of that, perhaps your producer can add a last special message to the queue to indicate it's the last one. When the consumer receive this special message, it could end himself its run loop. And in that case, you don't have to run consumer.stop... – Sébastien Doncker Jul 12 '15 at 10:39
@SébastienDoncker Thanks a lot, I will do that now and tell you about the results. – jd466 Jul 12 '15 at 10:41
Is it fine like this : if(key.equals(keywords[keywords.length-1]) ) { this.tweetBuffer.put(new Pair("!",null)); } – jd466 Jul 12 '15 at 10:58
if(result.getKey()!=new String("!")) resultsController.parse(result); else this.stop() – jd466 Jul 12 '15 at 10:58
be careful, I'm pretty sure you can't put a null pointer into BlockingQueues. And for the consumer part, you have to use if(result.getKey().equals("!")) – Sébastien Doncker Jul 12 '15 at 11:15

Different results in run and deburg mode with Multithread producer-consumer

3 Answers3