1

I'm writing an small program to put tweets from the Twitter public stream into an HBase database. The program uses two threads, one to collect the tweets and one to process them. The first thread uses twitter4j StatusListener to get the tweets and puts them in an ArrayBlockingQueue with an capacity of 100. The second thread takes an status from the queue, filters the needed data and moves them to the database. The processing takes more time, than the collecting of the status.

The producer looks like this:

public void onStatus(Status status) {
    try {
        this.queue.put(status);
    } catch(Exception ex) {
        ex.printStackTrace();
    }
}

The consumer uses take and calls an function to process the new status:

public void run() {
    try {
        while(true) {
            // Get new status to process
            this.status = this.queue.take();
            this.analyse();
        }
    } catch(Exception ex) {
        ex.printStackTrace();
    }
 }

In the main function the two threads were created and started:

ArrayBlockingQueue<Status> queue_public = new ArrayBlockingQueue<Status>(100);

Thread ta_public = new Thread(new TweetAnalyser(cl.getOptionValue("config"), queue_public));
Thread st_public = new Thread(new RunPublicStream(cl.getOptionValue("config"), queue_public));

ta_public.start();
st_public.start();

The program runs for awhile without any problem, but then stops suddenly. At this time the queue is full and it seems, that the consumer is not able to take a new status from it. I tried several variations of the producer/consumer pattern without success. No exception is thrown.

I don't know were to look for the failure. I hope someone could give me an hint or an solution.

Thomas Anderson
  • 11
  • 3
  • How long is a while? Does the failure occur immediately when the queue fills up, or is it happy for some time? Do you have any sort of `System.exit`-type calls in `analyse` (or methods called from there)? – Andy Turner Oct 19 '15 at 22:34
  • No, it simply filters the hashtags, username and text of the tweet and puts them into the database. – Thomas Anderson Oct 19 '15 at 22:36
  • Does it still fail if you don't call `analyse`? And what exactly do you mean by "stop" - the JVM exits, it just hangs, or something else? – Andy Turner Oct 19 '15 at 22:42
  • Thanks for your help! I will try it tomorow without the analyse, but I had an System.out.println betweet the take and the analyse command, which wasn't shown. By stop I mean, that the consumer hangs. It doesn't exit the JVM. If I use offer with an timeout in the producer it collects new tweets until I manually exit the program. – Thomas Anderson Oct 19 '15 at 22:58
  • The failure was in the analyse function and an typical failure because of an typo. I have a second queue inside the function and mixed the names up, so analyse blocked itself. – Thomas Anderson Oct 20 '15 at 07:24
  • Glad you managed to work it out - there wasn't an obvious problem in this code. Be sure to add the resolution as an answer. – Andy Turner Oct 20 '15 at 07:24

1 Answers1

0

If working with blocking queues double check for blocking commands (put and take for ArrayBlockingQueue) in the code and typos if working with multiple lists.

Thomas Anderson
  • 11
  • 3