9

I have a bit of an issue with an application running multiple Java threads. The application runs a number of working threads that peek continuously at an input queue and if there are messages in the queue they pull them out and process them.

Among those working threads there is another verification thread scheduled to perform at a fixed period a check to see if the host (on which the application runs) is still in "good shape" to run the application. This thread updates an AtomicBoolean value which in turn is verified by the working thread before they start peeking to see if the host is OK.

My problem is that in cases with high CPU load the thread responsible with the verification will take longer because it has to compete with all the other threads. If the AtomicBoolean does not get updated after a certain period it is automatically set to false, causing me a nasty bottleneck.

My initial approach was to increase the priority of the verification thread, but digging into it deeper I found that this is not a guaranteed behavior and an algorithm shouldn't rely on thread priority to function correctly.

Anyone got any alternative ideas? Thanks!

Duncan Jones
  • 67,400
  • 29
  • 193
  • 254
Smith Cosmin
  • 91
  • 1
  • Normally, priorities should not affect proper functioning, but this seems to be a tuning issue. Assuming the verification thread is low on CPU load, and the workers need more CPU, give them lower priority. Also, make sure the timeout for your atomic is not set too low. – Ralf H Feb 26 '13 at 14:27
  • 2
    Once again: show us the source code of your `verification thread`. – Andremoniy Feb 26 '13 at 14:47

5 Answers5

1

Instead of peeking into a regular queue data structure, use the java.util.concurrent package's LinkedBlockingQueue.

What you can do is, run an pool of threads (you could use executer service's fixed thread pool, i.e., a number of workers of your choice) and do LinkedBlockingQueue.take().

If a message arrives at the queue, it is fed to one of the waiting threads (yeah, take does block the thread until there is something to be fed with).

Java API Reference for Linked Blocking Queue's take method

HTH.

maggu
  • 1,201
  • 9
  • 9
  • Seconded. Don't just spin & peek in your worker threads, have them wait() on a lock or block on a queue when there's nothing to do. Most likely it's your over-active worker threads that are killing responsiveness. Do it properly. – Thomas W Mar 02 '13 at 02:04
0

More threads does not mean better performance. Usually if you have dual core, 2 threads gives best performance, 3 or more starts getting worse. Quad core should handle 4 threads best, etc. So be careful how much threads you use.

You can put the other threads to sleep after they perform their work, and allow other threads to do their part. I believe Thread.yield() will pause the current thread to give time to other threads.

If you want your thread to run continuously, I would suggest creating two main threads, thread A and B. Use A for the verification thread, and from B, create the other threads. Therefore thread A gets more execution time.

Steve
  • 1,469
  • 2
  • 11
  • 11
  • I thought that by default a thread get's the priority of the thread that created it so in the situation which you described the threads created with thread B will have the same prio as thread A. Correct me if I'm wrong – Smith Cosmin Feb 26 '13 at 14:53
  • Could be, I'm not sure, but what I'm saying here is that you keep switching between thread A and B. B will need to run multiple threads, while A will be a single thread, thus A will get much more time. By priority, I didn't mean the thread priority exactly, what I meant was it will have more execution time. Let me correct that to make it more clear. – Steve Feb 27 '13 at 06:36
0

One old school approach to throttling rate of work, that does not use a health check thread at all (and so by-passes these problems) is to block or reject requests to add to the queue if the queue is longer than say 100. This applies dynamic back pressure on to the clients generating the load, slowing them down when the worker threads are over loaded.

This approach was added to the Java 1.5 library, see java.util.concurrent.ArrayBlockingQueue. Its put(o) method blocks if the queue is full.

Chris K
  • 11,622
  • 1
  • 36
  • 49
0

Are u using Executor framework (from Java's concurrency package)? If not give it a shot. You could try using ScheduledExecutorService for the verification thread.

GJ13
  • 488
  • 3
  • 11
  • I'm actually using a ScheduledThreadPoolExecutor for the validation thread. The other threads are also created with ScheduledThreadPoolExecutor but in a different pool. The job that these threads execute has a while loop so they execute continously. Maybe I'm missing something. – Smith Cosmin Feb 26 '13 at 15:08
0

Seems you need to utilize Condition variables. Peeking will take cpu cycles.

http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/locks/Condition.html

Adnan Akbar
  • 718
  • 6
  • 16