2

I wrapped a ThreadPoolExecutor in an implementation of ExecutorService of my own, just to send it any filesystem writing task, so they would be treated sequencially and one-by-one. (No need to harass this poor disk writing head.)

The wrapper comes in handy by:

  • allowing me to Inject this ThreadPool as a Guice Singleton pretty much everywhere I need it
  • telling me in real-time how much more work there is left to do

This last feature is acomplished by the call to logUtils.writingHeartbeat(int) which logs a message about how many jobs are still in the queue if a "sufficient" time has been elapsed since last logging. It works pretty well in regards of writing logs at the desired intervals, but always tells me there is 0 files remaining to write. Which sounds fishy given the execution times.

What am I doing wrong?

@Singleton
public class WritersThreadPool implements ExecutorService {

    private final ThreadPoolExecutor innerPool;
    private final LogUtils logUtils;

    @Inject
    public WritersThreadPool(LogUtils logUtils) {
        innerPool = new ThreadPoolExecutor(1, 1, 0L, TimeUnit.MILLISECONDS, new LinkedBlockingQueue<Runnable>());
        this.logUtils = logUtils;
    }

    @Override
    public Future<?> submit(final Runnable r) {
        return innerPool.submit(new Callable<Void>() {
            @Override
            public Void call() throws Exception {
                r.run();
                logUtils.writingHeartbeat(innerPool.getQueue().size());
                return null;
            }
        });
    }

    (...) // Other implemented methods with no special behavior.
}
Silver Quettier
  • 2,045
  • 2
  • 26
  • 53
  • 1
    Have you tried `getTaskCount() - getCompletedTaskCount()` ? – Fildor Feb 24 '15 at 15:05
  • Maybe you should extend ThreadPoolExecutor and use afterExecute ... But I am still wondering why this is happening. I don't think it could be because of some optimization of the compiler ... – Fildor Feb 24 '15 at 15:15
  • I tried to log both `getTaskCount() - getCompletedTaskCount()` and `getTaskCount()`. It seems it always returns 1 remaining of now. So it's working, but the main thread do not continue after submitting a task. Wierd. – Silver Quettier Feb 24 '15 at 16:38
  • @Fildor - So the "0" was correct, and your suggestion allowed me to prove it. There's a problem elsewhere, but the logging is ok. (http://stackoverflow.com/questions/28712912/java7-pushing-file-i-o-in-a-separate-thread) I followed your ideas (I did not know about the afterExecute hook, thanks), so if you create an answer, I'll accept it. – Silver Quettier Feb 25 '15 at 06:51

2 Answers2

1

I share @chubbsondubs opinion, that there must be a synchronization issue elsewhere in the code.

My suggestions to prove some things were:

  • Try logging getTaskCountand getCompletedTaskCount. This led to your observation, that there is indeed only 1 Task in the queue at one given time.

  • Instead of composition, extend ThreadPoolExecutor and use the afterExecute hook. Maybe you can investigate who is synchronizing that should not that way.

Fildor
  • 14,510
  • 4
  • 35
  • 67
  • You were right indeed. I raised a new question hoping to understand why the parallelism isn't working the way I thought. Thank you for pointing me the hooks, I did not know them and it made my code much cleaner. :) – Silver Quettier Feb 25 '15 at 08:21
0

So I think the problem is that you are sequentially checking the queue size after you have fully run the Runnable being submitted. So the Runnable has fully completed it's work then it check the queue size which is going to be empty unless you had exhausted the number of threads in the innerPool. In other words there has to be something waiting in the queue for it to print out anything other than 0. The current job is being run by a thread so it's not on the queue.

chubbsondubs
  • 37,646
  • 24
  • 106
  • 138
  • Ah, no, I should have been clearer with that, but file I/O augments my execution time significantly (over 40 000 files to write), and they are processed one at a time (the ThreadPool has exactly one available Thread). – Silver Quettier Feb 24 '15 at 15:07
  • 1
    I think you must have some other place in your code that is serializing the adding of file IO to the queue that we can't see. – chubbsondubs Feb 24 '15 at 20:53