Sporadic problems in running a multi-threaded Java project in Win7

Question

I am working on a project that is both memory and computationally intensive. A significant portion of the execution utilizes multi-threading by a FixedThreadPool. In short; I have 1 thread for fetching data from several remote locations (using URL connections) and populating a BlockingQueue with objects to be analyzed and n threads that pick these objects and run the analysis. edit: see code below

Now this setup works like a charm on my Linux machine running OpenSUSE 11.3, but a colleague is testing it on a very similar machine running Win7 is getting custom notifications of timeouts on the queue polling (see code below), lots of them actually. I have been trying to monitor the processor use on her machine, and it appears that the software does not get any more than 15% of the CPUs while on my machine the processor usage hits the roof, just as I intended.

My question is, then, can this be a sign of "starvation" of the queue? Could it be so that the producer thread is not getting enough cpu time? If so how do I go about giving one particular thread in the pool higher priority?

UPDATE: I have been trying to pinpoint the problem, with no joy... I did however gain some new insights.

Profiling the execution of the code with JVisualVM demonstrates a very peculiar behavior. The methods are called in short bursts of CPU-time with several seconds of no progress in between. This to me means that somehow the OS is hitting the brakes on the process.
Disabling the anti-virus and back-up daemons do not have any significant affect on the matter
Changing the priority of java.exe (the only instance) through task manager (adviced here) does not change anything either. (That being said, I could not give "realtime" priority to java, and had to be content with "high" prio)
Profiling the network usage shows good flow of data in and out, so I am guessing that is not the bottleneck (while it is a considerable part of the execution time of the process, but that I know already and is pretty much the same percentage as what I get on my Linux machine).

Any ideas as to how the Win7 OS might be limiting the cpu time to my project? if it's not the OS, what could be the limiting factor? I would like to stress yet again that the machine is NOT running any other computation intensive at the same time and there is almost no load on the cpus other than my software. This is driving me crazy...

EDIT: relevant code

public ConcurrencyService(Dataset d, QueryService qserv, Set<MyObject> s){

    timeout = 3;
    this.qs = qserv;
    this.bq = qs.getQueue();
    this.ds = d;
    this.analyzedObjects = s;
    this.drc = DebugRoutineContainer.getInstance();
    this.started = false;

    int nbrOfProcs = Runtime.getRuntime().availableProcessors();
    poolSize = nbrOfProcs;
    pool = (ThreadPoolExecutor) Executors.newFixedThreadPool(poolSize);
    drc.setScoreLogStream(new PrintStream(qs.getScoreLogFile()));
}

public void serve() throws InterruptedException {
    try {
        this.ds.initDataset();
        this.started = true;
        pool.execute(new QueryingAction(qs));
        for(;;){
            MyObject p = bq.poll(timeout, TimeUnit.MINUTES);

            if(p != null){
                if (p.getId().equals("0"))
                    break;

                pool.submit(new AnalysisAction(ds, p, analyzedObjects, qs.getKnownAssocs()));
            }else 
                drc.log("Timed out while waiting for an object...");

        }

      } catch (Exception ex) {
            ex.printStackTrace();
            String exit_msg = "Unexpected error in core analysis, terminating execution!";

      }finally{
            drc.log("--DEBUG: Termination criteria found, shutdown initiated..");
            drc.getMemoryInfo(true);    // dump meminfo to log

            pool.shutdown();

            int mins = 2;
            int nCores = poolSize;
            long    totalTasks = pool.getTaskCount(), 
                    compTasks = pool.getCompletedTaskCount(),
                    tasksRemaining = totalTasks - compTasks,
                    timeout = mins * tasksRemaining / nCores;

            drc.log("--DEBUG: Shutdown commenced, thread pool will terminate once all objects are processed, " +
                        "or will timeout in : " + timeout + " minutes... \n" + compTasks + " of " +  (totalTasks -1) + 
                        " objects have been analyzed so far, " + "mean process time is: " +
                        drc.getMeanProcTimeAsString() + " milliseconds.");

            pool.awaitTermination(timeout, TimeUnit.MINUTES);
      }

}

The class QueryingAction is a simple Runnable that calls the data acquisition method in the designated QueryService object which then populates a BlockingQueue. The AnalysisAction class does all the number-crunching for a single instance of MyObject.

could be the case, but I am not familiar enough with win7 to check if that's the case, any suggestions on how to check that's the case?? — posdef, Dec 19 '11 at 12:15
How many consumer threads are you using? Have you tried to reduce the number to 1 and gradually increase it to see how the situation evolves? — Tudor, Jan 27 '12 at 13:45
Right now the poolsize == number of available cores. I have not tried to reduce the number of threads mostly because it does work full throttle when using a smaller dataset on that machine. Besides on another machine it appears to not have any problems at all. So I really don't think it's the number of threads, but how the OS manages the threads. But sure, if I get no other clues, I will eventually try and see if it makes a difference to decrease the number of threads in the pool. — posdef, Jan 27 '12 at 14:37
Ok, from your description it appears to be a starvation/lock contention problem. Have you tried to change the concurrent data structure to a different one? — Tudor, Jan 27 '12 at 17:48
What does means "polling timeouts"? Usual pool usage must hide all queue management inside, your producer just put apropriate Runnables inside pool. If you use it in more sofisticated way (making some handmade processing, as it seems to me from your description) -- please, clarify it. May be code snippet help us to understand issue — BegemoT, Jan 28 '12 at 15:39

score 3 · Answer 1 · answered Dec 19 '11 at 11:03

3

I suspect the producer thread is not getting/loading the source data fast enough. This might not be a lack of CPU but an IO related issue. (not sure why you have time outs on your BlockingQueue)

It might be worth having a thread which periodically logs things like the number of tasks added and the length of the queue (e.g. every 5-15 seconds)

answered Dec 19 '11 at 11:03

Peter Lawrey

525,659
79
751
1,130

While I do agree with you that it's most likely that the producer thread doesn't "produce" fast enough, I do not get what might be the reason for such differential behavior on two seemingly identical machines (except the OS). Regarding the queue; it's actually me logging my timeouts from the following call: `MyObj p = bq.poll(timeout, TimeUnit.MINUTES);` – posdef Dec 19 '11 at 12:14

score 3 · Answer 2 · answered Feb 01 '12 at 15:22

So, if I correctly understand your problem, you have one thread to fetch data, and several threads to analyse the fetched data. Your problem is that the threads are not correctly synchronized to run together and take full advantage of the processor.

You have a tipical producer-consumer problem with a single producer and several consumers. I advise you to remake your code a bit to have, instead, several independent consumer threads that are always waiting for resources to be available and only then running. This way you guarantee the maximum processor use.

Consumer thread:

while (!terminate)
{
    synchronized (Producer.getLockObject())
    {
        try
        {
            //sleep (no processing at all)
            Producer.getLockObject().wait(); 
        }
        catch (Exceptions..)
    }

    MyObject p = Producer.getObjectFromQueue(); //this function should be synchronized

    //Analyse fetched data, and submit it to somewhere...   
}

Producer thread:

while (!terminate)
{
    MyObject newData = fetchData(); //fetch data from remote location

    addDataToQueueu(newData); //this should also be synchronized

    synchronized (getLockObject())
    {
        //wake up one thread to deal with the data
        getLockObject().notify();
    }
}

You see that this way, your threads are always performing useful work or sleeping. This is just draft code to exemplify. See more explanation here: http://www.javamex.com/tutorials/wait_notify_how_to.shtml and here: http://www.java-samples.com/showtutorial.php?tutorialid=306

Thanks for the reply, and I will look into the details. But it appears to be that you're missing a very crucial and significant point; my code does get full processor power when I run it on my machine or another machine that runs Win7. What I am asking about is the reason of the differential performance on these (hardware-wise) similar machines. — posdef, Feb 01 '12 at 15:30
I had the very same issues some years ago, when working on a project that would run in several OSs. The difference you get is due to the thread schedulers of the operating systems. Though it is the java virtual machine to deal with the scheduling, it is still the operating system that dictates when the jVM works, and when it can access the remote systems you are using. It is very hard to specify the behaviour diference, and I believe you might be wasting your time in trying to find out. In the end, it will always be your code's fault. Try my approach, and tell me if you notice any improvement. — Anoyz, Feb 01 '12 at 15:57

score 1 · Answer 3 · answered Dec 19 '11 at 11:06

1

Priority won't help, since the problem is not an issue of deciding who gets precious resources -- resource usage isn't maxed. The only way the producer thread would not be getting enough CPU time is if it wasn't ready-to-run. Priority won't help, since the problem is not an issue.

How many cores does the machine have? It's possible that the producer thread is running full speed and there still just isn't enough CPU to go around. It's also possible the producer is I/O bound.

answered Dec 19 '11 at 11:06

David Schwartz

179,497
17
214
278

Both machines are i7 with 4cores and hyper threading (looking like 8 cores to the OS). We sit in the same physical building, and same floor, probably on the same switch. So I am not sure if there's a drastic difference between the I/O with regards to the world outside the boxes. :) My guess is that Win7 has much more threads associated with the OS, which might be the reason why things are not happening as efficiently as they do on Linux. However this is only a gut-feeling and I cannot back it up in any way :) – posdef Dec 19 '11 at 12:09
The best way to figure it out is to look in detail at what's happening on his machine. – David Schwartz Dec 19 '11 at 12:27
just to clarify, you suggest I profile the JVM while running the software, for instance with jvisualvm? Or do you refer to some other way of looking at it in detail? – posdef Dec 19 '11 at 13:07
I'm not recommending any particular tool. But you need some way to tell where the CPU is going, whether your producer thread is running or waiting for something, and so on. – David Schwartz Dec 19 '11 at 13:08
I see what you mean, I tried to profile the execution of the software (see edit) to get a better idea of what's going on. While it gave some insights, I still don't know what really happens instead of number-crunching :) – posdef Dec 20 '11 at 10:42

score 1 · Answer 4 · answered Dec 19 '11 at 11:08

1

You can try to separate the producer thread from the pool (i.e. create a distinct Thread and set the pool to have -1 the current capacity) and then set its priority to maximum via setPriority. See what happens, although priority rarely accounts for such a difference in performance.

answered Dec 19 '11 at 11:08

Tudor

61,523
12
102
142

That's what I was thinking as well, but I am not sure if that's an advisable method. I have read a number of posts on SO, and on other sources, advising against using `Thread` directly. But I am guessing this will be my last resort. – posdef Dec 19 '11 at 12:04
@posdef: As long as it solves the problem and you don't get mixed up managing both the producer and the consumers as different entities then I personally wouldn't advise against this strategy. But, as I said, it's hard to believe that priority is the only factor here... – Tudor Dec 19 '11 at 12:35
I guess I will eventually end up implementing it, in a future version of software, but as mentioned in my edit it's not the only problem right now. Apparently there are more serious issues in the code than whether or not the producer thread is in the pool. :( – posdef Dec 20 '11 at 10:44

score 1 · Answer 5 · answered Dec 20 '11 at 10:42

1

When you say URL connection, do you mean local or remote? It could be that network speed is slowing your producer down

answered Dec 20 '11 at 10:42

mbatchkarov

15,487
9
60
79

it is a remote connection. However as I have mentioned I strongly doubt that it's the network since, both the linux and win machines have comparable hardware, and sit on the same physical network. Besides the network profiling shows a nice high (almost) constant stream of data. – posdef Dec 20 '11 at 10:45

score 1 · Accepted Answer · answered Feb 02 '12 at 18:10

So after weeks of fiddling, wrestling in code and other types of suffering I think I had a breakthrough, "a moment of clarity" if you will...

I managed to show that the program can exhibits the same slow behavior on my Linux machine and can indeed run full throttle on the problematic Win-7 machine. The crux of the problem appears to be some sort of corruption of the system/cache files that are used to store the results of previous queries, and overall, speed up the analysis. You got to love the irony, in this case they appeared to be the reason for EXTREME slow analysis. In retrospect, I should have known (a la Occam's razor)...

I am still not sure what how the corruption occurs, but at least it's probably not related to different OS. Using the system files from my machine increases the output on the Win7 host up to about 40% only however. Profiling the process more has also revealed that, oddly enough, there is significantly more GC activity on Win7, which apparently took lots of CPU time from number crunching. Giving -Xmx2g takes care of excessive garbage collection and the CPU usage for the process shoots up to 95-96%, and threads run smoothly.

Now that my original question is answered, I have to say that overall java responsiveness is definitely better on Linux environment, even without allocating more heap memory, I can easily multi-task while I am running an extensive analysis in the background. Things are not as smooth in Win-7, e.x. resizing the GUI is significantly slow once the analysis takes off at full speed.

Thanks for all the replies, I am sorry for the partially misleading problem description. I merely shared what I found out while debugging to the best of my abilities. Anyways, I believe the bounty goes to Peter Lawrey, since he early on pointed to an I/O issue and it was his suggestion about a logger thread which eventually led me to the answer.

score 0 · Answer 7 · answered Nov 21 '13 at 08:27

Sorry not really an answer but did not fit inside comment and still it is worth the read I think:

well i am not JAVA friendly
but i have recently the same problem with C++ projects for machine control through USB.
On XP or W2K all goes perfectly for months of 24/7 operation on any 2 or more core machine
On W7 and strong enough machine all goes OK but sometimes (cca 1x per few hours) freezes for few seconds without obvious reason.
On W7 and relatively weak machine (2 core 1.66GHz T2300E notebook) the threads are freezing for some time and run again which under/overflows USB/WIN/App FIFOs and collapse communication ...
- it appears that nothing is blocked but the W7 sheduler just do not give CPU to the right threads occasionally.
- i thought that USB driver (JUNGO) communication freezes bud that is not true I measured it and it is OK even in freeze
- the freeze was about 6-15 seconds cca once per minute.
- after adding some safety sleeps to threads loops the freeze has shorten to about 0.5 sec
- but still there
- even if App do not Under/Overflows FIFOs the windows USB driver side do (few times per minute for few ms)
Change of exe/threads priority and class do not affect performance on W7 (on XP,W2K work as it should)

As you can see it seems we have most likely the same problem. In my case:

is not I/O related (when i replace USB thread with simulation of device it behaves similar)
adding Sleep to time critical code helps a lot
error is present also in low count of threads [2 fast (17ms) + 1 slow (250ms) + App code = 4]
my CPU consumption on W7 slow machine is also not 100% but about 95% which is OK because I have sleeps everywhere
my Apps use about 40-100MB of memory but are CPU computation demanding ...
- but not that much it could run safely on much slower machines
- but because of USB driver connection and multiple device support it need at least 2 cores
my next step is to add some kind of execution time logging/analyze to see what is happening in more detail
and also little rewrite of send/receive threads to see if it helps

When i learn something new/useful will add it.

score 0 · Answer 8 · answered Feb 01 '12 at 11:44

I would think it was some OS specific issue because that is the core difference between the two units. More specifically, something is slowing down the data arriving through the remote connection.

Find some traffic analysis tool such as Wireshark and/or Networx and try to discover if there is anything throttling the Win PC. Perhaps it is going through a proxy that has some kind of rate cap configured.

Sporadic problems in running a multi-threaded Java project in Win7

8 Answers8