3

I have been working on a project which could be summarized as "a computation intensive analysis tool" and I chose to write the code so that multiple threads are used to process several individual calculations in parallel.

After a series of problems due to lack of experience in programming multithreaded projects, it appears that I have finally managed to get it to work like I have intended. Calculations are picked one by one from a queue, and processed simultaneously. Following a particular advice (regarding appropriate number of threads) I got in response to a previous question I have set it up so that there's one thread populating the process queue, while the rest of the threads in the pool pick up processes from the queue in succession. On my i7 machine with 8-cores I have gone with 1 producer + 8 consumer = 9 threads.

Now my question is as follows:

It's no surprise that when everything works as intended, the CPU usage of the machine hits the roof. I have been following the CPU usage through Gnome System Monitor, and it seems like the cores are on 90-100% load during the analysis stage of the software. Other than slowing down other programs (e.g. browser, mail, pdf reader etc) that are running at the same time, are there any other potential problems I should be aware of? Could having a high load create any artifacts for instance in JVM?

Considering that my software is intended to run as a scientific analysis tool rather than regular desktop application, slowing down other programs is something I am willing to accept, but I would like to know if there are other problems I should be aware of. Any practical examples or previous experience is very welcome.

Thanks in advance,

Community
  • 1
  • 1
posdef
  • 6,498
  • 11
  • 46
  • 94

1 Answers1

3

If it is a computationally intensive task, then I have found it useful to run such tasks at background priority. Unless the task is memory intensive and causes massive paging, this should allow other processes to run largely unhindered while still allowing the computational process to get the lion's share of actual available CPU. If you are running on Windows, then sometimes the GUI becomes unresponsive and it can become difficult to kill the process or get other work done. (Unix/Linux is better in this regard, but it can still be irritating.)

You really want to ensure that you don't use so much memory that your machine starts swapping like mad.

Carl Staelin
  • 1,067
  • 10
  • 9
  • Is such memory consumption even possible in Java? Isn't size of memory pool fixed at startup? – Basilevs Mar 20 '11 at 13:51
  • sadly it is also a memory heavy process (lots of collections flying around). I am sure I could try and optimize the memory use to some extent but I dont think I can go below several hundred megs. as for running other programs, I don't see it as too much of a problem; I am running a large test now, and both individual programs (like firefox) and the system itself seems very responsive. I haven't had the chance to profile the software thoroughly, but once I do, I could try and see how memory consumption varies during runtime. – posdef Mar 20 '11 at 14:29
  • In java the memory size is specified at startup, but you can set it to a very large number. – Carl Staelin Mar 29 '11 at 12:08
  • In java the memory size is specified at startup, but you can set it to a very large number. – Carl Staelin Mar 29 '11 at 12:08
  • short update: the software eats about 1.5G memory at most, through-out execution. – posdef Dec 15 '11 at 09:47