5

I have a large python program doing some heavy numerical crunching using pyCUDA and numpy.

It's usually very fast, but seemingly randomly every now and then it will slow down by 3-4x with no apparent cause.
Watching CPU usage with htop shows that whenever it slows down, the CPU bar turns red, which as far as I understand, means 'kernel threads'.

enter image description here

But what does that really mean? How can I figure out what's causing that? It happens sometimes while the GPU is doing stuff, sometimes while the CPU is doing stuff. The program (which runs for several hours) slows down for an hour, recovers by itself and is fast for an hour, and then the problem relapses.

Any ideas? Further information you need?

talonmies
  • 70,661
  • 34
  • 192
  • 269
ali_p
  • 169
  • 1
  • 7
  • What does the memory use over time of your program look like? – Tom Dalton Mar 03 '16 at 16:52
  • Generally the memory usage is static, the way you see it there. There's a block of RAM that's used actively, and there's a large mem-mapped file (90GB in this case) which is only read from. The mem-map is used to cache the large file. – ali_p Mar 03 '16 at 17:06
  • Are you able to see (from top or otherwise) that the CPU is being consumed by your running process? Or by some other process (in which case, what process)? I'm assuming your program is a single process - is that right? Is it multithreaded? – Tom Dalton Mar 03 '16 at 17:13
  • Sorry for the delay. The program is multithreaded in some regions, but most of the time it's a single thread feeding data to the GPU and getting results back. I will check what top reports, thanks! – ali_p Mar 03 '16 at 21:39
  • 1
    I would also suggest watching I/O, e.g. using [`iotop`](http://guichaz.free.fr/iotop/). The slow-downs might correspond to big reads from your memmapped file. – ali_m Mar 03 '16 at 22:53

0 Answers0