1

I'm trying to process some data utilizing python but encounter "MemoryError" frequently. For my data processing, I need to run a loop ~400 times. The issue is, every time I get 20-30 times into this loop I get "MemoryError". While watching Task Manager, this appears to happen when python.exe*32 is ~1 GB. I would use 64-bit python but some of the libraries I need are only 32 bit. Furthermore, at the end of the main function I'm calling with every loop, I'm utilizing the del function for just about every variable I use. When the "MemoryError" occurs, I have to completely exit the interpreter, otherwise if I try to re-reun it again it automatically fails with "MemoryError".

I'm using Python(x,y) 2.7 and need the following: numpy scipy.io matplotlib pyopencl pyfft

Stefan Bollmann
  • 640
  • 4
  • 12
  • 32
user2208604
  • 301
  • 2
  • 6
  • 15
  • Is it possible for you to reproduce this error in a short program? If so, you should post it here. – Stefan Bollmann Apr 21 '14 at 14:13
  • I'm not really sure. The code I'm working from is very lengthy. I didn't write it originally, I'm just trying to use it for data processing and so a lot of the opencl stuff is over my head. If it counts for anything, the program always fails at the exact same line around ~30-40 iterations in s = zeros((A,B,C),complex128) – user2208604 Apr 21 '14 at 14:21
  • Which I don't understand, because ti seems like creating that zeros matrix shouldn't be very taxing? – user2208604 Apr 21 '14 at 14:21
  • for A =2400, B=256, C=25***** – user2208604 Apr 21 '14 at 14:30

1 Answers1

4

You said it crashed on a line like this:

s = zeros((A,B,C),complex128)

With A=2400, B=256, C=25. That would require 235 MB of memory. And not just any 235 MB: it must be contiguous, because NumPy expects to use it as a single array.

You also mentioned you're running this in a 32-bit process, and that it crashes when the process memory usage gets around 1 GB.

This is unfortunate but not entirely surprising. Consider that the total usable virtual memory in a 32-bit process is somewhere around 3 GB, and there will inevitably be some "holes", meaning you will never be able to allocate a single array 4 GB in size. But how much can you allocate in one piece? Well, that depends on the fragmentation of the memory you have allocated so far, as virtual addresses that are still needed by your program may become "sprinkled" around its 4 GB address space, and eventually you may become unable to allocate a quarter-gigabyte chunk all together.

What you should do here is to allocate your matrix once and reuse it. When your program has just started you are extremely likely to succeed in allocating the 235 MB array, and you can later clear it before reusing it without allocating again.

Alternatively you could (a) move to 64-bit, or (b) spend potentially quite a bit of time tracking down why you can't allocate a large contiguous chunk after running a garbage-collected language for a while.

John Zwinck
  • 239,568
  • 38
  • 324
  • 436
  • Wow, thanks for the detailed answer. Some of the memory management stuff is a bit over my head, but I get the general jest of your comment :D. So, my function that I'm calling every time through my loop is defining 's=' for each instance. I thought I would be okay here because I'm using 'del s' at the end of this function. I'll retry this script while allocating s only once. Thanks! – user2208604 Apr 21 '14 at 15:47