0

I am currently writing a python program that uses up a lot of RAM. I am aware that I could use the garbage collector to "free up" the ram and avoid this problem. However, I am using Numba to speed up my code, which makes the part where I would need to free the RAM incompatible with the garbage collector. Plus, it would slow down my code to the point where it would be totally useless anyway. Right now I am testing the code on a MacBook Pro which has 32 GB of RAM. I know a bit about software developing but not hardware. I am going to translate my code from the Mac onto raspberry pi's and Nano Pi Fire 3's. This is because I am using the multiprocessing module (Raspberry Pi has 4 cores and Nano Pi Fire 3 has 8). I am also obviously using the MPI module for passing messages. One solution I have found to add more RAM after brief searching has been swap files which are fairly easy to set up on the 2 types of boards I am using. However, I want to know if that is actually a viable option for a python code? If I allowed a swap file of 128 GB and utilized zRAM. This is for boards whose sole purpose is to run this code, would I no longer run into a problem with RAM?

This is the reason why I think that it is a problem with the RAM:

The first indication is if I don't speed up a specific function with Numba, then the program runs completely fine, except a little slower. However, when I use the speedup from Numba this is the crash report I receive:

VM Region Summary:
ReadOnly portion of Libraries: Total=688.6M resident=0K(0%) swapped_out_or_unallocated=688.6M(100%)
Writable regions: Total=1.0G written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=1.0G(100%)
 
                                VIRTUAL   REGION 
REGION TYPE                        SIZE    COUNT (non-coalesced) 
===========                     =======  ======= 
Dispatch continuations            32.0M        1 
Kernel Alloc Once                    8K        1 
MALLOC                           750.4M     1263 
MALLOC guard page                   16K        3 
MALLOC_LARGE (reserved)            384K        2         reserved VM address space (unallocated)
STACK GUARD                         32K        8 
Stack                             19.6M        8 
VM_ALLOCATE                      101.2M      723 
VM_ALLOCATE (reserved)           160.0M        2         reserved VM address space (unallocated)
__DATA                            28.3M      399 
__DATA_CONST                        20K        1 
__FONT_DATA                          4K        1 
__LINKEDIT                       407.9M      116 
__OBJC_RO                         32.2M        1 
__OBJC_RW                         1892K        2 
__TEXT                           280.7M      359 
__UNICODE                          564K        1 
shared memory                       12K        3 
===========                     =======  ======= 
TOTAL                              1.8G     2894 
TOTAL, minus reserved VM space     1.6G     2894 

Model: MacBookPro16,1, 8 processors, 8-Core Intel Core i9, 2.3 GHz, 16 GB, SMC

So, as can be seen there is a lot of RAM and swap space being used within my program. If I cannot find a solution or the adding of swap space to the boards does not work then obviously I will have to sacrifice speed for actually working. However, it would be nice if knowing the solution I think will work will actually work before I potentially waste time doing so.

Sam Moldenha
  • 463
  • 2
  • 11
  • 1
    Sorry, I'm not quite following your problem. What parts of this are based on assumptions and what parts are observations (e.g. numba not being garbage collected)? By garbage collections, do you mean *reference counting* /deallocation or the cyclic garbage collector? By multiprocessing, do you mean the ``multiprocessing`` standard library? How is MPI obvious, why not use the standard sync/comm primitives? Are you aware that Swap is orders of magnitude slower? Is your application time/realtime sensitive? Are you aware zRAM is *technically* Swap but lives in RAM, giving only a small RAM increase? – MisterMiyagi Sep 08 '20 at 06:26
  • For the MPI part, because I am building a cluster it needs a message passing interface. I know there is a problem with thee RAM because the program crashes at different itterations and comes up with a memory error. By garbage collecting I mean the cyclic garbage collector. Yes, I mean the standard `multiprocessing` library. Basically, I just want to know if file swapping can increase the allocated memory of a program. – Sam Moldenha Sep 08 '20 at 07:10
  • Note that at face value, the answer is "yes, Swap increases the allocatable space" – by definition. Since your *motivation* is that you do not want "to sacrifice speed for actually working", well, that is exactly what Swap does, so it's not clear whether *your problem* is solved by using Swap. – MisterMiyagi Sep 08 '20 at 08:07
  • 1
    Adding swap space will make it look like there more memory to utilise, but in reality there isnt, you would just be swapping out data from the memory to the disk, This could potentially slow down the process as you will have a lot more disk I/O to read and write from the swap space on the disk and load it back into memory to utilise it etc. – Chris Doyle Sep 08 '20 at 08:16

0 Answers0