0

I know that this question has been asked a lot of times, but the answers are not applicable. This is answer one of a parallelized loop using multiprocessing on StackoverFlow:

import multiprocessing as mp

def processInput(i):
    return i * i

if __name__ == '__main__':
    inputs = range(1000000)
    pool = mp.Pool(processes=4)
    results = pool.map(processInput, inputs)
    print(results)

This code works fine. But if I increase the range to 1000000000, my 16GB of Ram are getting filled completely and I get [Errno 12] Cannot allocate memory. It seems as if the map function starts as many processes as possible. How do I limit the number of parallel processes?

J. Doe
  • 11

2 Answers2

1

The pool.map function starts 4 processes as you instructed it (in the line processes=4 you instruct the pool on how many processes it can use to perform your logic).

There is however a different issue underlying this implementation. The pool.map function will return a list of objects, in this case its numbers. Numbers do not act like int-s in ANSI-C they have overhead and will not overflow (e.g. turn to -2^31 whenever reaching 2^31+1 on 32-bit). Also python lists are not array and do incur an overhead.

To be more specific, on python 3.6, running the following code will reveal some overhead:

>>>import sys
>>>t = [1,2,3,4]
>>>sys.getsizeof(t)
96
>>>t = [x for x in range(1000)]
>>>sys.getsizeof(t)
9024

So this means 24 bytes per number on small lists and ~9 bytes on large lists. So for a list the size of 10^9 we get about 8.5GB

EDIT: 1. As tfb mentioned, this is not even the size of the underlying Number objects, just pointers and list overhead, meaning there is much more memory overhead I did not account for in the original answer.

  1. Default python installation on windows is 32-bit (you can get 64-bit installation but you need to check the section of all available downloads in the python website), So I assumed you are using the 32-bit installation.
Fanchi
  • 757
  • 8
  • 23
  • 1
    I'm pretty sure that `sys.getsizeof` just tells you the size of the array object, not its contents. For instance `1.0*sys.getsizeof([[i for i in xrange(10)] for j in xrange(10000)])/10000` is about 8.7 on my implementation. So clearly it's not counting the objects in the array, just pointers to them. And `sys.getsizeof(1)` is 24. (Python doesn't seem to do anything clever with tagging so fixnums can be immediate as far as I can see.) –  Oct 22 '17 at 20:27
0

range(1000000000) creates a list of 10^9 ints. This is around 8GB (8 bytes per int on a 64-bit system). You are then trying to process this to create another list of 10^9 ints. A really really smart implementation might be able to do this on a 16GB machine, but its basically a lost cause.

In Python 2 you could try using xrange which might or might not help. I am not sure what the Python 3 equivalent is.

  • python 3 has deprecated the range implementation for the xrange generator, range() in python 3 is equal to using xrange() in python 2. – Fanchi Oct 22 '17 at 10:19
  • @Fanchi Thanks. I'm probably going to delete this answer because I think yours is better. –  Oct 22 '17 at 16:15