Global Variable in Multiprocessing Pool

Question

I know this issue has been discussed here before, but I just cannot find any method that works. I want to share a global variable between my multiprocessing processes without any of the processes changing it, i.e. they just need read access. As a simple example, take:

    def f(x):
        return x**GlobalVar

    if __name__ == '__main__':
        GlobalVar = 6
        pool = multiprocessing.Pool()
        res= pool.map(f,[1,2,3,4])
        print(res)

Now this obviously doesn't work as GlobalVar will not be accessible by the processes. So for it to work I would gave to evaluate GlobalVar, or import the it from a file, in each separate process. As in my application GlobalVar is a very large array, this is extremely wasteful. How can I easily share this Global Variable between the processes while just storing one copy of it in memory? I want to reiterate that the processes only need to read this global variable without changing it.

Answer 1

Very simple way is to pass it as an argument to the f which gets executed in each process. But if the global variable is too huge and you don't want to have a copy of it in every process and you only intend to perform the read operation then you can use shared memory.

Sample (Documented inline)

from multiprocessing import Pool
from multiprocessing import shared_memory
import numpy as np
def f(x):
    # Attach to the existing shared memory
    existing_shm = shared_memory.SharedMemory(name='abc123')
    # Read from the shared memory (we know the size is 1)
    c = np.ndarray((1,), dtype=np.int64, buffer=existing_shm.buf)
    return x*c[0]

if __name__ == '__main__':
    a = np.array([6])
    # Creates shared memory with name abc123
    shm = shared_memory.SharedMemory(create=True, size=a.nbytes, name="abc123")
    # Create numpy array backed by shared memory
    b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf)
    # copy the data into shared memory
    b[:] = a[:]
    with Pool(5) as p:
        print(p.map(f, [1, 2, 3]))

Output:

[6, 12, 18]

Find Official docs here.

Answer 2

Since the variable you wish to share is read-only and a "simple" integer, you just need to make it visible to your sub-processes in your multiprocessing pool by declaring it at global scope:

import multiprocessing

GlobalVar = 6

def f(x):
    return x**GlobalVar

if __name__ == '__main__':
    pool = multiprocessing.Pool()
    res= pool.map(f,[1,2,3,4])
    print(res)

Prints:

[1, 64, 729, 4096]

Discussion

It is always relevant when discussing Python and multiprocessing which platform you are running on and I have updated your tags to add Windows (although) the code as written now will work on Linux also.

On Windows when a new process is created (or processes when creating a pool of processes), spawn is used. This means that the new processes do not inherit the variables that had been established by the main process but instead a new Python interpreter is launched for each new process and execution is started from the top of the program. This is why you must enclose the code that launches new processes within a if __name__ == '__main__': block or else you would get into a recursive loop. But for that reason, you must move the declaration of GlobalVar to global scope or else that variable will not be defined for the newly created processes.

The other way of initializing global variables for each sub-process within the pool is with a pool initializer function, which enables you to do more elaborate things than this demonstrates:

import multiprocessing

def init_pool(the_int):
    global GlobalVar
    GlobalVar = the_int

def f(x):
    return x**GlobalVar

if __name__ == '__main__':
    GlobalVar = 6
    pool = multiprocessing.Pool(initializer=init_pool, initargs=(GlobalVar,))
    res= pool.map(f,[1,2,3,4])
    print(res)