This function does not work when using it in concurrent.futures

Question

I'm constructing a set of arrays using concurrent.futures.ProcessPoolExecutor() to make it faster. But for some reason the function does not work properly inside de ProcessPoolExecutor().

My code is the following:

def bitflip(int,p):
    mask = 1 << p
    return (int^ mask)

def check(num, n):
    return (num>>n)&1


def func(h):
    l = 20
    j = 1
    delt = -1

    A = np.zeros((2**8,2**8))
    i = 2**12
    for x in range(0,2**l):
        for y in range(0,l):
            k = (y+1)%l
            print(x//i,x,y)
            if check(x,y) == 0:
                A[x//i,x//i] += -j*h/2
            else:
                A[x//i,x//i] += j*h/2
            if check(x,y) == check(x,k):
                A[x//i,x//i] += -(j*delt)/2
            else:
                A[x//i,x//i] += (j*delt)/2
                b = bitflip(x,y)
                c = bitflip(b,k)
                A[x//i,c//i] += -j
    print(A)

t1 = time.perf_counter()
list_t = [16,13,12]
with concurrent.futures.ProcessPoolExecutor() as executor:
    executor.map(func,list_t)


t2 = time.perf_counter()

print(f'Finished in {t2-t1} seconds')

The output is like it only enter once in the first loop:

0 0 0
0 0 1
0 0 2
0 0 3
0 0 4
0 0 5
0 0 6
0 0 7
0 0 8
0 0 9
0 0 10
0 0 11
0 0 12
0 0 13
0 0 14
0 0 15
0 0 16
0 0 17
0 0 18
0 0 19
0 1 0
Finished in 0.21422048099339008 seconds

But if I comment the part the assign value to the array A, the it print the complete loop

Your main code should be inside `if __name__ == '__main__':` block when working with concurrency. — makozaki, Nov 07 '19 at 21:20
Where are `const`, `const_2`, `const_3` and `const_4` defined? Could the problem be that they aren't defined? It would make sense, seeing as commenting out the lines which refer to them makes it work. — kaya3, Nov 07 '19 at 22:09

makozaki · Accepted Answer · 2019-11-07T22:55:37.483

Remove print(x//i,x,y), it will spam your console with 2**21 logs and try looping first with some small power of 2 like l=4. When I try your code small l variable and with if __name__ == '__main__': it works fine. When you run it with l=20 it works as well but takes very long time to compute.

import time
import numpy as np
from concurrent import futures
def bitflip(int,p):
    mask = 1 << p
    return (int^ mask)

def check(num, n):
    return (num>>n)&1

def func(h):
    l = 20
    j = 1
    delt = -1

    A = np.zeros((2**8,2**8))
    i = 2**12
    for x in range(0,2**l):
        for y in range(0,l):
            k = (y+1)%l
            # print(x//i,x,y)
            if check(x,y) == 0:
                A[x//i,x//i] += -j*h/2
            else:
                A[x//i,x//i] += j*h/2
            if check(x,y) == check(x,k):
                A[x//i,x//i] += -(j*delt)/2
            else:
                A[x//i,x//i] += (j*delt)/2
                b = bitflip(x,y)
                c = bitflip(b,k)
                A[x//i,c//i] += -j
    print(A)

if __name__ == '__main__':
    t1 = time.perf_counter()
    list_t = [16,13,12]
    with futures.ProcessPoolExecutor() as executor:
        executor.map(func,list_t)
    t2 = time.perf_counter()
    print(f'Finished in {t2-t1} seconds')

Are you sure you are working with the same code as you posted/edited? It works fine with changes I suggested. I'm posting the whole part of code for clarity. — makozaki, Nov 07 '19 at 22:56
The real issue was in your function, nothing to do with concurrent.futures. Before using it, always run the function outside (in serial) to see the complete traceback. concurrent.futures will ensure that the task finishes even if there's an error. check [here](https://stackoverflow.com/questions/35711160/detect-failed-tasks-in-concurrent-futures) — Raphael, Nov 07 '19 at 23:03

This function does not work when using it in concurrent.futures

1 Answers1