Python Multiprocessing speed issue

Question

I have a nested for loop of the form

while x<lat2[0]:
    while y>lat3[1]:
        if (is_inside_nepal([x,y])):
            print("inside")
        else:
            print("not")
        y = y - (1/150.0)
    y = lat2[1]
    x = x + (1/150.0)
#here lat2[0] represents a large number

Now this normally takes around 50s for executing. And I have changed this loop to a multiprocessing code.

def v1find_coordinates(q):
  while not(q.empty()):

    x1 = q.get()
    x2 = x1 + incfactor
    while x1<x2:
        def func(x1): 
            while y>lat3[1]:
                if (is_inside([x1,y])):
                    print x1,y,"inside"
                else:
                    print x1,y,"not inside"
                y = y - (1/150.0)

        func(x1)
        y = lat2[1]
        x1 = x1 + (1/150.0)

incfactor = 0.7
xvalues = drange(x,lat2[0],incfactor)
#this drange function is to get list with increment factor as decimal
cores = mp.cpu_count()
q = Queue()
for i in xvalues:
    q.put(i)
for i in range(0,cores):
    p = Process(target = v1find_coordinates,args=(q,) )
    p.start()
    p.Daemon = True
    processes.append(p) 
for i in processes:
    print ("now joining")
    i.join()

This multiprocessing code also takes around 50s execution time. This means there is no difference of time between the two.

I also have tried using pools. I have also managed the chunk size. I have googled and searched through other stackoverflow. But can't find any satisfying answer.

The only answer I could find was time was taken in process management to make both the result same. If this is the reason then how can I get the multiprocessing work to obtain faster results?

Will implementing in C from Python give faster results?

I am not expecting drastic results but by common sense one can tell that running on 4 cores should be a lot faster than running in 1 core. But I am getting similar results. Any kind of help would be appreciated.

is Queue maybe a thread queue (import Queue) instead of a multiprocessing one? (from multiprocessing import Queue) — Hugo Walter, Jul 01 '16 at 14:21
I'm still learning multiprocessing myself, but i read that processor centered tasks gain nothing in speed if you are using multi-threading. I know you said you used pool before, but maybe look into it again. — Will.Evo, Jul 01 '16 at 14:25
@Will.Evo No, I am not using multi-threading here. I have created processes. And upon checking there are 4-5 processes. — Ritesh Jung Thapa, Jul 02 '16 at 11:55

score 0 · Answer 1 · answered Jul 01 '16 at 14:42

0

You seem to be using a thread Queue (from Queue import Queue). This does not work as expected as Process uses fork() and it clones the entire Queue into each worker Process

Use:

from multiprocessing import Queue

answered Jul 01 '16 at 14:42

Hugo Walter

3,048
1
16
10

I have done from multiprocessing import Queue not import queue – Ritesh Jung Thapa Jul 02 '16 at 11:56

Python Multiprocessing speed issue

1 Answers1