4

From Concurrently run two functions that take parameters and return lists?, i can run 2 functions in parallel by specifying 2 queues. 

from threading import Thread
from Queue import Queue

def func1(x):
    return [i*i for i in x]

nums1 = [1,2,3,4,5]; nums2 = [112,32,53,64,25]

def wrapper(func, arg, queue):
    queue.put(func(arg))

q1, q2 = Queue(), Queue()
Thread(target=wrapper, args=(func1, nums1, q1)).start() 
Thread(target=wrapper, args=(func1, nums2, q2)).start() 

print q1.get(), q2.get()

How to run N threads of the same functions with different parameters in parallel?

Currently, I'm hard coding and doing:

nums1 = [1,2,3,4,5]; nums2 = [112,32,53,64,25]
nums3 = [11,522,33,467,85]; nums4 = [12,2,5,4,1125]
q1, q2, q3, q4 = Queue(), Queue(), Queue(), Queue()
Thread(target=wrapper, args=(func1, nums1, q1)).start() 
Thread(target=wrapper, args=(func1, nums2, q2)).start() 
Thread(target=wrapper, args=(func1, nums3, q3)).start() 
Thread(target=wrapper, args=(func1, nums4, q4)).start() 
print q1.get(), q2.get(), q3.get()
Community
  • 1
  • 1
alvas
  • 115,346
  • 109
  • 446
  • 738
  • You can use pool.map from multiprocessing module http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers – Tymoteusz Paul Mar 15 '14 at 20:48
  • 1
    I have a problem for which I need to use parallel computing. I tried to use the code you gave and afterwards I planned to modify my code accordingly. In order to get a longer computation-time, I replaced nums1 and nums2 with `range(100000)`. I left everything else the same. When I run the script, I realize that only one core is active. Am I missing something out? Shouldn't be 2 cores active? – T-800 Jun 15 '14 at 14:18

1 Answers1

4

Queues are threadsafe. So you should be able to get away with two queues, to be shared between all your threads:

from threading import Thread
from multiprocessing import Queue

def func1(x):
    return [i*i for i in x]

nums = [1,2,3,4,5,112,32,53,64,25]

def wrapper(func, qIn, qOut):
    for arg in iter(qIn.get, None):
        qOut.put(func(arg))

qIn, qOut = Queue(), Queue()
chunksize = len(nums)/numThreads
for i in xrange(numThreads):
    qIn.put(nums[i*chunksize : (i+1)*chunksize])
numThreads = 2  # or N
for _ in xrange(numThreads):
    qIn.put(None)
for _ in xrange(numThreads):
    Thread(target=wrapper, args=(func1, qIn, qOut)).start() 

for _ in xrange(numThreads):
    print qOut.get()
inspectorG4dget
  • 110,290
  • 27
  • 149
  • 241
  • @alvas: how many ever you want. My example sets it to 2 in the line `numThreads = 2` – inspectorG4dget Mar 15 '14 at 21:22
  • 1
    @alvas: that's to be expected. Remember that you are splitting your `nums` into chunks for each thread to work on. If you have more threads than numbers, your chunk size becomes 0, which is why you get that behavior – inspectorG4dget Mar 15 '14 at 21:26