How to make dworkers for multiprocess?

Question

I am working on Distributed cluster computing. To implement such system I am trying to use python libs that is dask.distriuted. But there has a problem that is the dworkers are not for multiprocess, means 2 or 3 dworkers, works together but don't support multiple executions that support in multiprocessing lib.

for an example:

def testFun():
 while True:
  time.sleep(3)
  print('looping')

If I executes this function in the client.submit(testFun).It will execute this function for infinite times then it will never come to the next steps. Like for this program:

client.submit(testFun)
client.submit(testFun)

Here until execute the first line it will never come to the next line. I want to make that dworker for multiprocessing. How will I do this ?

score 1 · Accepted Answer · answered Dec 16 '16 at 02:03

1

That's because the function has the same signature, only runs one time.

You can tell by the key that is generated. See:

In [5]: client.submit(testFun)
<Future: status: pending, key: testFun-a4102f4653c498f9fafc90003d87bd08>

In [6]: client.submit(testFun)
<Future: status: pending, key: testFun-a4102f4653c498f9fafc90003d87bd08>

Try this

def testFun(x):
    while True:
        time.sleep(3)
        print('looping', x)

In [13]: client.submit(testFun, 1)
<Future: status: pending, key: testFun-afa640a088a357e5f8dd46c1937af3a7>

In [14]: client.submit(testFun, 2)
<Future: status: pending, key: testFun-98309530cb5b26d69131e54a521b8b40>

answered Dec 16 '16 at 02:03

R. Max

6,624
1
27
34

thanks it is working now. Can u tell me what is the parameter that ur passing in the submit function, one is function name .and what is other one?? – Saikat Kundu Dec 19 '16 at 17:05
@SaikatKundu the additional argument is passed to `testFun`. That makes the two tasks signature (function name + arguments) different. – R. Max Dec 20 '16 at 02:57

How to make dworkers for multiprocess?

1 Answers1