0

I am trying to share a pool object between multiple processes using the following code

from multiprocessing import Process, Pool
import time
pool = Pool(5)
def print_hello():
    time.sleep(1)
    return "hello"
def pipeline():
    print("In pipeline")
    msg = pool.apply_async(print_hello()).get(timeout=1.5)
    print("In pipeline")
    print(msg)
def post():
    p = Process(target = pipeline)
    p.start()
    return
if __name__ == '__main__':
    post()
    print("Returned from post")

However the code exists with the timout since get() doesnot return. I believe this has to do with pool being a globally accessible variable because it works just fine when I move pool to being local to pipeline function. Can anyone give me suggestions if there exists a workaround for this problem ?

Edit: finally got working with thread instead of process for running pipeline function.

Hardik Aggarwal
  • 49
  • 5
  • There's no point trying to share the Pool to child processes. What's your end goal here? – AKX Aug 19 '22 at 13:33
  • I want to make a pipeline such that for every request a pipeline is fired. Inside the pipeline I have multiple subjobs. Say j1 and j2. Now I want all atmax 5 j1 jobs to be running so creating a pool of j1 type jobs. When my j1 job of the pipeline finishes I want my j2 job to be fired in a separate pool of processes allowing max 10 worker processes. I don't see how to achieve this without using 2 pools (for j1 and j2) shared between multiple pipelines (or processes). – Hardik Aggarwal Aug 19 '22 at 13:39
  • Well, why not use a single master process and two pools then? If you want to use a single pool, you can just make it large enough for all tasks, then keep track of how many tasks of each type is currently running and wait until there's "space". – AKX Aug 19 '22 at 13:48
  • The problem is I can't even use a single pool (the code would then look like the above sample I posted). I don't want my post to be blocking. Hence I fire another subprocess and return immediately. Now this subprocess handles firing j1 and subsequently j2 on completion of j1. – Hardik Aggarwal Aug 19 '22 at 13:57
  • @HardikAggarwal According to your description, you are already creating a subprocess for every `post`. So why not start the jobs `j1` and `j2` there and then in the pipeline function itself instead of using a pool? – Charchit Agarwal Aug 19 '22 at 15:46
  • @Charchit I already did mention that I need a pool because I need to limit my number of active worker processes. Also j1 and j2 are sequential processes so they cannot be started simultaneously in pipeline function. – Hardik Aggarwal Aug 22 '22 at 05:35

0 Answers0