I have a Joblib.Parallel computation which runs in a multiprocessing.Process. I'm facing the issue that when I join the process it never ends, even if joblib has completed the job.
This is a minimal example:
from multiprocessing import Process
from joblib import Parallel, delayed
from time import sleep
def job():
Parallel(n_jobs=10, verbose=10)(delayed(sleep)(0.2) for _ in range(10))
def spawn_process():
t = Process(target=job, name='Job')
t.start()
t.join()
if __name__ == '__main__':
spawn_process()
I think this is related to this Joblib Parallel doesn't terminate processes. Are there some ways to gently stop these processes without abruptly killing them (which leads to some warnings from loky backend)?
