1

I'm using python's multiprocessing module for parallelising and processing my data. Before I close this process pool, I want to execute a function for each of the sub-processes that are spawned. How can I achieve this?

eg, below is a sample code:

from multiprocessing import Pool

def publish_metadata():
    print(str(metadata))

metadata = []

def process_row(row={}):
    global metadata
    metadata.append("something")
    # process the data


if __name__ == "__main__":
    pl = [////] # list to process data, each row is a dict
    pool = Pool(processes=10)
    pool.map(process_row, pl)
    pool.close()
    pool.join()

I want to run this function publish_metadata which will contain sub-process specific metadata, before I merge and close process pool (pool.close). How can I achieve that?

vish4071
  • 5,135
  • 4
  • 35
  • 65
  • 1
    I guess the *"normal"* way would be to submit 10 tasks to the pool with `None` as parameter before closing the pool. Then change `process_row` to check whether its parameter is `None` and if so do the exit processing, otherwise proceed with regular processing. Maybe Google *"poison pill"*. – Mark Setchell Mar 16 '22 at 21:44

0 Answers0