1

How to communicate two parallel processes (without terminating) and call a function to process partial results in Python multiprocessing?

I have a Metropolis-Hasting algorithm, and I already parallelize this and can generate two (or more) outputs (mcmc chains) at the same time (it works fine for me). However, I want each number of steps to evaluate the partial results of all the parallel process with a convergence function to determine if two chains (two different calls to the metropolis algorithm function) are similar.

My scheme is as follows:

from multiprocessing import Pool
#chain_num is the number of chains that I want, hence the number of parallel process
p = Pool(self.chain_num)

i = range(self.chain_num)

p.map(self.metropolis, i)

p.close()

#where the metropolis function is an mcmc algorithm
def metropolis(self):
   #do something in each call

   # when the number of calls is 100,
   # I want to compare the partial outputs
   # of all processes in parallel with a
   # external function that works fine.
   # If the comparison through this external function
   #satisfy a condition, then
   # I want to interrupt and finish.

Could you help me or advise me?

Thank you!

Isidrogv
  • 26
  • 2
  • Nobody seems to be answering you, so I'll ask what is it that you want to send between the processes exactly? And have you tried a multiprocessing queue? – Mark Setchell Mar 27 '20 at 22:20
  • Thank you for your answer. In other words, I want to pause my parallel processes and do a comparison between them. Precisely I want to make the Gelman-Rubin diagnosis (for convergence) in parallel. I also try to use a queque, but I have a simmilar problem. – Isidrogv Mar 28 '20 at 00:40

0 Answers0