3

The thing is that:

def main_fun(x):
    ...
    print(x)

if __name__ == "__main__":
    with Pool(5) as pool:
        pool.map(main_fun,range(10000))
    pool.close()
    pool.join()

My question is that: if I run the code on my own computer, it output subprocess print result. But when I submit it as a job to the cluster, I could not see the print result until the whole programs finished. How could I fix it? By the way, the cluster uses the Slurm.

Arty
  • 14,883
  • 6
  • 36
  • 69
master_zhen
  • 37
  • 1
  • 4

1 Answers1

1

Try doing print(x, flush = True) instead of just print(x).

Flush-variant of call does immediate flush of buffers so that printing is seen right away. While non-flush variant may keep string inside buffer until later time when it is flushed to screen.

Also always do pool.close() and pool.join() inside with block (at the end), not out-side like you did. For the case of just .map() function it doesn't matter, because it is blocking call, but for some other cases it is important to close/join inside with block.

Arty
  • 14,883
  • 6
  • 36
  • 69
  • @master_zhen Also notice, I just updated my answer with instructions regarding close/join, it is important for some of cases. – Arty Mar 16 '21 at 08:44