How to fix the broken pipe error in the following code?

Question

When I run the following part of the code it generates a broken pipe error. How can I fix this issue? The error generally refers to code line which says agent[0].start(). Thank you.

def main():

   np.random.seed(RANDOM_SEED)
   assert len(PACKET_SIZE) == A_DIM

   # create result directory
   if not os.path.exists(SUMMARY_DIR):
       os.makedirs(SUMMARY_DIR)

   # inter-process communication queues
   net_params_queues = []
   exp_queues = []
   for i in range(NUM_AGENTS):
       net_params_queues.append(mp.Queue(1))
       exp_queues.append(mp.Queue(1))


   # create a coordinator and multiple agent processes
   # (note: threading is not desirable due to python GIL)
   coordinator = mp.Process(target=central_agent,
                            args=(net_params_queues, exp_queues))
   coordinator.start()
   all_cooked_time, all_cooked_bw, _ = load_trace.load_trace(TRAIN_TRACES)
   agents = []
   print(NUM_AGENTS)
   for i in range(NUM_AGENTS):
       agents.append(mp.Process(target=agent,
                                args=(i, all_cooked_time, all_cooked_bw,
                                      net_params_queues[i],
                                      exp_queues[i])))
   #print(agents)
   #for i in range(NUM_AGENTS):
   agents[0].start()

   # wait unit training is done
   coordinator.join()


if __name__ == '__main__':
   main()

The complete traceback is shown below. I can add the complete code of the program if it is very necessary. I have copied it from somewhere and I am trying to understand it.


BrokenPipeError                           Traceback (most recent call last)
<ipython-input-24-152dc3806edb> in <module>
    332 
    333 if __name__ == '__main__':
--> 334     main()

<ipython-input-24-152dc3806edb> in main()
    325     #print(agents)
    326     #for i in range(NUM_AGENTS):
--> 327     agents[0].start()
    328 
    329     # wait unit training is done

C:\ProgramData\Anaconda3\lib\multiprocessing\process.py in start(self)
    110                'daemonic processes are not allowed to have children'
    111         _cleanup()
--> 112         self._popen = self._Popen(self)
    113         self._sentinel = self._popen.sentinel
    114         # Avoid a refcycle if the target function holds an indirect

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    221     @staticmethod
    222     def _Popen(process_obj):
--> 223         return _default_context.get_context().Process._Popen(process_obj)
    224 
    225 class DefaultContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\context.py in _Popen(process_obj)
    320         def _Popen(process_obj):
    321             from .popen_spawn_win32 import Popen
--> 322             return Popen(process_obj)
    323 
    324     class SpawnContext(BaseContext):

C:\ProgramData\Anaconda3\lib\multiprocessing\popen_spawn_win32.py in __init__(self, process_obj)
     63             try:
     64                 reduction.dump(prep_data, to_child)
---> 65                 reduction.dump(process_obj, to_child)
     66             finally:
     67                 set_spawning_popen(None)

C:\ProgramData\Anaconda3\lib\multiprocessing\reduction.py in dump(obj, file, protocol)
     58 def dump(obj, file, protocol=None):
     59     '''Replacement for pickle.dump() using ForkingPickler.'''
---> 60     ForkingPickler(file, protocol).dump(obj)
     61 
     62 #

BrokenPipeError: [Errno 32] Broken pipe

It probably depends on what happens in the `agent` function, which is not shown. So there is not enough context here to give a meaningful answer. Also, if you get an exception, please add the *complete traceback* to your question. — Roland Smith, Aug 23 '19 at 13:19
@RolandSmith thank you for your comment. I will add the complete traceback to the question — Frank Moses, Aug 23 '19 at 13:29
@RolandSmith I have added the complete traceback to my question. — Frank Moses, Aug 23 '19 at 13:33

score 1 · Answer 1 · answered Aug 23 '19 at 13:45

1

From the traceback it is clear that you are using multiprocessing from IPython on ms-windows.

Unfortunately, that doesn't work very well. AFAICT, it has to do with how IPython runs code in a notebook in combination with how multiprocessing has to work on ms-windows.

Try saving your code in a file and run that script from the command-line instead of in IPython.

answered Aug 23 '19 at 13:45

Roland Smith

42,427
3
64
94

Thank you for your answer. Can you guide me how can I do this on my computer. I have jupyter installed but I do not know how to open the python command prompt. – Frank Moses Aug 23 '19 at 13:59
@FrankMoses I'm not a ms-windows user, so I can't help you with that. – Roland Smith Aug 23 '19 at 14:11

How to fix the broken pipe error in the following code?

1 Answers1