2

I have a code running in Python 3.7.4 which forks off multiple processes. I believe I'm hitting a known issue (issue6721: https://github.com/python/cpython/issues/50970). I setup the child process to send "progress report" through a pipe to the parent process and noticed that sometimes a log statement doesn't get printed and that the code gets stuck in a deadlock situation. After reading issue6721, I'm not sure I'm still understanding why parent might hold logger Handler lock after a log statement is done execution (i.e the line that logs is executed and the execution has moved to the next line of code). I totally get it that in the context of C++, the compiler might re-arrange instructions. Not fully understand it in context of Python. In C++ I can have barrier instructions to stop the compiler moving instructions beyond a point. Is there something similar that can be done in Python to avoid having a lock getting copied to child process? I have seen solutions using "atfork" which is a library that seems not supported (so I can't really use it). Does anyone know a reliable and standard solution to this problem?

Amir
  • 421
  • 1
  • 4
  • 14
  • A known issue of which project? – mkrieger1 Feb 17 '23 at 00:06
  • aah thanks. just added a link to the issue. – Amir Feb 17 '23 at 00:09
  • "a reliable and standard solution to this problem" use spawn instead of fork. some libraries (even stdlib) have implemented at-fork cleanup, but it's very difficult to know if they're entirely safe. Future changes to the library can in some cases reintroduce these types of problems, so the general solution to this entire category of problem is to not do the unsafe thing in the first place (forking with threads). – Aaron Feb 17 '23 at 19:04
  • Been thinking about the points you're making. In this case there is a large data structure that can't be copied repeatedly between parent process and child process. Seems like using fork is inevitable. Is there a way to get the benefits of spawn and still be able to avoid copying the data structure? Also towards your point, I know for sure the root-cause is logger but we don't have threads in the code. Could the deadlock happen between parent process and the child process on the logger? – Amir Feb 22 '23 at 19:03

0 Answers0