I found myself dealing with multiprocessing recently and I have about 10 processes that are printing simultaneously on different files but on the same console(Eclipse console). The amount of prints is about 40 to 50 lines per second for each process. I am using a Logger object that is instantiate separately for each process, and a print method that adds a timestamp, a level and a tag of the process name to the message to print and prints the message. What happens is that some of the lines are printed correctly and others are printed on the same line, obviously because they are printing at the same exact time.
The output I am getting is:
2018/09/05 10:43:20.362 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.362 | INFO | P1 | Processing line 208
2018/09/05 10:43:20.362 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.378 | INFO | P1 | Processing line 209
2018/09/05 10:43:20.378 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.378 | INFO | P1 | ------------------------------------------------------------------------2018/09/05 10:43:20.378 | P2 | Processing line 107
2018/09/05 10:43:20.378 | INFO | P2 | ------------------------------------------------------------------------
2018/09/05 10:43:20.384 | INFO | P2 | Processing line 108
2018/09/05 10:43:20.384 | INFO | P2 | ------------------------------------------------------------------------2018/09/05 10:43:20.384 | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.384 | INFO | P1 | Processing line 210
2018/09/05 10:43:20.384 | INFO | P1 | ------------------------------------------------------------------------
What I would like instead is:
2018/09/05 10:43:20.362 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.362 | INFO | P1 | Processing line 208
2018/09/05 10:43:20.362 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.378 | INFO | P1 | Processing line 209
2018/09/05 10:43:20.378 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.378 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.378 | INFO | P2 | Processing line 107
2018/09/05 10:43:20.378 | INFO | P2 | ------------------------------------------------------------------------
2018/09/05 10:43:20.384 | INFO | P2 | Processing line 108
2018/09/05 10:43:20.384 | INFO | P2 | ------------------------------------------------------------------------
2018/09/05 10:43:20.384 | INFO | P1 | ------------------------------------------------------------------------
2018/09/05 10:43:20.384 | INFO | P1 | Processing line 210
2018/09/05 10:43:20.384 | INFO | P1 | ------------------------------------------------------------------------
In the example above I've shown only INFO lines, but if something happens while processing the line I will have the next print to be an ERROR describing what happend. I've tried to only print the ERROR lines and works just fine since there are not many to print. But sometimes having just the error is not enough. There are other lines that tell me a lot more than "Processing line" and I can't miss them from console, I would have to open the file to see them.
Now, I was wondering if there is anyway of accessing the data that is being printed to the console or doing a global lock for prints. I was also thinking about a parallel Logger that gets all the standard output of the processes and handles it before printing but this would mean I should keep track of which process made the print and write it to a specific file.
Any suggestions?
