0

I've encountered some logging voodoos in my project.

After reading the followings: QueueHandler QueueListener

I designed the logging in my program as follows: in the main process - root logger assigned with a QueueHandler. That queue is listened by a QueueListener which its target is a plain StreamHandler. My program at some point of execution, sends an analyzes task to a pool of processes. Each pool process is assigned with the queue created in main process's QueueHandler, so it can send log records to the console as well.

Note about my program - it's python 2 program that I backported the QueueHandler's and QueueListener's code from python 3 manually.

An inconsistent problem appears in my program execution (but mostly in its tests): The program gets stuck, no messages are printed to the console. If the StreamHandler is swapped with FileHandler - tests pass consistently.

Here is the env I used for testing:

interpreter: PyPy 5.6.0 with GCC 4.8.4

pytest version: '3.2.3'

logging version: '0.5.1.2'

uname -a: Linux ip-172-31-36-72 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

I reckon it's not common to share a github repo - but I extracted my program into a very small project which the problem reproduces (sometimes). Feel free to check it out: log_voodoo

Why does this happen? Are there any solutions to overcome this?

NI6
  • 2,477
  • 5
  • 17
  • 28

2 Answers2

1

You could have used the logutils project, which backports the QueueHandler and QueueListener to Python 2.x.

You should set up with:

  • Main process - configure QueueListener and StreamHandler only
  • Workers - QueueHandler only

Ideally, the workers should create and add the QueueHandler themselves rather than relying on inheriting it from the main process (if they're forked from it). The logging cookbook has an example setup (working code, AFAIK).

Vinay Sajip
  • 95,872
  • 14
  • 179
  • 191
0

Without diving into the code, you might want to try with PyPY 6.0, there were some issues that were fixed between those two version.

mattip
  • 2,360
  • 1
  • 13
  • 13