1

I have a Python program using a RotatingFileHandler for logging. The logging file handler opens the logfile in exclusive mode (once) and keeps it open until the app closes. The problem is that I need to allow other processes to read the logfile while the Python program is still running.

In past projects using C++, I created a queued logger. It maintained a queue of log entries. A secondary worker thread would regularly check the queue and if there were any entries, open the logfile, dump the entries to the file, and immediately close the file until more log entries are queued. This meant that (in processor time) >99% of the time, the file would be closed and available for other processes to peek into the logfile.

(From a bit of digging, I'm under the impression that the Python logging class already handles the queuing of log entries... That's not the part I'm asking about.)

Is there a simple way to accomplish a normally-closed logfilehandler in Python? (Preferably without having to add a 3rd party library or subsystem.)

John L. Stanley
  • 71
  • 1
  • 6
  • Writing log records to a queue is possible by attaching a `QueueHandler` to the logger; I would then init and start a `QueueListener`, passing the same queue to the listener that was passed to `QueueHandler`. However, all file handlers from stdlib keep logfile opened as you've already found out; you will have to write a custom `FileHandler` impl that opens the file on emit and closes it immediately after the emit is done. Then pass an instance of the custom handler on `QueueListener` init and you should be good to go. – hoefling Jul 10 '18 at 22:40
  • Right now, your approach seems the most likely to work within the constraints I have on my project. If you submitted it as an answer, I would definitely up-vote it (although, I'm going to hold off a bit to see if anyone has a simpler solution before "accepting" an answer). Do you know any open source project that has something like this already working? – John L. Stanley Jul 20 '18 at 00:03
  • If I took this approach, is there a way for a QueueListener to know if it's receiving the last item in the queue? If there is, then I could use this to leave the file open while there are more items to write and only close it when the queue goes dormant. – John L. Stanley Jul 20 '18 at 00:28
  • 1
    Sure, just override the `QueueListener` class with your own impl. `QueueListener` has a reference to publisher queue (via `self.queue`), so overriding the [`QueueListener.dequeue`](https://docs.python.org/3/library/logging.handlers.html#logging.handlers.QueueListener.dequeue) method with `record = super().dequeue(); if self.queue.empty(): do_custom_stuff(); return record` should be a straightforward task. – hoefling Jul 20 '18 at 08:18
  • Or you could even pass the queue reference to file handler and do the queue check in `emit`. This way, you use the standard `QueueListener` and need to write only the file handler. Whatever fits your current architecture best. – hoefling Jul 20 '18 at 08:26

2 Answers2

1

As suggested in the comments, I'd go with using QueueHandler as the single root handler, combined with the QueueListener that acts on new records arriving. Other than that, a custom RotatingFileHandler is necessary that will close the file after the record is persisted and if no records are left in queue.

Disclaimer: the code below is untested.

import logging
import queue

global que_listener


class MyHandler(logging.RotatingFileHandler):

    def __init__(self, queue, *args, **kwargs):
        super().__init__(*args, delay=True, **kwargs)
        self.queue = queue

    def emit(self, record):
        if self.stream is None:
            self.stream = self._open()
        super().emit(record)
        if self.queue.empty():
            self.stream.close()
            self.stream = None


def init_logging():
    que = queue.Queue(-1)
    root_handler = QueueHandler(que)
    file_handler = MyHandler(que)
    que_listener = QueueListener(que, file_handler)
    root = logging.getLogger()
    root.addHandler(root_handler)
    que_listener.start()  # starts a separate thread to listen for queue updates


def cleanup_logging():  # stop listener on program exit
    que_listener.stop()

I used delay=True so the handler won't immediately open and lock the file on init as opposed to the default behaviour. Also, since the file is closed between records persisting, think about proper error handling in emit (file removed / locked by another process / etc).

hoefling
  • 59,418
  • 12
  • 147
  • 194
0

To me it seems that creating a separate thread managing this specific topic. As you pointed would be the best strategy.

The thread could perform read/write for every other programs.

You can perfectly design this thread in python using the logging library.

Another strategy would be to use a system such as Sentry or Kibana

Jonathan DEKHTIAR
  • 3,456
  • 1
  • 21
  • 42
  • The "perform read/write for every other programs" part of your response is unclear. I have no way to write -to- the other programs. They expect to be able to simply open a text file and look at the contents. (As past projects allowed them to do.) I have been asked to stick to standard Python libraries as much as possible (trying to make this a stand-alone process with minimal setup requirements), so learning Sentry or Kibana and figuring out how to add them is not likely to happen. – John L. Stanley Jul 10 '18 at 22:19
  • You can do pipes and publish the PID of your process ;) Or you can create a webservice and receive requests. It depends on how distributed is your application. You may find already working systems in github. Very common problem ! – Jonathan DEKHTIAR Jul 10 '18 at 22:24
  • Yes, I suppose I "could" do many of those things. You keep trying to sell me on approaches that don't fit the parameters of the question as I stated at the start. I'm looking for a solution that doesn't require re-engineering the other programs that use these logs. (As I've mentioned before....) – John L. Stanley Jul 20 '18 at 00:00