0

I'm making my first steps in Python and have come to a point where I need a logging module. The reasons I didn't opt for a rotating file handler are:

  1. I wanted a new folder to be created each time the code is run, hosting the new log files.
  2. Using conventional filenames (myName_X.log, and not the default myName.log.X).
  3. Limit log files by number of lines, and not file size (as done by the rotating file handler).

I've written such a module, using Python's built in logging module, but i have two problems:

  1. The new folder and file are created, and the logging data is printed into the file. However, when main() is run for the second time (see code below), the newly created file is locked, and cannot be deleted from the file explorer unless I close the IDE or release the lock through Process Explorer.
  2. The IPython interpreter freezes the second time I run main(). If I try the pdb module, it freezes as well.

I'm using WinPython 3.3.5 (with Spyder 2.3.0beta). I spent hours trying to find a solution to this problem. I don't know if it is a problem in my code or rather a bug with Spyder.

General coding remarks are always welcome.

main_example.py:

import myLogging


def main():

    try:
        myLoggerInstance = myLogging.MyLogger()


        # Do stuff...


        # logging example
        for i in range(0, 3):
            msg = 'Jose Halapeno on a stick {0}'.format(i)
            myLoggerInstance.WriteLog('DEBUG', msg)
        print('end of prints...')


    finally:
        myLoggerInstance._closeFileHandler()
        print('closed file handle...')


if __name__ == "__main__":
    main()

myLogging.py:

import logging
import time
import os


class MyLogger:
    _linesCounter = 0
    _nNumOfLinesPerFile = 100000
    _fileCounter = 0
    _dirnameBase = os.path.dirname(os.path.abspath(__file__))
    _dirname = ''
    _filenameBase = 'logfile_{0}.log'
    _logger = logging.getLogger('innerGnnLogger')
    _severityDict = {'CRITICAL' : logging.CRITICAL, 'ERROR': logging.ERROR, 'WARNING':         
    logging.WARNING, 'INFO': logging.INFO, 'DEBUG': logging.DEBUG}


    @staticmethod
    def __init__():
        # remove file handle
        MyLogger._closeFileHandler()

        # create folder for session
        MyLogger._dirname = MyLogger._dirnameBase + time.strftime("\\logs_%Y_%m_%d-    
        %H_%M_%S\\")
        MyLogger._dirname = MyLogger._dirname.replace('\\\\', '/')
        if not os.path.exists(MyLogger._dirname):
            os.makedirs(MyLogger._dirname)

        # set logger
        MyLogger._logger.setLevel(logging.DEBUG)


        # create console handler and set level to debug
        MyLogger._hConsole = logging.StreamHandler()
        MyLogger._hFile = logging.FileHandler(MyLogger._dirname + \
            MyLogger._filenameBase.format(MyLogger._fileCounter))
        MyLogger._hConsole.setLevel(logging.WARNING)
        MyLogger._hFile.setLevel(logging.DEBUG)

        # create formatter
        MyLogger._formatter = logging.Formatter('%(asctime)s %(filename)s, %(funcName)s, %(lineno)s, %(levelname)s: %(message)s')

        # add formatter to handlers
        MyLogger._hConsole.setFormatter(MyLogger._formatter)
        MyLogger._hFile.setFormatter(MyLogger._formatter)

        # add handlers to logger
        MyLogger._logger.addHandler(MyLogger._hConsole)
        MyLogger._logger.addHandler(MyLogger._hFile)


    @staticmethod
    def _StartNewFileHandler():

        MyLogger._closeFileHandler()

        # create new file handler
        ++MyLogger._fileCounter
        MyLogger._hFile = logging.FileHandler(MyLogger._dirname + \
            MyLogger._filenameBase.format(MyLogger._fileCounter))
        MyLogger._hFile.setLevel(logging.DEBUG)
        MyLogger._hFile.setFormatter(MyLogger._formatter)
        MyLogger._logger.addHandler(MyLogger._hFile)


    @staticmethod
    def WriteLog(severity, message):
        if (len(MyLogger._logger.handlers) < 2):
            MyLogger._StartNewFileHandler()
        MyLogger._linesCounter += 1
        MyLogger._logger.log(MyLogger._severityDict[severity], message)
        if (MyLogger._linesCounter >= MyLogger._nNumOfLinesPerFile):
            MyLogger._logger.info('Last line in file')
            MyLogger._StartNewFileHandler()
            MyLogger._linesCounter = 0


    @staticmethod
    def _closeFileHandler():
        if (len(MyLogger._logger.handlers) > 1):
            MyLogger._logger.info('Last line in file')
            MyLogger._logger.handlers[1].stream.close()
            MyLogger._logger.removeHandler(MyLogger._logger.handlers[1])
GL246
  • 273
  • 2
  • 9
  • I think this post will answer your problem: https://stackoverflow.com/questions/14388608/python-opening-a-file-without-creating-a-lock – matan7890 Aug 09 '14 at 13:50
  • This looks like a workaround for deleting files. I believe it would still prevent my code from being run multiple times. I checked Python's logging directly, and it works fine, so the file handling part seems to work as expected. I suspect a reference problem (maybe because of the 'import' or the static class 'MyLogger'). – GL246 Aug 09 '14 at 14:34
  • @GL246 How are you running it, exactly? This is working fine on my machine. – Patrick Collins Aug 10 '14 at 07:27
  • @PatrickCollins, I'm running it with the gui commands "run" and "debug" from the Spyder menu. Have you tried running `main` several times in a row? Can you delete the newly created folders? What OS are you using? (I'm using Windows 7 64bit) – GL246 Aug 10 '14 at 09:15
  • @GL246 I'm on Linux, I was running it inside an IPython shell. I didn't see any directories actually created though -- I suspect that's some cross-OS issue and I'm not sure I can provide any help for that reason. – Patrick Collins Aug 10 '14 at 10:06
  • Well, I found the problem. The handlers list has been growing each time `main` was run, because I deleted the second element of the handler's list: MyLogger._logger.handlers[1].stream.close(). Mutating a list using explicit indexing was a bad idea, and instead the hFile handle itself should be deleted. – GL246 Aug 30 '14 at 22:19

0 Answers0