How to use Python to log to a new directory each day?

Question

I'm setting up a readout system that takes data from a number of instruments and needs to log the data to a log file. This system will be running for weeks at a time, and so each day should have a log file. Since these instruments are being manipulated over this time, they may also have log files associated with their status.

With this, I have a directory in which all of the logs are stored, for example 'C:/logs'. Since there will be multiple log files associated with each day, I'd like to automate the creation of a new subdirectory in the the logs folder each day, so the structure of the files are something like 'C:/logs/20190814' for August 14, 'C:/logs/20190815' for the 15th, and so on. Then, in each daily directory I would have a number of log files such as 'data.log', 'instrument1.log', 'instrument2.log', etc.

Ideally, these would roll over at midnight each day.

I have been using the Python Logging module to attempt to create these log files. I have been able to implement the TimedRotatingFileHandler, but the problem with this is

(1) I want to change the directory that the log files are in based on the day, but leave their titles the same (e.g. 'C:/logs/20190814/data.log', 'C:/logs/20190815/data.log')

(2) the TimedRotatingFileHandler saves the files not with a '%Y%m%d.log' extension, but rather '.log.%Y%m%d', which is inconvenient to work with. I'd like to create a new directory each day and start writing a new log in the new day's directory.

Answer 1

Using the framework from another StackOverflow question that's similar but not exactly what I needed, I was able to get the behavior that I wanted. Here's the custom class that updates the logging TimedRotatingFileHandler class.

class MyTimedRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
def __init__(self, log_title, whenTo="midnight", intervals=1):
    self.when = whenTo.upper()
    self.inter = intervals
    self.log_file_path = os.path.abspath(os.path.join(os.path.dirname(__file__), "logs"))
    if not os.path.isdir(self.log_file_path):
        os.mkdir(self.log_file_path)
    if self.when == "S":
        self.extStyle = "%Y%m%d%H%M%S"
    if self.when == "M":
        self.extStyle = "%Y%m%d%H%M"
    if self.when == "H":
        self.extStyle = "%Y%m%d%H"
    if self.when == "MIDNIGHT" or self.when == "D":
        self.extStyle = "%Y%m%d"

    self.dir_log = os.path.abspath(os.path.join(self.log_file_path, datetime.now().strftime(self.extStyle)))
    if not os.path.isdir(self.dir_log):
        os.mkdir(self.dir_log)
    self.title = log_title
    filename = os.path.join(self.dir_log, self.title)
    logging.handlers.TimedRotatingFileHandler.__init__(self, filename, when=whenTo, interval=self.inter, backupCount=0, encoding=None)
    self._header = ""
    self._log = None
    self._counter = 0

def doRollover(self):
    """
    TimedRotatingFileHandler remix - rotates logs on daily basis, and filename of current logfile is time.strftime("%m%d%Y")+".txt" always
    """
    self.stream.close()
    # get the time that this sequence started at and make it a TimeTuple
    t = self.rolloverAt - self.interval
    timeTuple = time.localtime(t)

    self.new_dir = os.path.abspath(os.path.join(self.log_file_path, datetime.now().strftime(self.extStyle)))

    if not os.path.isdir(self.new_dir):
        os.mkdir(self.new_dir)
    self.baseFilename = os.path.abspath(os.path.join(self.new_dir, self.title))
    if self.encoding:
        self.stream = codecs.open(self.baseFilename, "w", self.encoding)
    else:
        self.stream = open(self.baseFilename, "w")

    self.rolloverAt = self.rolloverAt + self.interval

Answer 2

Here is an example:

import logging
import time

from logging.handlers import TimedRotatingFileHandler

#----------------------------------------------------------------------
def create_timed_rotating_log(path):
""""""
logger = logging.getLogger("Rotating Log")
logger.setLevel(logging.INFO)

handler = TimedRotatingFileHandler(path,
                                   when="m",
                                   interval=1,
                                   backupCount=5)
logger.addHandler(handler)

for i in range(6):
    logger.info("This is a test!")
    time.sleep(75)

#----------------------------------------------------------------------
if __name__ == "__main__":
log_file = "timed_test.log"
create_timed_rotating_log(log_file)

This example will rotate the log every minute with a back up count of 5. A more realistic rotation would probably be on the hour, so you would set the interval to 60 or the when to “h”. When this code is run, it too will create 6 files, but instead of appending integers to the log file name, it will append a timestamp using the strftime format %Y-%m-%d_%H-%M-%S.

Answer 3

I just did this exact same thing. You can do whatever type of processing in the 'my_namer' function. See below:

import logging
from logging.handlers import TimedRotatingFileHandler

def my_namer(default_name):
    # This will be called when doing the log rotation
    # default_name is the default filename that would be assigned, e.g. Rotate_Test.txt.YYYY-MM-DD
    # Do any manipulations to that name here, for example this changes the name to Rotate_Test.YYYY-MM-DD.txt
    
    # Get directory name of default_name
    dir_name = os.path.dirname(default_name)
    base_filename, ext, date = default_name.split(".")
    print(dir_name)
    day = date.split("_")[0]
    if os.path.exists(os.path.join(dir_name,day)) == False:
        os.mkdir(os.path.join(dir_name,day))
    
    return f'{dir_name}\{day}\file_name_{day}.{ext}'


logger = TimedRotatingFileHandler('logs/file_name.log', when="midnight", backupCount=30)
logger.namer = my_namer
logging.basicConfig(level=logging.INFO,format='[%(levelname)s:%(asctime)s] %(filename)s:%(lineno)d - %(message)s',datefmt='%m-%d-%Y %I:%M:%S %p'
                        ,handlers=[logging.StreamHandler(),logger])