Python Custom logging handler to create each log file per key in the dictionary

Question

I have a dictionary with some sample data like below

{"screener": "media", "anchor": "media","reader": "media"}

and I wanted to create a log file for each key in the dictionary. I'm planning to use this logging in streaming job which will get reused for every batch in the streaming. here I'm planning to use the rotating file handler per key as well.

here is my snippet

import logging
from logging.handlers import RotatingFileHandler
import time

logger = logging.getLogger('my_logger')
logger.setLevel(logging.DEBUG)

dict_store = {"screener": "media", "anchor": "media", "reader": "media"}
dict_log_handler = {}


def createFileHandler(name):
    handler = RotatingFileHandler(name, maxBytes=2000000000, backupCount=10)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    return handler


def runBatch():
    print("logging batch")
    for name in dict_store.keys():
        print(name)
        if name in dict_log_handler:
            print(f"getting logger from dict_handler {name}")
            handler = dict_log_handler[name]
        else:
            handler = createFileHandler(name)
            dict_log_handler[name] = handler

        logger.addHandler(handler)
        logger.info('Hello, world!')
        logger.info('Hello, world!')
        logger.info('Hello, world!')
        logger.info('Hello, world!')
        logger.info('Hello, world!')
        logger.info('Hello, world!')
        logger.removeHandler(handler)
        time.sleep(0.1)


for i in range(0, 3):
    runBatch()

It is working as expected currently. I'm just thinking of implementing similar stuff inside the overriding or creating a custom handler (like if we pass a name, it should automatically do this stuff) and the overall expectation is it should not affect the performance.

Question is, I wanted to wrap this in a class and using it directly. possible ?

Answer 1

You question is not clear what exactly you want to do, but if the idea is to use multiple loggers as your code shows then you can do something like this:

logging.getLogger(name) this is the method which is used to access the specific logger, in your code you are using the same logger but using addHandler and removeHandler to switch to specific logger.

You can create multiple loggers like this:

import logging
from logging.handlers import RotatingFileHandler


dict_store = {"screener": "media", "anchor": "media", "reader": "media"}


for name in dict_store:
    logger = logging.getLogger(name)
    logger.setLevel(logging.DEBUG)

    handler = RotatingFileHandler(name, maxBytes=2000000000, backupCount=10)
    formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')
    handler.setFormatter(formatter)

    logger.addHandler(handler)

You can wrap the above code in your own logging class/method and use it as needed. Keep in mind this needs to called only once.

Next time you can access the specific log and use the logging method:

logger = logging.getLogger(<logger_name>)
logger.debug("debug")
logger.info("info")
logger.warning("warning")
logger.error("error")
logger.critical("critical")