Logging each API calls into a separate file

Question

I have a Django Application whose each API calls are associated with a transaction_id. I want to create separate log files for each transactions_id. In simple words I want to have multiple files which I will be using for logging.

How can i do this using Django's built in logging system ?

I can have multiple handlers in a single logger. But as per my requirement FileHandlers has to added in run time whose file name will be the transaction_id. This can be done. But the problem is if I have 4 transactions running at a time, 4 handlers will be added to the same logger, and according to documentation logs will be sent to each handlers resulting in 1 transaction log file logging the rest 3 transaction's logs as well.

Following is what I have come with:

class TransactionLogger:
  def __init__(self, transaction_id):
    self.logger = logging.getLogger('transaction_logger')
    logger = self.logger
    fileHandler = logging.FileHandler(transaction_id, mode='a')
    formatter = logging.Formatter('%(levelname)s %(asctime)s %(filename)s:%(lineno)s - %(funcName)s() ] %(message)s')
    fileHandler.setFormatter(formatter)

    self.logger.addHandler(fileHandler)
    self.logger.propagate = False

At the beginning of each transactions I instantiate the logger as :

logger = TransactionLogger(transaction_id).logger

and log as follows:

logger.debug("Hello World")

How can I maintain n number of log files which will be generated dynamically and log into each file based on transaction_id without interfering with other files.

Any help is appreciated.

You might not know it yet, but you don't want to have that. It you want to understand, create a folder with a few thousand files (or how many more you would expect) insided and list it, record the time taken, double the number of files and notice the chance of time it takes to list. — Klaus D., Sep 09 '18 at 19:24

score 0 · Answer 1 · answered Oct 24 '18 at 12:53

I would not say that its a good design to store logs like this. A better approach would be to write custom format to have transaction id in each log via which you will be able to filter your all logs.

Still here are are two ways you can achieve this:

1) By using logging._acquireLock() and logging._releaseLock() , OR you can use locking via LOCK as explained here.

2) Create a new logger every time (by inheriting logging.Manager and adding new logger to self.loggerDict ) and delete it at the end of the execution (so that system doesn't go out of memory).

Logging each API calls into a separate file

1 Answers1