Python logging : Log data to server using the logging module

Question

The logging module offers the possibility of using HTTPHandler, which does not align to my requirement due to the limitations with formatting.

As given in the documentation, https://docs.python.org/3/library/logging.handlers.html, Using setFormatter() to specify a Formatter for a HTTPHandler has no effect.

My aim is to Log events in my application, and collect them on a local server. I am using JSON-Server to mock REST API (https://github.com/typicode/json-server). I have referred to this link : How to set up HTTPHandler for python logging, as a possible solution but i am not able to get what is desired.

My code:

"""class CustomHandler(logging.handlers.HTTPHandler):
    def __init__(self):
        logging.handlers.HTTPHandler.__init__(self)

    def emit(self, record):
        log_entry = self.format(record)
        # some code....
        url = 'http://localhost:3000/posts'
        # some code....
        return requests.post(url, log_entry, json={"Content-type": "application/json"}).content """

def custom_logger(name):

    logger = logging.getLogger(name)

    formatter_json = jsonlogger.JsonFormatter(
        fmt='%(asctime)s %(levelname)s %(name)s %(message)s') 

    requests.post('http://localhost:3000/posts', json= {"message" : "1" } ) 

 
    filehandler_all = logging.FileHandler('test.log')
    filehandler_all.setLevel(logging.DEBUG)
    filehandler_all.setFormatter(formatter_json)           
    logger.addHandler(filehandler_all)


    #http_handler = logging.handlers.HTTPHandler('http://localhost:3000' ,
     #"/posts", "POST")
    #
    # http_handler = CustomHandler()
   # http_handler.setFormatter(formatter_json)  
   # http_handler.setLevel(logging.DEBUG)
   
    return logger

logger = custom_logger("http")
logger.exception("{'sample json message' : '2'}")

The comments are for testing, and ease of reproduction of code.

In the above code snippet, the filehandler handles the json files perfectly, but the HTTPHandler does not. I tried creating a CustomHandler as specified in the link which in principle should work but I am not able to figure it out the details.

Will constructing a CustomHandler with changes to the "mapLogRecord" and the "emit" methods make sense?.

The most important thing, is to get the data in JSON Format.

Any other ideas to approach this can also be helpful!

changes to `mapLogRecord` make sense. changes to `emit` do not make sense. the fundamental issue is that what is logged are not strings, but `LogRecord` objects, and these objects need to be turned into strings some way. — blues, Jul 17 '20 at 17:46
Thanks, that made sense to proceed in the right direction. Another thing, is to maybe use the HTTP Handler directly, and filter out the log messages on server. This is my last approach, will update this question as and when I find a solution. — DragonsCanDance, Jul 20 '20 at 07:35

DragonsCanDance · Accepted Answer · 2020-07-21T12:12:42.437

Okay, so this is a solution which can output logs in JSON Format on server. It uses a custom logger. I am also able to format the messages for suitability. The code below gives the output in json format and uses the request module.

class RequestsHandler(logging.Handler):
    def emit(self, record):
        log_entry = self.format(record)
        return requests.post('http://localhost:3000/posts',
                             log_entry, headers={"Content-type": "application/json"}).content

class FormatterLogger(logging.Formatter):
    def __init__(self, task_name=None):
        
        super(FormatterLogger, self).__init__()

    def format(self, record):
        data = {'@message': record.msg,                
                 '@funcName' : record.funcName,
                 '@lineno' : record.lineno,
                 '@exc_info' : record.exc_info, 
                 '@exc_text' : record.exc_text,                 
                 }           

        return json.dumps(data)



def custom_logger(name):
    logger = logging.getLogger(name)
    custom_handler = RequestsHandler()
    formatter = FormatterLogger(logger)
    custom_handler.setFormatter(formatter)
    logger.addHandler(custom_handler)
    
    return logger


logger = custom_logger("http")
logger.exception("{'sample json message' : '2'}")

The data variable controls the parameters that can be added to the message.

The output :

 {
      "@message": "{'sample json message' : '2'}",
      "@funcName": "<module>",
      "@lineno": 62,
      "@exc_info": [
        null,
        null,
        null
      ],
      "@exc_text": null,
      "id": 138
    }

Python logging : Log data to server using the logging module

1 Answers1