How to set up HTTPHandler for python logging

Question

I'm trying to use HTTPHandler class of standard python logging library to send logs. I need to make a https post request with basic credentials(username and password). This is how i'm setting up the HTTPHandler-

host = 'example.com'
url = '/path'
handler = logging.handlers.HTTPHandler(host, url, method='POST', secure=True, credentials=('username','password'), context=None)
logger.addHandler(handler)

But the problem is, I'm not getting anylogs in my remote server.I'm not even seeing any exception from the handler. Am I setting up the handler arguments incorrectly? I can send similar logs using simple pythong http request-

url = 'https://username:password@example.com/path'
headers = {'content-type': 'application/json'}
jsonLog = { 'id': '4444','level': 'info', 'message': 'python log' };

r = requests.post(url, data = json.dumps(jsonLog), headers=headers)

Do i need to setup header somehow because of json content-type? If yes than how do i set that up in the httphandler?

Update

I thought I should update what I ended up doing. After numerous search i found i can create a custom handler by overriding emit() of logging.Handler.

class CustomHandler(logging.Handler):
    def emit(self, record):
        log_entry = self.format(record)
        # some code....
        url = 'url'
        # some code....
        return requests.post(url, log_entry, headers={"Content-type": "application/json"}).content

Feel free to post if any has any better suggestions.

Answer 1

Expanding on the solution saz gave, here's how add a custom HTTP handler that will forward the logs emitted to the specified URL using a bearer token.

It uses a requests session instead of having to establish a new session every log event.

Furthermore, if the request fails it attempts to resend the logs for a given number of retries.

Note: make sure your logging handler is as simple as possible to prevent the application from halting because of a log event.

I tested it with a simple localhost echo server and it works.

Feel free to suggest any changes.

import json
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class CustomHttpHandler(logging.Handler):
    def __init__(self, url: str, token: str, silent: bool = True):
        '''
        Initializes the custom http handler
        Parameters:
            url (str): The URL that the logs will be sent to
            token (str): The Authorization token being used
            silent (bool): If False the http response and logs will be sent 
                           to STDOUT for debug
        '''
        self.url = url
        self.token = token
        self.silent = silent

        # sets up a session with the server
        self.MAX_POOLSIZE = 100
        self.session = session = requests.Session()
        session.headers.update({
            'Content-Type': 'application/json',
            'Authorization': 'Bearer %s' % (self.token)
        })
        self.session.mount('https://', HTTPAdapter(
            max_retries=Retry(
                total=5,
                backoff_factor=0.5,
                status_forcelist=[403, 500]
            ),
            pool_connections=self.MAX_POOLSIZE,
            pool_maxsize=self.MAX_POOLSIZE
        ))

        super().__init__()

    def emit(self, record):
        '''
        This function gets called when a log event gets emitted. It recieves a
        record, formats it and sends it to the url
        Parameters:
            record: a log record
        '''
        logEntry = self.format(record)
        response = self.session.post(self.url, data=logEntry)

        if not self.silent:
            print(logEntry)
            print(response.content)

# create logger
log = logging.getLogger('')
log.setLevel(logging.INFO)

# create formatter - this formats the log messages accordingly
formatter = logging.Formatter(json.dumps({
    'time': '%(asctime)s',
    'pathname': '%(pathname)s',
    'line': '%(lineno)d',
    'logLevel': '%(levelname)s',
    'message': '%(message)s'
}))

# create a custom http logger handler
httpHandler = CustomHttpHandler(
    url='<YOUR_URL>',
    token='<YOUR_TOKEN>',
    silent=False
)

httpHandler.setLevel(logging.INFO)

# add formatter to custom http handler
httpHandler.setFormatter(formatter)

# add handler to logger
log.addHandler(httpHandler)

log.info('Hello world!')

Answer 2

You will need to subclass HTTPHandler and override the emit() method to do what you need. You can use the current implementation of HTTPHandler.emit() as a guide.

Answer 3

following up to istvan, you can use threads to prevent slowing down the program

import asyncio
import concurrent.futures
executor = concurrent.futures.ThreadPoolExecutor(max_workers=10)
import time
import json
import logging
import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.retry import Retry

class CustomHttpHandler(logging.Handler):
    def __init__(self, url: str, token: str, silent: bool = True):
        '''
        Initializes the custom http handler
        Parameters:
            url (str): The URL that the logs will be sent to
            token (str): The Authorization token being used
            silent (bool): If False the http response and logs will be sent 
                           to STDOUT for debug
        '''
        self.url = url
        self.token = token
        self.silent = silent

        # sets up a session with the server
        self.MAX_POOLSIZE = 100
        self.session = session = requests.Session()
        session.headers.update({
            'Content-Type': 'application/json',
            'Authorization': 'Bearer %s' % (self.token)
        })
    
        self.session.mount('https://', HTTPAdapter(
            max_retries=Retry(
                total=5,
                backoff_factor=0.5,
                status_forcelist=[403, 500]
            ),
            pool_connections=self.MAX_POOLSIZE,
            pool_maxsize=self.MAX_POOLSIZE
        ))
        
        super().__init__()

    def emit(self, record):
        '''
        This function gets called when a log event gets emitted. It recieves a
        record, formats it and sends it to the url
        Parameters:
            record: a log record
        '''
        
        executor.submit(actual_emit, self, record)

def actual_emit(self, record):
    
    logEntry = self.format(record)
    response = self.session.post(self.url, data=logEntry)
    print(response)

    if not self.silent:
        print(logEntry)
        print(response.content)

# create logger
log = logging.getLogger('test')
log.setLevel(logging.INFO)

# create formatter - this formats the log messages accordingly
formatter = logging.Formatter(json.dumps({
    'time': '%(asctime)s',
    'pathname': '%(pathname)s',
    'line': '%(lineno)d',
    'logLevel': '%(levelname)s',
    'message': '%(message)s'
}))

# create a custom http logger handler
httpHandler = CustomHttpHandler(
    url='<URL>',
    token='<YOUR_TOKEN>',
    silent=False
)

httpHandler.setLevel(logging.INFO)
log.addHandler(httpHandler)

def main():
    print("start")
    log.error("\nstop")
    print("now")

if __name__ == "__main__":
    main()

what this program does is send the logs to the threadpoolexecutor, with 10 max threads, if there are more logs then the threads can handle, it should queue up, this prevents slowdowns of the program. What you can also do, atleast what I am doing on my project of making a local host logging central database and viewer, I make a seperate thread on the serverside, and then instantly return a HTTP response to make it so all the database stuff happens after the HTTP resonse has been send back. This removes the need for threads on client, seen it is on localhost and then latancy is almost 0