compile regex from log file into a dict, error when log file has a new attribute

Question

So I need to parse an nginx log file. Half way through the log file a new variable was added at the very end to each line.

I used https://github.com/bbb1991/nginx-log-parser/blob/master/main.py as an inspiration (i.e. used most of the code).

import re





REQUEST_TIME_CONF = '$remote_addr - $remote_user [$time_local] "$request" ' \
                    '$status $body_bytes_sent $http_referer" ' \
                    '"$http_user_agent" "$gzip_ratio" "$request_time"'                    

def get_requests(file_name):
    """

    """

    file_to_open = open(file_name,"r+")
    log_lines = file_to_open.readlines()

    lines = []

    log_pattern = ''.join(
        '(?P<' + g + '>.*?)' if g else re.escape(c)
        for g, c in re.findall(r'\$(\w+)|(.)', REQUEST_TIME_CONF))

    for line in log_lines:
        lines.append(find(log_pattern,line))
    return lines

def find(log_pattern, text):
    match =  re.match(log_pattern, text)
    if match:
        return match
    else:
        return False

def process_log(log_file):

    requests = get_requests(log_file)
    #print(requests)

    for x in range(len(requests)):

        request = requests[x]

        request = request.groupdict()

        remote_addr = request.get('remote_addr')
        remote_user = request.get('remote_user')
        time_local = request.get('time_local')
        request_item = request.get('request')
        status = request.get('status')
        body_bytes_sent = request.get('body_bytes_sent')
        http_referer = request.get('http_referer')
        http_user_agent =  request.get('http_user_agent')
        gzip_ratio = request.get('gzip_ratio')
        try:
            request_time = request.get('request_time')
        except AttributeError:
            request_time = None        



       # print(remote_addr,remote_user,time_local,request_item,status,
            #  body_bytes_sent,http_referer,http_user_agent,gzip_ratio,
            #  request_time)
        print(request)

access_log_to_parse = '/Users/username/Documents/Development/sample_access.log'
process_log(access_log_to_parse)

The sample_access.log file looks like this:

10.1.0.59 - - [12/Jul/2017:17:57:56 +0600] "POST /court/ws/avf HTTP/1.1" 500 296 "-" "CodeGear SOAP 1.3" "0.01" "0.003"
10.1.0.59 - userTest [12/Jul/2017:17:57:56 +0600] "POST /court/ws/avf HTTP/1.1" 500 296 "-" "CodeGear SOAP 1.3" "0.01"

Nginx has a specific log format which is declared in the REQUEST_TIME_CONF

I have removed the request_time from the last line to simulate an instance where the log line does not have this attribute.

So when there is a request_time present it will need to write the request_time value, otherwise just write None.

The following error is produced when the code runs:

AttributeError: 'bool' object has no attribute 'groupdict'

I did some more research on this and it looks like the re module returns a TRUE or FALSE value when something is matched (or not), as you can see I naively tried a try/catch for the request_time thinking that if the value isnt there I can just pass a Null to it but that didnt work.

So from what it looks like I think there needs to be some sort of check in the log_pattern regex findall function or during the re.match but my python skills are quite lacking (hence the code borrowing! hah)

score 0 · Answer 1 · answered May 01 '18 at 13:52

re module returns match or None never boolean, but your find function can return False if match is None. In this case result should not be appeneded to a list.

for line in log_lines:
    request = find(log_pattern,line)
    if request:
        lines.append(request)

compile regex from log file into a dict, error when log file has a new attribute

1 Answers1