best way to parsing Large files by regex python

Question

I have to parse a large log file (2GB) using reg ex in python. In the log file regular expression matches line which I am interested in. Log file can also have unwanted data.

Here is a sample from the file:

"#DEBUG:: BFM [L4] 5.4401e+08ps MSG DIR:TX SCB_CB TYPE:DATA_REQ CPortID:'h8 SIZE:'d20 NumSeg:'h0001 Msg_Id:'h00000000"

My regular expression is ".DEBUG.*MSG."

First I will split it using the white spaces then the "field:value" patterns are inserted into the sqlite3 database; but for large files it takes around 10 to 15 minutes to parse the file.

Please suggest the best way to do the above task in minimal time.

Could you post an exemple of log line and pattern you are looking for ? — Julien Chappuis, Aug 29 '13 at 11:37
Format will be similar to "#DEBUG:: BFM [L4] 5.4401e+08ps MSG DIR:TX SCB_CB TYPE:DATA_REQ CPortID:'h8 SIZE:'d20 NumSeg:'h0001 Msg_Id:'h00000000" and reg ex is ".DEBUG.*MSG.". First i will split it using the white spaces then the "field:value" patterns are inserted into the database. — user2109788, Aug 29 '13 at 12:09
You might be better off using a program that works on a stream of data, like `sed`. — Burhan Khalid, Aug 29 '13 at 12:45

score 2 · Accepted Answer · edited May 23 '17 at 12:30

As others have said, profile your code to see why it is slow. The cProfile module in conjunction with the gprof2dot tool can produce nice readable information

Without seeing your slow code, I can guess a few things that might help:

First is you can probably get away with using the builtin string methods instead of a regex - this might be marginally quicker. If you need to use regex's, it's worthwhile precompiling outside the main loop using re.compile

Second is to not do one insert query per line, instead do the insertions in batches, e.g add the parsed info to a list, then when it reaches a certain size, perform one INSERT query with executemany method.

Some incomplete code, as an example of the above:

import fileinput

parsed_info = []
for linenum, line in enumerate(fileinput.input()):
    if not line.startswith("#DEBUG"):
        continue # Skip line

    msg = line.partition("MSG")[1] # Get everything after MSG
    words = msg.split() # Split on words
    info = {}
    for w in words:
        k, _, v = w.partition(":") # Split each word on first :
        info[k] = v

    parsed_info.append(info)

    if linenum % 10000 == 0: # Or maybe  if len(parsed_info) > 500:
        # Insert everything in parsed_info to database
        ...
        parsed_info = [] # Clear

PaulMcG · Answer 2 · 2013-08-29T13:12:16.937

0

"The best way to do the above task in minimal time" is to first figure out where the time is going. Look into how to profile your Python script to find what parts are slow. You may have an inefficient regex. Writing to sqlite may be the problem. But there are no magic bullets - in general, processing 2GB of text line by line, with a regex, in Python, is probably going to run in minutes, not seconds.

Here is a test script that will show how long it takes to read a file, line by line, and do nothing else:

from datetime import datetime

start = datetime.now()
for line in open("big_honkin_file.dat"):
    pass
end = datetime.now()
print (end-start)

edited Aug 29 '13 at 13:12

answered Aug 29 '13 at 11:42

PaulMcG

62,419
16
94
130

Thanks for the reply Paul, Actually I am new to python. I am using this in one of my projects. so can you please tell me, generally, how long will it take to read a large file(say 2GB) line by line and print it? and also how to know that the reg ex is efficient. If you can provide some good links regarding this also fine. – user2109788 Aug 29 '13 at 12:01
@user2109788 - you will learn more by writing this simple test script yourself. Why not time the simplest control case? Write a script that reads a file line by line, don't do anything yet. See how long that takes. Then open an output file, and write the line that was read to the output file. See how long that takes. These little tests should give you some idea of what the lower bound will be for your regex program. As for improving the regex itself, google for "regex performance" and you will get a number of helpful articles on the subject. – PaulMcG Aug 29 '13 at 13:08

score 0 · Answer 3 · answered Aug 29 '13 at 12:26

Paul's answer makes sense, you need to understand where you "lose" time first. Easiest way if you don't have a profiler is to post a timestamp in milliseconds before and after each "step" of your algorithm (opening the file, reading it line by line (and inside, time taken for the split / regexp to recognise the debug lines), inserting it in the DB, etc...).

Without further knowledge of your code, there are possible "traps" that would be very time consuming : - opening the log file several times - opening the DB every time you need to insert data inside instead of opening one connection and then write as you go

Luckily, python has a fantastic built-in detailed profiler, so the alternative here is not required. — Slater Victoroff, Aug 29 '13 at 12:47

best way to parsing Large files by regex python

3 Answers3