Function to identify lines and writing in one file (csv)

Question

I have a log file with a lot of lines. The log file is in csv Format. I am searching in that file for different Messages and I want to store them in a different file.

How can I manage that?

Currently I am doing it like this:

with open('/tmp/result/warnings/test/test3.csv', 'r') as input_file:
    with open('/tmp/result/warnings/test/test4.csv', 'w') as output_file:
        for line in input_file:
            if not "Failed to open output file" in line:
                output_file.write(line)

with open('/tmp/result/warnings/test/test4.csv', 'r') as input_file:
    with open('/tmp/result/warnings/test/test5.csv', 'w') as output_file:
        for line in input_file:
            if not "Invalid file length of" in line:
                output_file.write(line)

Can I do it like looking up several Messages in once and then write in in one file?

Are you trying to find messages that match all conditions, or send messages with different contents to different files? — jonrsharpe, Mar 17 '14 at 15:23
I am trying to find messages matching all conditions and write those to one file. — Tiger1982, Mar 18 '14 at 07:00

score 0 · Accepted Answer · answered Mar 17 '14 at 15:18

with open('/tmp/result/warnings/test/test3.csv', 'r') as input_file:
    with open('/tmp/result/warnings/test/test4.csv', 'w') as output_file:
    for line in input_file:
        conditions = [not "Failed to open output file" in line, not "Invalid file length of" in line]
        if all(conditions):
            output_file.write(line)

Will allow you to check against multiple conditions and if they are all true, write to the file.

score 0 · Answer 2 · answered Mar 17 '14 at 15:58

Here is a super-modularized version:

# assumes Python 2.7
import csv

def read_csv(fname, header=False, **kwargs):
    with open(fname, "rb") as inf:
        incsv = csv.reader(inf, **kwargs)
        if header:
            head = next(incsv, [])
        for row in incsv:
            yield row

def write_csv(fname, rows, **kwargs):
    with open(fname, "wb") as outf:
        outcsv = csv.writer(outf, **kwargs)
        outcsv.writerows(rows)

def multi_filter(items, any_of=None, all_of=None):
    if any_of is None and all_of is None:
        return items

    if any_of is None:
        test_any = lambda item: True
    else:
        test_any = lambda item: any(cond(item) for cond in any_of)

    if all_of is None:
        test_all = lambda item: True
    else:
        test_all = lambda item: all(cond(item) for cond in all_of)

    return (item for item in items if test_any(item) and test_all(item))

def csv_filter(inf, outf, any_of=None, all_of=None):
    write_csv(outf, multi_filter(read_csv(inf), any_of, all_of))

conditions = [
    lambda row: not row[2].startswith("Failed to open output file"),
    lambda row: not row[2].startswith("Invalid file length of")
]

inf = '/tmp/result/warnings/test/test3.csv'
outf = '/tmp/result/warnings/test/test5.csv'
csv_filter(inf, outf, all_of(conditions))

Hey, yes it's Python 2.7 (sorry I forgot). I will test ypour code today. Thanks so far! — Tiger1982, Mar 18 '14 at 06:51
Your solution worked, but I took the one from C.B. because it fits better my program and what I want to do. It leads to less lines. - Thank you! — Tiger1982, Mar 18 '14 at 14:11

Function to identify lines and writing in one file (csv)

2 Answers2