Search log file for value with bash

Question

I am trying to create a script which will query a log file (or files) for a particular value and using the values given print out not just the line that it is on but all the lines around it as long as they are not blank.

Essentially I want to run it like this:

logSearch.sh -f FilenameToSearch.log -s SearchTerm -b "20130323 12:00:00" e- "20130323 14:21:00"

FilenameToSearch.log contains the following:

03-10-2013 12:11:30
JunkData
JunkData

03-23-2013 12:00:00
JunkStill
Since
ValueLooking
ForIs
NotHere

03-23-2013 12:10:00
NotJunk
SearchTerm
ValueHere
NeedTo
GetAll
Yay

03-23-2013 12:10:30
BackToJunk
Blah

03-23-2013 12:11:00
SearchTerm
MorePrint

03-23-2013 15:10:00
SearchTerm
ButAfterGiven
Time
So
Junk

It will search the log file and find anything that matches the first search term in between the date-time values given (they are optional).

So it would return the following and pipe it to a new file:

03-23-2013 12:10:00
NotJunk
SearchTerm
ValueHere
NeedTo
GetAll
Yay

03-23-2013 12:11:00
SearchTerm
MorePrint

I have the basic code here I just havent gotten to actually processing the data yet so anything that you all could help with would be VERY APPRECIATED! I will keep working on this throughout the week and flesh it out more as the days progress but any ideas you have (since yall are likely better at writing these scripts) would be useful

Code

#!/bin/bash -eu

sytax="Proper invocation is logSearch.sh -s search-term -f filename-directory [b - datetimevalue1] [e - datetimevalue2]";
necVal="Error. -s and -f need to be specified";

usage () { echo $sytax; exit 1;}
error () { echo $necVal; usage; exit 2;}

options='s:f:b:e:x'

while getopts $options option
do
    case $option in

        s)
            searchFor=$OPTARG
            ;;
        f)
            inFilesLike=$OPTARG
            ;;
        b)
            betweenThis=$OPTARG
            ;;
        e)
            andThis=$OPTARG
            ;;
        *)
            usage;
            ;;
    esac
done

shift $(($OPTIND - 1))

if [ -z ${searchFor:-} ] || [ -z ${inFilesLike:-} ]
then
    error;
else
    echo "Search for : " $searchFor;
    echo "in files like: " $inFilesLike;
fi

if [ ! -z ${betweenThis:-} ]
then
    echo "Starting search at first occurance of: " $betweenThis;
else
    echo "No value to start search from. Beginning from the start of file(s).";
fi

if [ ! -z ${andThis:-} ]
then
    echo "Ending search at first occurance of: " $andThis;
else
    echo "No value to end search at. Will search to the end of file(s).";
fi

#BEGIN CODE TO SEARCH THROUGH $INFILESLIKE FILES FOR PARTICULAR VALUES

John Zwinck · Answer 1 · 2013-03-30T05:54:55.253

0

Here's a Python 3 program I tested on your example input:

#!/usr/bin/env python

import re

# TODO: use argparse to get these values from the command line
filename = "t.txt"
term = "SearchTerm"
start = "20130323 12:00:00"
end = "20130323 14:21:00"

lineCache = None # will be a list once we find the start time
lineMatch = False # will be True when the search term was found in a block

for line in open(filename):
    match = re.match('(\d{2})-(\d{2})-(\d{4})( \d{2}:\d{2}:\d{2})\n', line)
    if match: # got a timestamp
        time = ''.join(map(match.group, [3, 1, 2, 4])) # "YYYYMMDD HH:MM:SS"
        if time > end:
            break;
        if time >= start:
            lineCache = [line]
            lineMatch = False
    elif lineCache is not None: # got a regular text line
        if term in line:
            lineMatch = True
            print(''.join(lineCache), end='') # print earlier lines from block
            lineCache = []
        if lineMatch:
            print(line, sep='')
        else: # haven't found the search term yet; store line in case we do
            lineCache.append(line)

edited Mar 30 '13 at 05:54

answered Mar 28 '13 at 14:21

John Zwinck

239,568
38
324
436

Thank you for the information. I am downloading Python now and will test this to see if it works. I was looking for a bash example to go on though since I already have bash installed and wanted to work from that but that lineCache idea is pretty cool so Ill see if I can implement that in my current script – IceTechnology1906 Mar 28 '13 at 17:54
Currently getting a syntax error at: `print ''.join(lineCache), # print earlier lines from this block` – IceTechnology1906 Mar 28 '13 at 18:01
The error didnt say. What python editor would you suggest I use? I installed Python 3.3 and that was what I ran the above code in – IceTechnology1906 Mar 29 '13 at 13:10
Aha, in Python 3 it's necessary to say `print(foo)` instead of `print foo`. I've updated my answer accordingly (but haven't tested it with Python 3 myself). Just change those bits and try again. – John Zwinck Mar 30 '13 at 05:45

Search log file for value with bash

Code

1 Answers1