problem with shell script to get last logs for 10min

Question

I wrote a script that check last httpd error logs for the last 10 min but i got an error it stay looping: the problem is that the output of the script is a loop of errors until the log file ends I tyed to get the date - 10 min by running the cmd date --date='-10min' then I parsed the log file line by line then checked if the hours and minutes the every line in the log file is greater or equal to the hours and mins of date -10 mins here is a part of the output:

./test2.sh: line 26: [: -ge: unary operator expected
./test2.sh: line 26: [Mon: command not found
./test2.sh: line 26: [: -ge: unary operator expected
./test2.sh: line 26: [Mon: command not found
./test2.sh: line 26: [: -ge: unary operator expected
./test2.sh: line 26: [Mon: command not found
./test2.sh: line 26: [: -ge: unary operator expected
./test2.sh: line 26: [Mon: command not found

when i tryed to debug it here is a part of the problem and its repeat in every line of the log file :

+ IFS=
+ read -r line
+ errorBool=0
+ [[ [Sun Apr 28 03:52:39.791442 2019] [autoindex:error] [pid 15012] 
[client 127.0.0.1:49054] AH01276: Cannot serve directory /var/www/html/: 
No matching DirectoryIndex (index.html,index.php) found, and server- 
generated directory index forbidden by Options directive == *:error* ]]
++ awk -F : '{printf $1}'
++ '[Sun' Apr 28 03:52:39.791442 '2019]' '[autoindex:error]' '[pid' 
'15012]' '[client' '127.0.0.1:49054]' AH01276: Cannot serve directory 
/var/www/html/: No matching DirectoryIndex '(index.html,index.php)' found, 
and server-generated directory index forbidden by Options directive
test2.sh: line 26: [Sun: command not found
++ awk '{printf $4}'
+ '[' -ge 12 ']'
test2.sh: line 26: [: -ge: unary operator expected

Here is the code :

#!/bin/bash


#SCRIPT TO CHECK THE ERROR LOGS IN THE LAST 10 MINS

#VARIABLES

#NUMBER OF ERROR LOGS
errorCount=0
DATE_10_MIN=$(date --date='-10min' | awk '{print $4}' )
DATE_10_MIN_HOURS=$(date --date='-10min' | awk '{print $4}' | awk -F : 
'{print $1} ')
DATE_10_MIN_MIN=$(date --date='-10min' | awk '{print $4}' | awk -F : 
'{print $2} ')

#_______________________#

while IFS= read -r  line ; do


#BOOLEAN TO CHECK IF THE LINE HAVE THE EXPRESSION
errorBool=0


#if [[ $($line | awk '{print $4 }' | cut -c-8) -gt $DATE_10_MIN   ]] ; 
then

    if [[ $line  == *:error*  ]] ; then
            if [ [ $($line  | awk  '{print $4}' | awk -F : '{print $1}' )  
-ge $DATE_10_MIN_HOURS ] && [ $($line | awk  '{print $4}' | awk -F : 
'{print $2}')  -ge $DATE_10_MIN_MIN ] ]; then
            errorBool=1
            (( errorCount++ ))
            echo $line
            fi
    fi
 done < /var/log/httpd/error_log

 echo "There were $errorCount error logs in the last 10 mins "

Answer 1

This works on my test system. To test it properly, you have to change the dates in the input data :) .

Code

#!/bin/bash
#Script to check the error logs in the last 10 mins

#Variables.  Note: user variables should not be all uppercase.
errorCount=0
    # Number of error logs found
date_10_min_ago="$(date --date='-10min' +'%s')"
    # Time in seconds so it can be compared.
    # See https://unix.stackexchange.com/a/170982/63804 .

#_______________________#

while IFS= read -r line ; do
    if [[ $line  == *:error*  ]] ; then
        line_timestamp="$(awk -F '[][]' '{print $2}' <<<"$line")"
            # Get the date string, which is between brackets
        record_time="$(date --date="$line_timestamp" +'%s')"
            # Convert the date string to seconds.
            # Thanks to https://stackoverflow.com/a/1842754/2877364

        if (( record_time > date_10_min_ago)) ; then
            (( ++errorCount ))
            echo $line
        fi
    fi
done < 178.txt

echo "There were $errorCount error logs in the last 10 mins "

# vi: set ts=4 sts=4 sw=4 et ai:

Sample input

[Mon May 6 07:35:39.791442 2019] [autoindex:error] [pid 15012] [client 127.0.0.1:49054] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive - this error is older than 10 min
[Mon May 6 08:35:39.791442 2019] [autoindex:error] [pid 15012] [client 127.0.0.1:49054] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive
[Mon May 6 08:35:40.123456 2019] [autoindex:success] [pid 15012] [client 127.0.0.1:1337] Example input that doesn't contain the word "e r r o r"

Output

[Mon May 6 08:35:39.791442 2019] [autoindex:error] [pid 15012] [client 127.0.0.1:49054] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.php) found, and server-generated directory index forbidden by Options directive
There were 1 error logs in the last 10 mins

Explanation

• Comparing dates is much easier if they are numbers. The date +'%s' format gives a numeric count of seconds^* for each date.
• You can extract the timestamp from a line with awk -F '[][]'. The [][] is a regular expression that matches a [ or a ]. Therefore, awk -F '[][]' '{print $2}' gets the text of the error log's timestamp. Then date is used to convert that date text to a count of seconds for comparison.
• General comments: I prefer to wrap all my $() as "$()" if I want all the output together. I think that is clearer.

• As @lojza also mentioned, in general, any bash command must be entirely on one line. So DATE_10_MIN_HOURS=$(some random code) is fine, but

  DATE_10_MIN_HOURS=$(some random code
  and more code on a different line)
  

  will cause an error.

^{* This may not work as-is across leap seconds.}

Answer 2

I see newline after awk -F : and I would write:

DATE_10_MIN=`date --date='-10min' '+%H:%M:%S'` 
DATE_10_MIN_HOURS=`date --date='-10min' '+%H'`
DATE_10_MIN_MIN=`date --date='-10min' '+%M'`