Issues with echo commands?

Question

I run this script as quanta suggest me

#!/bin/bash

EDATE=$(tail -1 a | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < a

I checked that and it seems that your script want to check log file line by line. Since it has more than 14000 items it takes a lot. So when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked! It's impossible! It just work for short log file I think. Is the problem because of while?

It seems like this could be a lot more efficient, you're right. Could you add a few lines of the original source file and few lines of the intended output for those lines? — SmallClanger, Sep 23 '11 at 13:35
So for a given apache log file, all you want is a file `as1` containing one unix epoch time (5 days prior to the log date) per line? Or do you want the file to contain all of the intermediate output, too? — SmallClanger, Sep 23 '11 at 14:01
yeah exactly file as1 contains one date and that is 5 days prior to the end date. I just need that file not all of those echo commands ! — matarsak, Sep 23 '11 at 14:11
I checked that and it seems that your script want to check log file line by line . since it has more than 14000 items it takes alot . so when I run it the cursor goes to next line and next line and I should press it 14000 times so that the whole log file be checked ! it's impossible ! it just work for short log file I think . is the problem because of while ? — matarsak, Sep 24 '11 at 17:05
If it is sorted in ascending order, just get the line number of matching pattern, and print from that to the end of file, instead of convert to Epoch time, ... http://serverfault.com/questions/314842/how-to-manage-dates-in-shell — quanta, Sep 24 '11 at 17:12
while I run this script i face with these errors ! nadate: invalid date ` ::' date: invalid date ` ::' — matarsak, Sep 26 '11 at 18:24

SmallClanger · Accepted Answer · 2011-09-23T14:49:46.667

2

This one liner (I've split it for clarity) should give you the same result. You can optionally add > as1 to the end to redirect the output to a file. Put the path to the apache log file where I've put <logfile>

for d in \
 $(sed -nre 's/.*\[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' <logfile> | date +%s -f-);
do echo $[ $d - 86400 * 5]; done

The date command doesn't need an explicity UTC formatted date for the -dargument, although it doesn't understand dates as the apache logs output them, so I've done some substitution to swap the slashes and the colon separating the date and time with spaces.

edited Sep 23 '11 at 14:49

answered Sep 23 '11 at 14:40

SmallClanger

9,127
1
32
47

I run that but unsuccessful ! I could not believe that just one line code could do what I wrote ! ./8: command substitution: line 4: syntax error near unexpected token `|' ./8: command substitution: line 4: `sed -nre 's/.*\[(..)\/(...)\/(....):(..:..:..) .*/\1 \2 \3 \4/p' | date +%s -f-)' – matarsak Sep 23 '11 at 15:04
Drop the `<` and `>` from the `aa.log` bit. You just need the filename there. – SmallClanger Sep 23 '11 at 15:14
I dont know what your magic code suppose to do but it gives me out put like this 1041022520 so your code convert the time to epoch time ? because I recognize that you also subtract 5 days , I dont understand – matarsak Sep 23 '11 at 15:18
That looks about right, given that the log file sample you gave is from 2002. Use http://www.epochconverter.com/ to convert those numbers back to dates and you should see that they're 5 * 24 hours priror to the date in your log file. – SmallClanger Sep 23 '11 at 15:27
It's a little bit different, my code get the end date and subtract 5 days from it and then print all of those 5 last days ! your's subtract 5 days from all of the log file ! – matarsak Sep 23 '11 at 15:37
So what you actually want is the *full lines* from the original log file that are within 5 days of the last date in the file? – SmallClanger Sep 23 '11 at 16:02
yeah exactly , and then I want to generate 5 to X . – matarsak Sep 23 '11 at 16:23

quanta · Answer 2 · 2011-09-24T00:50:22.317

Here for you:

#!/bin/bash

EDATE=$(tail -1 aa.log | awk '{ print substr($4, 2, length($4)) }')
EEPOCH=$(date -d "$(echo "$EDATE" | awk 'BEGIN { FS = "[/:]"; } \
           { print $1" "$2" "$3" "$4":"$5":"$6 }')" +%s)
time=$(expr 60 \* 60 \* 24 \* 5)
SEPOCH=$(expr $EEPOCH - $time)

while read line
do
    DATE=$(echo $line | awk '{ print substr($4, 2, length($4)-1) }' | \
        awk 'BEGIN { FS = "[/:]"; } { print $1" "$2" "$3" "$4":"$5":"$6 }')
    DEPOCH=$(date -d "$DATE" +%s)
    [[ $DEPOCH -ge $SEPOCH && $DEPOCH -le $EEPOCH ]] && echo $line | \
        awk '{ print substr($4, 2, length($4)) }' >> as1
done < aa.log

UPDATE

The input:

213.46.27.204 - - [21/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [22/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [23/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [24/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [25/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [26/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [27/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [28/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [29/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [30/Dec/2002:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""
213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" ""

The output:

$ ./five_last_days.sh 
27/Dec/2002:12:55:21
28/Dec/2002:12:55:21
29/Dec/2002:12:55:21
30/Dec/2002:12:55:21
01/Jan/2003:12:55:21

the output is so long ! it print from 8th of December till the end ! while it should print from 27th ! the last item in log file is 213.46.27.204 - - [01/Jan/2003:12:55:21 +0100] "GET /scripts/..%252f../winnt/system32/cmd.exe?/c+dir HTTP/1.0" 404 - "" "" — matarsak, Sep 23 '11 at 17:39
Cut 30 last lines from your input, run my script, and edit your first post (to keep formatting) to show me the output. — quanta, Sep 24 '11 at 00:51
I checked the script with your data and the output was the same as yours , but in original logfile it does nothing ! actually when I run the cursor goes to next line and next line and after a long time I press control+C , maybe it try to check the whole logfile and because it has 14000 line it takes so much time ! I could not press enter for 2 hours so that the whole log file be checked ,how could I upload the original logfile here? — matarsak, Sep 24 '11 at 16:28

Issues with echo commands?

2 Answers2