awk a huge logfile from behind until timestamp

Question

i want to get the last part since a given timestamp "t0" from a possible huge logfile (>50..1000mb):

     __________________
    |1 xxx xxx ...     |
    |2 xxx ...         |     uninteresting part
    |4 ...             |
    |...               |
 ___|423 ...           | ___ timestamp t0
    |425 xxx ...       |
    |437 ...           |
    |...               |     <-- i want this part ( from t0 to EOF)
    |__________________|

and an additional constraint is that i want to do this using simple bash commands. a simple solution may be:

awk '$1 > 423' file.log

but this scans the whole file with all the unintresting lines. there's the command tail but i just can give him the number of last lines i want which i don't know - i just know the timestamp. is there a way "awking" from behind and stop processing when the first timestamp doesn't match?

score 4 · Accepted Answer · answered Apr 30 '11 at 19:17

4

tac is your friend here:

tac file.log | awk '{ if ($1 >= 423) print; else exit; }' | tac

tac will dump each line of a file starting with the last line, then working to the beginning of the file. do it once to get the lines you want, then do it again to fix their order.

answered Apr 30 '11 at 19:17

linuts

6,608
4
35
37

perfect! this strips down the processing time **to lower than 5%** compared with the simple awk solution above. thx you :) – kraiz May 03 '11 at 12:03

score 1 · Answer 2 · answered Apr 29 '11 at 13:49

If I understand right you just need to get n lines from a timestamp regexp to the end of the file.

Lest say your huge file is something like this:

~$ cat > file << EOF
rubish
n lines of rubish
more rubish
timestamp regexp
interesting
n interesting lines
interesting
end of file
EOF

If you are able to get a feasible regexp for the timestamp you are looking for, you can get the part you want with sed:

~$ sed -n '/timestamp regexp/,$ {p}' file
timestamp regexp
interesting
n interesting lines
interesting
end of file

score 1 · Answer 3 · answered Apr 29 '11 at 14:04

Using standard Unix commands, there isn't much you can do other than scan the entire file. If you write your own program, you could do a binary search on the file:

seek to a point in the file,
read forwards to the next start of record,
check whether the timestamp is too big or too small,
and iterate until you find the right point in the file.

You might even do a search with linear interpolation rather than a pure binary search if the time stamps are pure numbers; it probably isn't worth the extra coding if the stamps are more complex, but it depends on how often you're going to need this.

Indeed, unless you are going to be doing this a lot and can demonstrate that the performance is a problem, I'd go with the simple awk solution.

score 0 · Answer 4 · answered Apr 29 '11 at 08:07

0

you can poll until you hit "423". Just a hypothetical example (not tested)

n=100 # number of lines you want to go back
while true
do
  if tail -$n file | grep -q "423" ;then
     tail -$n file | awk '$1>423'
     break
  else
     ((n+=100)) # increment every 100 lines
  fi
done

answered Apr 29 '11 at 08:07

ghostdog74

327,991
56
259
343

That has quadratic behaviour as the length of the tail grows, I think. – Jonathan Leffler Apr 29 '11 at 14:04
I think, too. Moving instead of enlarging the tail-"window" would be better. – kraiz May 03 '11 at 12:22

awk a huge logfile from behind until timestamp

4 Answers4