3

2011-04-13 00:09:07,731 INFO [STDOUT] 04/13 00:09:07 Information...

Hi everyone. I would post some of my code, but I don't even think it's worthy of posting. What I'm trying to do is that I've got a log file with lines like above. What I need to do is take the last lines timestamp, and keep all the lines from the last 5 minutes (rather than the last 200 lines or whatever....which would be easier). Could anyone help? I've searched the web, some decent tips, but still nothing going and frustrated as hell. Thanks!

toneman
  • 45
  • 5

4 Answers4

5

Here's a simple Perl script that iterates over the file and prints every line whose timestamp is within 5 minutes of the time at the start of execution. For more efficiency, and assuming that the lines are in timestamp order, you could modify this to set a boolean flag when it encounters the first printable line and skip the testing from that point forwards.

#!/usr/bin/perl
use POSIX qw(mktime);

$now = time();
while(<>)
{
    ($yy,$mm,$dd,$h,$m,$s,$t) = /^(\d+)-(\d+)-(\d+)\s+(\d+):(\d+):(\d+),(\d+)/;
    $t = mktime($s+$t/1000, $m, $h, $dd, $mm-1, $yy-1900);
    print "$_" if ($t >= $now-300);
}
Jim Garrison
  • 85,615
  • 20
  • 155
  • 190
  • Why do you use `ge` after you have gone to the trouble of converting the timestamp into a number? – mob Apr 25 '11 at 20:02
  • 1
    The program's start time is already in [`$^T`](http://perldoc.perl.org/perlvar.html#%24%5eT). – Greg Bacon Apr 25 '11 at 20:51
  • So I guess I wasn't too clear, but the last line of the server.log should end like this: 2011-04-13 23:58:21,110 INFO [STDOUT] 04/13 23:58:21 Information [scheduler-2] - Run Client Storage Purge, Rather than go from execution time and 5 minutes from there, I need to read at the end of the log for the last timestamp, and 5 minutes from there. Thanks! (Sorry about the formatting, I'm still new to the site) – toneman Apr 26 '11 at 20:35
2

I take it by your latest comment that you are interested in finding out how to find the timestamp that is last in your log, and the entries that are 5 minutes before that.

I think Jim Garrison's solution could be patched to replace this:

$now = time();

with this:

open F, "<server.log" or die $!;
seek F,-1000,2; # set pos to last 1000 bytes
my @f = <F>;
$_ = $f[$#f];
($yy,$mm,$dd,$h,$m,$s,$t) = /^(\d+)-(\d+)-(\d+)\s+(\d+):(\d+):(\d+),(\d+)/;
$now = mktime($s+$t/1000, $m, $h, $dd, $mm-1, $yy-1900);

$now should now contain the last timestamp in the log.

I approximated "-1000" to be long enough to go past at least one line in the log. You could set it much higher if you expect to have long lines in the log, but from what I saw, the last log entry "should" be fairly short.

If you have a huge log file and want to increase performance in the following search, you can use an estimation and perform a seek to find the last, say, 1000000 bytes in the file with:

seek F, -1000000, 2;

Good luck!

TLP
  • 66,756
  • 10
  • 92
  • 149
1

Iterate over all the lines, using regexp grab: 00:09:07, and check against current time (localtime, etc...). if the file contains entries from different dates, then also grab the dates using regexp, and again compare using the output of locatime

snoofkin
  • 8,725
  • 14
  • 49
  • 86
0

How to modify your script to make it work with the logs below

Dec 18 09:41:18 sd
Dec 18 09:46:29 sds
Dec 18 09:48:39 sds
Dec 18 09:48:54 sds
Dec 18 09:54:47 sds
Dec 18 09:55:33 sds
Dec 18 09:55:38 sds
Dec 18 09:57:58 sds
Dec 18 09:58:10 sds
Dec 18 10:00:50 sdsd
Dec 18 10:03:43 sds
Dec 18 10:03:50 sdsd
Dec 18 10:04:06 sdsd
Dec 18 10:04:15 sdsd
Dec 18 10:14:50 wdad
Dec 18 10:19:16 sdadsa
Dec 18 10:19:23 dsds
Dec 18 10:21:03 sadsd
Dec 18 10:22:54 adas
Dec 18 10:27:32 qadad
user1204671
  • 183
  • 1
  • 1
  • 8