19

i'm trying to build a shell script to monitor some log files. I'm using a command like this:

tail -f /var/somelog | grep --line-buffered " some test and p l a c e h o l d e r" | cut -f 3,4,14 -d " "

the log file is like:

some test and p l a c e h o l d e r 3
some test and p l a c e h o l d e r 4
some test and p l a c e h o l d e r 5
some test and p l a c e h o l d e r 6

and so on.. My issue is that the output of the command does not display the last line

some test and p l a c e h o l d e r 6

until line

some test and p l a c e h o l d e r 7

is added to the log.

I hope I made clear my issue. Can anyone help me to solve this? Thank you :)

nullrevolution
  • 3,937
  • 1
  • 18
  • 20
Andrea Tullis
  • 191
  • 1
  • 1
  • 6
  • 4
    Maybe the log lines are not being ended with a line feed, but started with one instead? In this case line 6 would not really be complete until line 7 starts and provides that line feed. If you look at the log file, showing all characters, do you see a line feed at the end of the final line? – David Ravetti Jan 16 '13 at 14:37
  • Look for a trailing new-line ("\n") using `od -c /var/somelog`. – cdarke Jan 16 '13 at 14:58
  • Do you see the same output with just `tail -f` ? If that is the case then @David has pointed out it correctly. – mtk Jan 16 '13 at 16:39
  • I'm not al work at the moment, but if I run tail -f i can see the full log. I will try your solutions this evening and keep you posted :) – Andrea Tullis Jan 16 '13 at 17:30
  • 2
    my guess is that you will see the full output (including the last line) with `tail -f`, and even with `tail -f | grep --line-buffered "pattern"`, but you will lose the last line once you introduce cut into the pipeline. – nullrevolution Jan 16 '13 at 18:34
  • @nullrevolution you made the point. – Andrea Tullis Jan 16 '13 at 21:52

1 Answers1

25

the problem is almost certainly related to how grep and cut buffer their output. here's a hack that should get you around the problem, though i'm sure there are prettier ways to do it:

tail -f /var/somelog | while read line; do echo "$line" | grep "some test and p l a c e h o l d e r" | cut -f 3,4,14 -d " "; done

(don't forget the ; done at the end of the command)

alternatively, because gawk doesn't buffer it's output, you could use it in place of cut to avoid the cumbersome while loop:

tail -f log | grep --line-buffered "some test and p l a c e h o l d e r" | gawk '{print $3,$4,$14}'

check out http://www.pixelbeat.org/programming/stdio_buffering/ for more info on buffering problems.

nullrevolution
  • 3,937
  • 1
  • 18
  • 20
  • 1
    It's also quite probable that whatever is writing to `/var/somelog` is buffering, so `tail` may not even see changes until it overflows a buffer or flushes its output some other way... – twalberg Jan 16 '13 at 16:38
  • I have tried the first solution and is working. Thank you for your help! – Andrea Tullis Jan 16 '13 at 21:57
  • glad to help. don't forget that the best way to say thanks around here is to accept/upvote helpful answers! :) – nullrevolution Jan 16 '13 at 23:28
  • oh man I can't believe it but cut appears to buffer its output, whereas grep et al don't. Oh man that feels lame... – rogerdpack Jul 30 '13 at 23:29