4

Here is my sample log file.http://pastebin.com/DwWeFhJk.

When I am doing

tail -f log | awk '{if (NF>3) {print $1}; }'

the result I am getting is correct 

64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10
64.242.88.10

But when I am doing:

tail -f log |
awk '{if (NF>3) {print $1}; }' |
awk '{print $1}'

I am not getting any output. Even no output in case of

tail -f log | awk '{if (NF>3) {print $1}; }' | grep "64"

I am not getting the reason why the output of the first awk is not getting passed as the input of the second awk/grep after the pipe.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
Harsh Sharma
  • 10,942
  • 2
  • 18
  • 29
  • The command `tail -f log | awk '{if (NF>3) {print $1}; }' | grep "64"` gives me the same output as you show above. Are you sure that it is not giving you output? If it isn't, what OS are you using? – John1024 Oct 31 '15 at 05:36
  • I don't think it depends on OS. Anyways I am using arch. – Harsh Sharma Oct 31 '15 at 05:42
  • awk problems do commonly depend on OS when the OS is solaris. arch, by contrast, should be fine. Since you mentioned that the problem occurs for `tail -f` but not for `tail`, I believe that Jonathan has identified the most likely issue. – John1024 Oct 31 '15 at 05:59

3 Answers3

5

When the output of the first awk is going to the terminal, the output is line-buffered, so each line is printed as it is produced. When the output is going to the second awk or the grep, it is fully buffered. The output won't be sent until the buffer is full. When enough extra records are appended to the log, then the second awk will a buffer full of data to process. Until then, nothing will happen.

Jonathan Leffler
  • 730,956
  • 141
  • 904
  • 1,278
  • 1
    So, would `stbuf -oL` help here? – John1024 Oct 31 '15 at 05:51
  • If you mean [`stdbuf -oL awk …`](https://www.gnu.org/software/coreutils/manual/html_node/stdbuf-invocation.html), then probably. – Jonathan Leffler Oct 31 '15 at 05:57
  • How to get the desired result in case of awk. Also why the line-buffered thing is not happening after tail -f only. I mean output of tail -f should also get line buffered and no output should be seen even in the case of tail -f log | awk '{if (NF>3) {print $1}; } – Harsh Sharma Oct 31 '15 at 05:58
  • 1
    @HarshSharma: You can look up the `stdbuf` command which @John1024 mentioned, and for which I've provided a link to the documentation. As to `tail -f`, it probably explicitly sets line buffering, whereas `awk` does not. At least, that's a reasonable guess. The `tail -f` will never generate an EOF (well, not until someone or something kills it with an interrupt or other signal). – Jonathan Leffler Oct 31 '15 at 06:03
  • A command should use interactive line buffering when detects that its input or output is being sent to a tty. If both input and output are connected to a pipe, the command may choose to do full buffering. That is why the option (in some awk versions) for interactive exists (Please take a look at my answer). –  Oct 31 '15 at 09:36
5

You start the command with tail -f, that keeps the output open and therefore does not send a needed newline to the other commands.

This works perfectly fine:

cat log | awk '{if (NF>3) {print $1}; }' | grep 64

So, the problem is buffering. The middle awk is doing normal buffering instead of interactive buffering. This works (non-portably) with mawk:

tail -f log | mawk -W interactive '{if (NF>3) {print $1}; }' | awk '{print}'

You could read GNU description of the issue.

In any case, just check that the awk used in the middle could be told to buffer interactively.

Added:

The command system("") seems to unblock the buffering. It is POSIX, but does not work with mawk.

tail -f log | awk '{if (NF>3) {print $1}; system("")}' | awk '{print}'
2

search for "parallel --pipe" in the link to avoid the buffering

https://www.gnu.org/software/parallel/parallel_tutorial.html

nabeel
  • 319
  • 1
  • 9