How to prevent pipes from delaying output?

Question

When I run below command in bash then it waits until the program has finished before flushing all output. If I remove the pipes then it prints each line immediately.

{ for i in `seq 3` ; do echo $i ; sleep 1 ; done ; } \
    | perl -p -e 's,(.*ERROR.*),\e[01;31m\1\e[00m,g' \
    | perl -p -e 's,(.*WARNING.*),\e[01;33m\1\e[00m,g' \
    | perl -p -e 's,(.*TCPEchoTest.*),\e[01;30m\1\e[00m,g' \
    | perl -p -e 's,(.*enters.*),\e[00;33m\1\e[00m,g'

How can I use the pipes and still have each line printed immediately?

Answer 1

Pipes typically have a default 4kB buffer where the output may be held until the program finishes executing.

Try using the stdbuf utility to disable this buffer so that text is output immediately:

stdbuf -i0 -o0 -e0 command | perl ...

See also:

Turn off buffering in pipe

Answer 2

perl is doing the buffering, so the perl commands are what you need to change to turn it off. Set the $| variable in each perl command, like this:

{ for i in `seq 3` ; do echo $i ; sleep 1 ; done ; } \
    | perl -p -e 'BEGIN{$|=1}s,(.*ERROR.*),\e[01;31m\1\e[00m,g' \
    | perl -p -e 'BEGIN{$|=1}s,(.*WARNING.*),\e[01;33m\1\e[00m,g' \
    | perl -p -e 'BEGIN{$|=1}s,(.*TCPEchoTest.*),\e[01;30m\1\e[00m,g' \
    | perl -p -e 'BEGIN{$|=1}s,(.*enters.*),\e[00;33m\1\e[00m,g'

the BEGIN block causes the assignment to happen once, before reading any input, instead of doing it after each line is read with -p.

Answer 3

First of all, use a log coloring tool like colortail, colorize or ccze.

In the more general case, the problem is that libc will automatically optimize throughput by buffering when stdout is not a terminal. This can dramatically reduces the number of syscalls for non-interactive jobs. For this and more, compare interactive and non-interactive output.

When the process thinks it's not interactive, while you know it is, there are several ways of convincing it to flush its buffers after each logical write:

The Wumpus's approach, interpretter instructions for changing buffering:

• $|=1 or the equivalent STDOUT->autoflush(1) in Perl
• fflush() after writing in awk
• sys.stdout.flush() after writing in Python

Program specific settings:

• -u in Python and GNU sed
• --line-buffered in GNU awk

Opening a PTY for the terminal, making stdout a TTY and circumventing the problem:

• script -q /dev/null yourcommand
• unbuffer from the expect package

And finally $LD_PRELOAD hacks that inject options into a process's C runtime (neatly wrapped in the GNU coreutils stdbuf, suggested by Vilhelm)

Answer 4

You can concatenate several lines of perl code with repeated -e options (be sure to end them with ; -- they are strung together to form a program). And you can make your pipes piping "hot" with $|=1. See the perl manual on $| for details (2/3 down the page, search for OUTPUT_AUTOFLUSH).

{ for i in `seq 3` ; do echo $i ; sleep 1 ; done ; } \
  | perl -p -e 'BEGIN{$|=1};' \
            -e 's,(.*ERROR.*),\e[01;31m\1\e[00m,g;' \
            -e 's,(.*WARNING.*),\e[01;33m\1\e[00m,g;' \
            -e 's,(.*TCPEchoTest.*),\e[01;30m\1\e[00m,g;' \
            -e 's,(.*enters.*),\e[00;33m\1\e[00m,g;'

This prints 1, 2, 3 with one second in between each number. In fact, the BEGIN line is not needed when perl output is to the terminal. But you want it if you keep piping to another program.