4

There is a file located at $filepath, which grows gradually. I want to print every line that starts with an exclamation mark:

while read -r line; do 
    if [ -n "$(grep ^! <<< "$line")" ]; then
        echo "$line"
    fi
done < <(tail -F -n +1 "$filepath")

Then, I rearranged the code by moving the comparison expression into the process substitution to make the code more concise:

while read -r line; do
    echo "$line"
done < <(tail -F -n +1 "$filepath" | grep '^!')

Sadly, it doesn't work as expected; nothing is printed to the terminal (stdout).

I prefer to write grep ^\! after tail. Why doesn't the second code snippet work? Why putting the command pipe into the process substitution make things different?

PS1. This is how I manually produce the gradually growing file by randomly executing one of the following commands:

echo ' something' >> "$filepath"
echo '!something' >> "$filepath"

PS2. Test under GNU bash, version 4.3.48(1)-release and tail (GNU coreutils) 8.25.

Kevin Dong
  • 5,001
  • 9
  • 29
  • 62
  • @iBug Wow. Thanks for the feedback. I re-tried the above code carefully, and it still does work. I am using GNU Bash of version 4.3.48(1)-release on Ubuntu 16.04. – Kevin Dong Apr 07 '18 at 06:46
  • for second version, can you try `grep --line-buffered` see https://stackoverflow.com/questions/13858912/piping-tail-output-though-grep-twice – Sundeep Apr 07 '18 at 07:00
  • @Sundeep Thanks. Your comment is sightly earlier than iBug's post. – Kevin Dong Apr 07 '18 at 07:02
  • Unbelievable... It actually has nothing to do with `tail -F`, which confused me for quire some time before figuring out that it's `grep`'s fault. – iBug Apr 07 '18 at 07:04
  • @Sundeep you beat me to it. Your comment turns out to be the correct answer. – iBug Apr 07 '18 at 07:05
  • the duplicate is about *grep twice*. maybe change title with "Piping tail output though grep"? @Sundeep – kyodev Apr 07 '18 at 09:53
  • @kyodev yeah, the duplicate question's title could be bettered.. though not sure what'd be best... the issue is not restricted to grep alone... it would happen whenever tail -f/F output is processed by a tool (that doesn't do line-buffering) and is then passed on to another process.. – Sundeep Apr 07 '18 at 10:01
  • yes, iBug said that wel: "grep is not line-buffered when its stdout isn't connected to a tty". in fact, it's the duplicate title that should be changed, **twice** or more is not important. here the title could also be improved, imho @Sundeep – kyodev Apr 07 '18 at 10:07
  • @kyodev Thanks. I changed the title for programmers who will face the similar situation to search more easily with the following keywords: **no output**, `tail`, and `grep`. – Kevin Dong Apr 07 '18 at 10:22

1 Answers1

4

grep is not line-buffered when its stdout isn't connected to a tty. So it's trying to process a block (usually 4 KiB or 8 KiB or so) before generating some output.

You need to tell grep to buffer its output by line. If you're using GNU grep, this works:

done < <(tail -F -n +1 "$filepath" | grep '^!' --line-buffered)
                                               ^^^^^^^^^^^^^^^
iBug
  • 35,554
  • 7
  • 89
  • 134