In tail -f, how do I filter out stuff that has certain keywords?

Question

I want to tail -f my logs. However, I want to filter out everything that has the words:

"ELB", "Pingdom", "Health"

Answer 1

I don't know about using awk instead of grep, but this works for me:

tail -f file.log | grep -Ev '(ELB|Pingdom|Health)'

EDIT: As dmourati and Caleb pointed out, you could also use egrep instead of grep -E for convenience. On some systems this this will be an link to the same binary, in others a copy of it supplied by the grep package. Either way it lives as an alternative to the -E switch. However, according to the GNU grep man page:

[…]two variant programs egrep and fgrep are available. egrep is the same as grep -E. fgrep is the same as grep -F. Direct invocation as either egrep or fgrep is deprecated, but is provided to allow historical applications that rely on them to run unmodified.

Since they are synonymous commands, it comes down to preference unless you don't have egrep at all. However for forward compatibility it is recommended to use the grep -E syntax since the other method is officially deprecated.

Answer 2

Try piping it to egrep with a pipe separated lists of words you want to filter out:

tail -f log_file | egrep -v 'ELB|Pingdom|Health'

Note that using parenthesis around the list of matches is optional. Since the | is treated as a logical OR operator by grep whether it occurs as part of a sub-group or not. '(ELB|Pingdom|Health)' would function exactly the same. For some, the syntax may be more obvious; I find it easier to type without since I can switch from a single match to a list of possible matches without going back to add the parenthesis.

For extra credit, it's worth mentioning that multitail does ninja foo when it comes to filtering output. For example you could filter for your words like this:

multitail -e ELB -e Pingdom -e Health -f log_file

You could also use it to color or otherwise highlight the output instead of just filtering it.

EDit: See DTests answer and the comments for the full explanation of how egrep is just a deprecated alternate way to fire off grep -E.

Answer 3

tail -f /path/to/log | egrep -v 'ELB|Pingdom|Health'

Answer 4

Why do you want to log this information?

• Is it strictly for archival?
• Do you want to conditionally execute different scripts depending on different keywords or patterns in the log files?

If you want to have scripted behavior depending on the content of the log files, you may wish to do your filtering using Expect. ( http://en.wikipedia.org/wiki/Expect ) Expect is a Tcl extension but There is also a Python version of Expect.

Expect gives you this powerful flexible switch like statement that lets you specify different behaviors conditionally depending on the states, or patterns present in your input stream. For example:

expect {  
    "password:" {  
        send "password\r"  
    } 
    "yes/no)?" {  
        send "yes\r"  
        set timeout -1  
    }  
    timeout {  
        exit  
    }   
    -re . {  
        exp_continue  
    }  
    eof {  
        exit  
    }  
}

So you specify patterns in the expect statement, and you specify different behaviors, and you can wrap the whole thing in a loop, and you can easily write very powerful filters that also write portions of your input to different files, or drop it altogether, or take actions and run other scripts depending on what is in your input.

So, it comes down to why are you trying to filter your log files, to take action on log input, or just for archival reasons?