tail multiple files and grep the output

Question

I would like to grep a pattern from multiple log files which are being constantly updated by some processes and tail the output of this grep continuosly. Below command doesnt work and I get

tail: warning: following standard input indefinitely is ineffective

tail -f  | grep --line-buffered "Search this: " /var/links/proc2/id/myprocess*/Daily/myprocess*.log

Can someone help sort this out?

Duplicate of http://superuser.com/questions/59471/simple-use-of-tail-grep-multiple-conditions — Evan Carroll, Dec 19 '14 at 23:42

Anshul Goyal · Accepted Answer · 2013-09-30T10:29:58.500

14

You should have a look at multitail tool (Install using sudo apt-get install multitail)

In short, with multitail, you need to use the --mergeall flag for viewing output of all in one place

multitail --mergeall /var/links/proc2/id/myprocess*/Daily/myprocess*.log  | grep --line-buffered "Search this: "

You can do the same without using grep

multitail -E "Search this: " --mergeall /var/links/proc2/id/myprocess*/Daily/myprocess*.log

To view the output individually using multitail, this will give the filename as well.

multitail -E "Search this: " /var/links/proc2/id/myprocess*/Daily/myprocess*.log

edited Sep 30 '13 at 10:29

answered Sep 30 '13 at 08:35

Anshul Goyal

73,278
37
149
186

3

What version of multitail are you using? I am unable to grep the output of multitail with the version packaged with Debian: _multitail 5.2.13_ – Thor Sep 30 '13 at 10:07
@Thor Corrected my answer. – Anshul Goyal Sep 30 '13 at 10:22
@mu無 Once you're finished, how do you get rid of the bar underneath your terminal? I did Ctrl+C and it exited the multitail, but that bar is still there. Not even logging out of the server helped. I had to completely close my terminal. Annoying!!! – DevOpsSauce Sep 11 '19 at 13:01
@IRGeekSauce Don't have access to a linux machine right now, so can't check. You can try asking a new question on the site for help from community :) – Anshul Goyal Sep 11 '19 at 13:44
1

@mu無 I figured it out. Just type "clear". – DevOpsSauce Sep 11 '19 at 15:10

score 11 · Answer 2 · edited Sep 30 '13 at 12:39

11

the mistake is that you give the files to the grep command and not the tail.

the tail -f needs to get the files as input. try:

tail -f  /var/links/proc2/id/myprocess*/Daily/myprocess*.log | grep --line-buffered "Search this: "

to get also the file names (however it will not be like grep output it is):

tail /var/links/proc2/id/myprocess*/Daily/myprocess*.log | grep --line-buffered -e'^==> .* <==$' -e'Search this: '

edited Sep 30 '13 at 12:39

Thor

45,082
11
119
130

answered Sep 30 '13 at 08:08

Udy

2,492
4
23
33

3

thanks. This works partially but I dont get the filenames where the pattern is found? – 212 Sep 30 '13 at 08:19
1

solution: `for f in path/to/files*.txt; do echo $f; tail $f | grep 'SEARCH-THIS'; done;` – cgnorthcutt Feb 14 '21 at 04:31

score 1 · Answer 3 · edited May 23 '17 at 11:53

This is an interesting question and the simple answer should be: Use the prefix switch with tail, but unfortunately this is currently not implemented in most versions of tail.

As I see it, you have two options: adapt the standard tools to the task (see Udys answer) or write your own tool with your favorite scripting/programming language.

Below is one way you could do it with the File::Tail::Multi module for perl. Note that you may need to install the module from CPAN (cpan -i File::Tail::Multi).

Save the following script e.g. mtail to your executable path and make the script executable.

#!/usr/bin/env perl

use File::Tail::Multi;

$| = 1;  # Enable autoflush

$tail = File::Tail::Multi->new(RemoveDuplicate => 0,
                               OutputPrefix    => 'f',
                               Files           => \@ARGV);

while(1) { $tail->read; $tail->print; sleep 2 }

Change OutputPrefix to 'p' if you prefer full path prefixes.

Run it like this:

mtail /var/links/proc2/id/myprocess*/Daily/myprocess*.log | grep --line-buffered "Search this: "

You do not need to specify --line-buffered when grep is the last command, so this is sufficient:

mtail /var/links/proc2/id/myprocess*/Daily/myprocess*.log | grep "Search this: "

1) How is this different from using multitail? 2) Will it output by filenames, as OP asks in comments to @Udys answer> — Anshul Goyal, Sep 30 '13 at 14:00
@ansh0l: 1) multitail outputs its results in curses, which is not greppable. 2) Yes, each line has the originating file name prefixed. — Thor, Sep 30 '13 at 15:11
Ah ok, +1 for curses. But with -E flag of multitail, you won't need to do the grep itself. — Anshul Goyal, Sep 30 '13 at 15:23

tail multiple files and grep the output

3 Answers3

Linked

Related