Stateful tail (only shows the new lines from the last execution)

Question

I want to be able to see how many lines were added to a file since the last quering without reading the whole file again.

Something like :

ptail my_file | fgrep "[ERROR]" | wc -l

A solution in simple Perl would be prefered, since I don't have an easy access to a compiler.

score 2 · Answer 1 · answered Oct 29 '08 at 20:30

2

since does exactly that although it is in C.

answered Oct 29 '08 at 20:30

Steve Schnepp

4,620
5
39
54

VonC · Answer 2 · 2008-10-29T21:00:52.323

May be this Perl package can help you:

File::Tail::Multi

Derived from MultiTail, this perl library makes it easy to tail a dynamic list of files and match/except lines using full regular expressions and even maintains their state locally.

EXAMPLE use File::Tail::Multi;

$tail1=File::Tail::Multi->new (  OutputPrefix => "f", 
                                 Debug => "$True", 
                                 Files => ["/var/adm/messages"]
                              );
while(1) {
    $tail1->read;
    #
    $tail1->print;
    sleep 10;
}

$tail1=File::Tail::Multi->new : Create new ptail object
Files => Tail file /var/adm/messages
OutputPrefix => Prepend the name of the file beginning of each line in object attribute "LineArray"
$tail1->read : Read all line from files
$tail1->print : Print all line in object attribute "LineArray";

Here the state is maintained, but only in one execution of the program. I would like the state to be saved across program executions. — Steve Schnepp, Oct 30 '08 at 13:02

score 2 · Accepted Answer · answered Oct 29 '08 at 23:50

Although it consumed the lines for other purposes, I have written code which does essentially this before.

All you need to do is record the byte offset (with tell) and inode (with stat) for each file after the tail is complete. The next time it's run against the file, first check the inode (with stat) again. If the inode has changed or the file is smaller than the recorded offset, then it's a different file (deleted and recreated, log got rotated, etc.), so you should show it from the beginning; otherwise, seek to the recorded offset and display it from there.

score 2 · Answer 4 · answered Oct 30 '08 at 13:18

I implemented a minimal version of a pure Perl version :

#! /usr/bin/perl
# Perl clone of since(1)
# http://welz.org.za/projects/since
#

use strict;
use warnings;

use Fcntl qw/ SEEK_SET O_RDWR O_CREAT /;
use NDBM_File;

my $state_file = "$ENV{HOME}/.psince";

my %states;
tie(%states, 'NDBM_File', $state_file, O_CREAT | O_RDWR, 0660)
        or die("cannot tie state to $state_file : $!");

while (my $filename = shift) {
        if (! -r $filename) {
                # Ignore
                next;
        }
        my @file_stats = stat($filename);
        my $device = $file_stats[0];
        my $inode = $file_stats[1];
        my $size = $file_stats[7];
        my $state_key = $device . "/" .$inode;
        print STDERR "state_key=$state_key\n";

        if (! open(FILE, $filename) ) {
                print STDERR "cannot open $filename : $!";
                next;
        }

        # Reverting to the last cursor position
        my $offset = $states{$state_key} || 0;
        if ($offset <= $size) {
                sysseek(FILE, $offset, SEEK_SET);
        } else {
                # file was truncated, restarting from the beginning
                $offset = 0;
        }

        # Reading until the end
        my $buffer;
        while ((my $read_count = sysread(FILE, $buffer, 4096)) > 0) {
                $offset += $read_count;
                print $buffer;
        }
        # Nothing to read
        close(FILE);
        $states{$state_key} = $offset;
}

# Sync the states
untie(%states);

@Dave: It's almost like your algorithm, except that i don't use tell, but an internal maintained counter.

Stateful tail (only shows the new lines from the last execution)

4 Answers4