46

I'm trying to parse some log files as they're being written in Go but I'm not sure how I would accomplish this without rereading the file again and again while checking for changes.

I'd like to be able to read to EOF, wait until the next line is written and read to EOF again, etc. It feels a bit like how tail -f looks.

Nick
  • 9,792
  • 7
  • 50
  • 60

5 Answers5

63

I have written a Go package -- github.com/hpcloud/tail -- to do exactly this.

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{Follow: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

...

Quoting kostix's answer:

in real life files might be truncated, replaced or renamed (because that's what tools like logrotate are supposed to do).

If a file gets truncated, it will automatically be re-opened. To support re-opening renamed files (due to logrotate, etc.), you can set Config.ReOpen, viz.:

t, err := tail.TailFile("/var/log/nginx.log", tail.Config{
    Follow: true,
    ReOpen: true})
for line := range t.Lines {
    fmt.Println(line.Text)
}

Config.ReOpen is analogous to tail -F (capital F):

 -F      The -F option implies the -f option, but tail will also check to see if the file being followed has been
         renamed or rotated.  The file is closed and reopened when tail detects that the filename being read from
         has a new inode number.  The -F option is ignored if reading from standard input rather than a file.
mat007
  • 905
  • 8
  • 16
Sridhar Ratnakumar
  • 81,433
  • 63
  • 146
  • 187
9

You have to either watch the file for changes (using an OS-specific subsystem to accomplish this) or poll it periodically to see whether its modification time (and size) changed. In either case, after reading another chunk of data you remember the file offset and restore it before reading another chunk after detecting the change.

But note that this seems to be easy only on paper: in real life files might be truncated, replaced or renamed (because that's what tools like logrotate are supposed to do).

See this question for more discussion of this problem.

Scott Stensland
  • 26,870
  • 12
  • 93
  • 104
kostix
  • 51,517
  • 14
  • 93
  • 176
  • 1
    Notify and winfsnotify are [exp](http://weekly.golang.org/pkg/exp/) packages currently. – Sonia Apr 13 '12 at 15:13
  • @kostix I actually thought your answer on the other question was more helpful. But this answer was useful too. – Nick Apr 15 '12 at 01:04
7

A simple example:

package main

import (
    "bufio"
    "fmt"
    "io"
    "os"
    "time"
)

func tail(filename string, out io.Writer) {
    f, err := os.Open(filename)
    if err != nil {
        panic(err)
    }
    defer f.Close()
    r := bufio.NewReader(f)
    info, err := f.Stat()
    if err != nil {
        panic(err)
    }
    oldSize := info.Size()
    for {
        for line, prefix, err := r.ReadLine(); err != io.EOF; line, prefix, err = r.ReadLine() {
            if prefix {
                fmt.Fprint(out, string(line))
            } else {
                fmt.Fprintln(out, string(line))
            }
        }
        pos, err := f.Seek(0, io.SeekCurrent)
        if err != nil {
            panic(err)
        }
        for {
            time.Sleep(time.Second)
            newinfo, err := f.Stat()
            if err != nil {
                panic(err)
            }
            newSize := newinfo.Size()
            if newSize != oldSize {
                if newSize < oldSize {
                    f.Seek(0, 0)
                } else {
                    f.Seek(pos, io.SeekStart)
                }
                r = bufio.NewReader(f)
                oldSize = newSize
                break
            }
        }
    }
}

func main() {
    tail("x.txt", os.Stdout)
}
seduardo
  • 704
  • 6
  • 6
4

I'm also interested in doing this, but haven't (yet) had the time to tackle it. One approach that occurred to me is to let "tail" do the heavy lifting. It would likely make your tool platform-specific, but that may be ok. The basic idea would be to use Cmd from the "os/exec" package to follow the file. You could fork a process that was the equivalent of "tail --retry --follow=name prog.log", and then listen to it's Stdout using the Stdout reader on the the Cmd object.

Sorry I know it's just a sketch, but maybe it's helpful.

laslowh
  • 8,482
  • 5
  • 34
  • 45
  • 1
    Instead of executing tail, simply make it read from os.Stdin and have something else pipe the data to you instead. That way you don't need to figure out what the process should be, someone else gives it to you. This would make it work not only with tools like "tail" but also for any platform that supports piping output for logs (e.g. Apache web server supports custom loggers by executing an application to pipe data to it). – pasamio Jan 25 '16 at 16:18
3

There are many ways to do this. In modern POSIX based Operating Systems, one can use the inotify interface to do this.

One can use this package: https://github.com/fsnotify/fsnotify

Sample code:

watcher, err := fsnotify.NewWatcher()
if err != nil {
    log.Fatal(err)
}

done := make(chan bool)

err = watcher.Add(fileName)
if err != nil {
    log.Fatal(err)
}
for {
    select {
    case event := <-watcher.Events:
        if event.Op&fsnotify.Write == fsnotify.Write {
            log.Println("modified file:", event.Name)

        }
}

Hope this helps!

mirage
  • 632
  • 10
  • 21